JP2022026016A - Information processing device, information processing method, and program - Google Patents

Abstract

To provide an information processing device capable of reducing an operator load and highly accurately detecting various abnormalities from a video, an information processing method, and a program.SOLUTION: An abnormality detection device 1 (information processing device) comprises: a video acquisition unit which acquires video data; a feature extraction unit which extracts voice features from the video data acquired by the video acquisition unit and extracts image features from the video data; an abnormal scene candidate detection unit which detects an abnormal scene candidate from the video data on the basis of the voice features extracted by the feature extraction unit; an abnormality determination unit which determines the abnormal scene candidate detected by the abnormal scene candidate detection unit to be abnormal, normal, or the other on the basis of the voice features and the image features; and a scene presentation unit which presents the abnormal scene candidate via a user interface and accepts an input of information to be added to the presented abnormal scene candidate via the user interface when the abnormality detection unit determines that the abnormal scene candidate belongs to the other.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing apparatus, an information processing method and a program, and more particularly to a technique for analyzing an image and detecting an abnormality.

Recent video distribution services enable real-time distribution of not only video content created by content providers but also video content created by general users.
In such a video distribution service, the distributed video is monitored so that the distributed video content does not include so-called abnormal scenes that are inappropriate for viewing, and the detected abnormal scene is erroneously viewed. It is necessary to delete abnormal scenes, stop distribution, delete distribution accounts, etc. so that they will not be deleted. Such anomalous scenes include non-family-safe (NFS) scenes that are unsuitable for family viewing, such as violent scenes and non-children's scenes.

Patent Document 1 discloses an elevator monitoring device that detects abnormal behavior of an occupant from shooting data taken by a security camera provided in an elevator car.
Specifically, in the monitoring device of Patent Document 1, the rampage determination threshold value is obtained according to a predetermined frequency band extracted from the result of frequency analysis of the voice data of the occupant collected by the intercom installed in the car. And statistically calculate the amount of variation in the movement of the occupants from the shooting data taken by the security camera. The monitoring device of Patent Document 1 further compares the calculated amount of variation in the movement of the occupant with the rampage determination threshold value, and considers the movement of the occupant as abnormal behavior when the amount of variation in the movement of the occupant is equal to or greater than the rampage determination threshold value. Judge the rampage. As a result, even if the occupant can move only slightly, the abnormal behavior is determined from the shooting data.

Japanese Unexamined Patent Publication No. 2013-63810

However, in the technique of Patent Document 1, since the detectable abnormality is limited to the rampage of the occupant in the elevator, it is difficult to appropriately detect various abnormal scenes that can be included in various video contents. ..

In particular, video distribution services are broadly segmented according to the age group and preferences of the main target users, and the range of abnormal scenes that are inappropriate for viewing varies depending on the video distribution service. .. Furthermore, since the frequency of abnormal scenes appearing in video content is usually small, it is not suitable for creating a general-purpose database of learning data required for supervised machine learning. On the other hand, if an attempt is made to detect an abnormal scene from video content by unsupervised machine learning, the detection accuracy will decrease.

By the way, in the video created by the content provider, tag information such as whether or not the video content distributed by the content provider includes a violent scene, whether or not the content is for children, or whether or not there is an age limit is included. It is often added, and it is also possible to have the content provider tag the existence of an abnormal scene when creating content.
On the other hand, video content created by general users, which has been increasing in recent years, often does not have tag information of such abnormal scenes added, or even if it is added, tagging is not always the video distribution service. May not be appropriate.

For this reason, conventionally, depending on the video distribution service, the operator constantly monitors the video content to be distributed, and when an abnormal scene is found in the video content, an age limit is set for the video content or the video is concerned. The distribution of content has been stopped, which has increased the time and work load of the operator who monitors the video, and also the psychological burden. At the same time, there was a risk that an abnormal scene might be overlooked by manually monitoring the video content.

The present invention has been made to solve the above problems, and an object thereof is an information processing device and an information processing method capable of detecting various abnormalities from an image with high accuracy while reducing the load on an operator. And to provide the program.

In order to solve the above problems, one aspect of the information processing apparatus according to the present invention is a video acquisition unit for acquiring video data and an audio feature extracted from the video data acquired by the video acquisition unit. A feature extraction unit that extracts image features from, an abnormal scene candidate detection unit that detects abnormal scene candidates from the video data based on the audio features extracted by the feature extraction unit, and the abnormal scene candidate detection unit. An abnormality determination device that determines whether the abnormality scene candidate detected by the above-mentioned is abnormal, normal, or other based on the voice feature and the image feature, and the abnormality determination device, the candidate of the abnormality scene. If is determined to belong to something else, the candidate for the abnormal scene is presented via the user interface, and the input of information to be added to the presented candidate for the abnormal scene is accepted via the user interface. It is equipped with a scene presentation unit.

The abnormal scene candidate detection unit may detect the abnormal scene candidate from the video data by using unsupervised learning.

The abnormal scene candidate detection unit may detect the candidate of the abnormal scene from the video data by directly separating the abnormal audio feature without generating a model of the normal audio feature group.

The abnormal scene candidate detection unit may separate the abnormal voice feature by calculating the path length in the isolation forest of each voice feature.

The feature extraction unit may extract audio features represented by the Mel Frequency spectrogram of the audio data in the video data.

The feature extraction unit may calculate the Mel Frequency Cepstrum Coefficients (MFCC) from the voice data, connect the calculated MFCC to the Mel frequency, and extract the voice feature.

The scene presenting unit may add information input via the user interface to the voice feature and the image feature and store the information in the storage device as learning data for the abnormality determining device.

When the number of the learning data stored in the storage device exceeds a predetermined threshold value, the abnormality scene candidate detection unit may cause the abnormality determination device to determine the candidate for the abnormality scene.

When the number of the learning data stored in the storage device is within a predetermined threshold value, the abnormal scene candidate detection unit bypasses the determination by the abnormal device and causes the scene presentation unit to display the abnormal scene. You may be asked to present a candidate.

In the abnormality determination device, the difference between the number of abnormal samples located in the vicinity of the candidate for the abnormal scene and the number of normal samples in the feature space in which the audio feature and the image feature are integrated is within a predetermined threshold value. In this case, the candidate for the abnormal scene may be determined elsewhere.
The abnormality determination device may determine a candidate for the abnormality scene by the k-nearest neighbor method.

From the image features extracted by the feature extraction unit, supervised learning is used to analyze facial emotions contained in the video data, and the analyzed facial emotion features are supplied to the abnormality determination device. An emotion analysis unit may be further provided.

When the emotion analysis unit detects a candidate for the abnormal scene from the video data based on the analyzed emotion of the face, the abnormal scene candidate detection unit detects the abnormal scene based on the audio feature. You may let it run.

One aspect of the information processing system according to the present invention is an information processing system including a server and at least one client device connected to the server via a network, wherein the server acquires video data. Based on the acquisition unit, the feature extraction unit that extracts audio features from the video data acquired by the video acquisition unit and extracts the image features from the video data, and the audio features extracted by the feature extraction unit. Anomalous scene candidate detection unit that detects anomalous scene candidates from the video data and anomalous scene candidates detected by the anomalous scene candidate detection unit are abnormal, normal, based on the audio features and the image features. When the abnormality determination device for determining any of the above and the other and the abnormality determination device determine that the candidate for the abnormality scene belongs to the other, the candidate for the abnormality scene is presented and presented via the user interface. It has a scene presentation unit that accepts input of information to be added to a candidate for an abnormal scene through the user interface, and a transmission unit that transmits the candidate for the abnormal scene to the client device. The client device presents a receiving unit that receives the candidate for the abnormal scene transmitted from the server, a candidate for the abnormal scene received by the receiving unit, and adds the candidate for the presented abnormal scene. It has the user interface that accepts input of information to be input, and a transmission unit that transmits information to be added to the candidate of the abnormal scene that the user interface has received input to the server.

One aspect of the information processing method according to the present invention is an information processing method executed by an information processing apparatus, in which a step of acquiring video data and audio features are extracted from the acquired video data and an image is obtained from the video data. A step of extracting a feature, a step of detecting a candidate for an abnormal scene from the video data based on the audio feature extracted by unsupervised learning, and a step of detecting the candidate for the abnormal scene by the abnormality determiner. , The step of determining any of anomalies, normals, and others based on the audio features and the image features, and when the anomaly determination device determines that the candidate for the anomalous scene belongs to the other, the anomaly. It includes a step of presenting a scene candidate via a user interface and accepting input of information to be added to the presented abnormal scene candidate via the user interface.

One aspect of the information processing program according to the present invention is an information processing program for causing a computer to execute information processing, and the program includes a video acquisition process for acquiring video data and the video acquisition process. A feature extraction process that extracts audio features from the video data acquired by the above and extracts image features from the video data, and a candidate for an abnormal scene from the video data based on the audio features extracted by the feature extraction process. Abnormal scene candidate detection processing for detecting abnormal scenes, and abnormal, normal, and other abnormal scene candidates detected by the abnormal scene candidate detection processing by the abnormality determining device based on the audio features and the image features. When the abnormality determination process for determining any of the above and the abnormality determination device determine that the candidate for the abnormality scene belongs to the other, the candidate for the abnormality scene is presented via the user interface, and the presented abnormality is presented. This is for executing a process including an input / output process for receiving input of information to be added to a scene candidate via the user interface.

According to the present invention, it is possible to detect various abnormalities from an image with high accuracy while reducing the load on the operator.
The above-mentioned object, aspect and effect of the present invention and the above-mentioned object, aspect and effect of the present invention not described above are to be used by those skilled in the art to carry out the following invention by referring to the accompanying drawings and the description of the scope of claims. It can be understood from the form of.

FIG. 1 is a block diagram showing an example of the functional configuration of the abnormality detection device according to the first embodiment of the present invention. FIG. 2 is a flowchart showing an example of a processing procedure of an abnormality scene detection process executed by the abnormality detection device according to the first embodiment. FIG. 3 is a diagram showing an example of audio data segments separated from video data by the feature extraction unit of the abnormality detection device. FIG. 4 is a diagram showing an example of melspectogram features of anomalous scene candidates extracted from voice data by a feature extraction unit of an anomaly detection device. FIG. 5 is a diagram showing an example of a melspectogram feature of a normal scene extracted from audio data by a feature extraction unit of an abnormality detection device. FIG. 6 is a diagram showing an example of a voice feature in which a mel frequency cepstrum and a melspectogram output by the feature extraction unit of the abnormality detection device to the abnormality scene candidate detection unit are connected. FIG. 7 is a diagram illustrating the separation of anomalous feature points of the isolation forest used by the anomaly scene candidate detection unit of the anomaly detection device in unsupervised learning. FIG. 8 is a schematic diagram showing an example of a decision tree of an isolation forest used by an abnormality scene candidate detection unit of an abnormality detection device in unsupervised learning. FIG. 9 is a schematic diagram illustrating an example of the k-nearest neighbor method used by the abnormality determination device of the abnormality detection device to determine an abnormality of an abnormality scene candidate. FIG. 10 is a flowchart showing an example of a processing procedure of an abnormality scene detection process executed by the abnormality detection device according to the modified example of the first embodiment. FIG. 11 is a block diagram showing an example of the functional configuration of the abnormality detection device according to the second embodiment of the present invention. FIG. 12 is a flowchart showing an example of a processing procedure of an abnormality scene detection process executed by the abnormality detection device according to the second embodiment. FIG. 13 is a flowchart showing an example of a processing procedure of the abnormal scene detection process executed by the abnormality detecting device according to the modified example of the second embodiment. FIG. 14 is a schematic diagram showing an example of a facial emotion recognition result recognized by the emotion analysis unit of the abnormality detection device according to the second embodiment from the image data of the video data. FIG. 15 is a block diagram showing an example of the hardware configuration of the abnormality detection device according to each embodiment of the present invention.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. Among the components disclosed below, those having the same function are designated by the same reference numerals, and the description thereof will be omitted. The embodiments disclosed below are examples as means for realizing the present invention, and should be appropriately modified or modified depending on the configuration of the apparatus to which the present invention is applied and various conditions, and the present invention is described below. Is not limited to the embodiment of the above. Moreover, not all combinations of features described in the present embodiment are essential for the means of solving the present invention.

(Embodiment 1)
The abnormality detection device according to the present embodiment extracts the characteristics of the audio data and the image data from the video data, and uses the multimodal characteristics of the audio data and the image data to generate an abnormality scene from the video data in a plurality of stages. Detects semi-automatically.
In the following, the anomaly detection device first detects anomaly scene candidates by unsupervised learning based on the characteristics of audio data extracted from video data streamed in real time, and then the detected anomaly scene candidates. Is determined by the anomaly judge as one of an abnormal scene, a normal scene, and a scene that requires the operator's judgment, and the video data of the candidate of the abnormal scene that is determined to require the operator's judgment is presented, and the abnormal scene by the operator is presented. An example of adding the confirmation input of whether or not to the feature of the video data and accumulating it as learning data for the abnormality determining device will be described.

However, this embodiment is not limited to this. For example, the anomaly detection device may detect an abnormal scene ex post facto from the recorded video data. Further, for example, the abnormal scene detection is variably controlled according to the number of accumulated learning data, and all of the detected abnormal scene candidate video data is presented to the operator by bypassing the abnormality judge. Alternatively, the abnormality determination device may automatically detect the abnormality scene based on the audio and image characteristics of the video data of the detected abnormality scene candidate. In the latter case, the threshold value for determining an abnormality may be set relatively low, and only the abnormal scene near the threshold value may be appropriately confirmed and presented to the operator.

<Functional configuration of anomaly detection device>
FIG. 1 is a block diagram showing an example of the functional configuration of the abnormality detection device 1 according to the present embodiment.
The abnormality detection device 1 shown in FIG. 1 includes a data acquisition unit 11, a feature extraction unit 12, an abnormality scene candidate detection unit 13, an abnormality determination device 14, and a scene presentation unit 15.
The abnormality detection device 1 may be communicably connected to a client device 3 composed of a PC (Personal Computer) or the like via a network. In this case, the anomaly detection device 1 is mounted on the server, and the client device 3 may provide a user interface for the anomaly detection device 1 to input / output information to / from the outside, and the scene of the anomaly detection device 1 may be provided. A part or all of each component 11 to 15 including the presentation unit 15 may be provided.

The data acquisition unit 11 acquires video data streamed in real time and supplies the acquired video data to the feature extraction unit 12. The video data is moving image data including audio data and visual data, but the data acquisition unit 11 acquires still image data including audio data in place of the moving image data. It may be supplied to the feature extraction unit 12.
The data acquisition unit 11 may acquire video data pre-recorded in a non-volatile storage device such as an HDD of the abnormality detection device 1 instead of the video data to be streamed, and the recorded video data may be used as an opposite device. May be received via communication I / F.
The data acquisition unit 11 also receives input of various parameters necessary for executing the abnormal scene detection process in the abnormality detection device 1. The data acquisition unit 11 may accept input of various parameters via the user interface of the client device 3 which is communicably connected to the abnormality detection device 1.

The feature extraction unit 12 separates audio data from the video data supplied from the data acquisition unit 11, and extracts audio features from the separated audio data.
The feature extraction unit 12 also separates the image data from the video data supplied from the data acquisition unit 11 and extracts the image features from the separated image data.
The feature extraction unit 12 supplies the extracted audio features and image features to the abnormality scene candidate detection unit 13 together with the video data.

The abnormal scene candidate detection unit 13 detects an abnormal scene candidate from the video data based on the audio feature supplied from the feature extraction unit 12, and supplies the detected abnormal scene candidate to the abnormality determining device 14. The abnormal scene candidate detection unit 13 may also supply the detected abnormal scene candidate to the scene presentation unit 15 by bypassing the abnormality determining device 14.

The abnormal scene includes, but is not limited to, a scene unsuitable for family viewing (Non-Family-Safe: NFS), such as a violent scene or a scene not intended for children. Abnormal scenes should not be finally distributed from the audio and images of the video data by the scene or content that is stipulated in the rules or regulations of each video distribution service that it should not be distributed via the video distribution service. It shall include a wide range of scenes or contents determined manually.
Details of the feature extraction process executed by the feature extraction unit 12 and the abnormal scene candidate detection process executed by the abnormal scene candidate detection unit 13 will be described later with reference to FIGS. 3 to 8.

In the present embodiment, the abnormal scene candidate detection unit 13 detects candidates for abnormal scenes by classifying voice features by unsupervised learning. In the video data to be streamed, the frequency of filing anomalous scenes is small, and the criteria for whether or not to make an anomalous scene vary from video distribution service to video distribution service. When is changed, it is difficult to prepare appropriate teacher data in advance, and it is difficult to realize highly accurate classification by supervised learning. In the present embodiment, by classifying audio features from only audio data from audio data by unsupervised learning, it is possible to detect candidates for abnormal scenes with high accuracy and low load even with a small number of samples. ..

The abnormality determination device 14 inputs the candidate of the abnormal scene supplied from the abnormal scene candidate detection unit 13, and inputs the input video data of the candidate of the abnormal scene to any of the abnormal scene, the normal scene, and the scene requiring the judgment of the operator. Classify into crabs. The abnormality determination device 14 stores the abnormal scene candidates classified into either the abnormal scene or the normal scene among the classification results of the abnormal scene candidates in the learning data DB (database) 2 with the classification results added. To go. Further, the abnormality determining device 14 supplies the candidate of the abnormal scene classified as the scene requiring the operator's judgment from the candidates of the abnormal scene to the scene presentation unit 15.

In the present embodiment, the abnormality determining device 14 uses a feature space in which the audio features and image features of the video data extracted by the feature extraction unit 12 are integrated to select candidates for abnormal scenes input by supervised learning. , Abnormal scenes, normal scenes, and scenes that require the judgment of the operator. The abnormality determining device 14 may execute learning using the classification result of the candidate of the abnormal scene accumulated in the learning data DB 2 as teacher data.
Details of the abnormality scene determination process executed by the abnormality determination device 14 will be described later with reference to FIG.

The scene presenting unit 15 presents a candidate for an abnormal scene, which is supplied from the abnormality determining device 14 and is classified as a scene requiring an operator's judgment, to the outside via a display device or the like, and accepts an operator's confirmation input. The abnormality detection device 1 may also present the abnormality scene candidate supplied from the abnormality scene candidate detection unit 13 to the outside and accept the confirmation input of the operator.
The abnormality detection device 1 may use the display device or the like of its own device as a user interface, but presents an abnormality scene candidate to the outside via the user interface of the client device 3 communicably connected to the abnormality detection device 1. , Or the operator's confirmation input may be accepted.
In this case, the abnormality detection device 1 further transmits a transmission / reception unit for transmitting the abnormality scene candidate supplied from the abnormality scene candidate detection unit 13 to the client device 3 and receiving the operator's confirmation input transmitted from the client device 3. You may be prepared. The client device 3 includes a transmission / reception unit that receives an abnormality scene candidate transmitted from the abnormality detection device 1 and transmits an operator confirmation input for the abnormality scene candidate presented via the user interface to the abnormality detection device 1. You can do it.

The operator confirms that the presented abnormal scene candidate is either an abnormal scene or a normal scene by comparing the image of the video data of the abnormal scene candidate presented by the scene presentation unit 15 with the sound. , The confirmation result is input to the scene presentation unit 15. The operator can take predetermined measures for the candidate of the abnormal scene confirmed to be the abnormal scene. For example, the confirmed abnormal scene may be deleted from the video data to be distributed, the distribution of the video data may be stopped, or the account of the user who distributes the video data may be suspended.
The scene presentation unit 15 adds a confirmation result (annotation of an abnormal scene or a normal scene) confirmed and input by the operator to the presented audio feature and image feature of the candidate of the abnormal scene, and adds the confirmation result (annotation of the abnormal scene or the normal scene) to the learning data DB 2 as learning data. Store.

As a specific example, it is assumed that the abnormal scene candidate detection unit 13 detects a sound like shooting a gun from the sound features extracted from the video data, and detects a scene including the sound as a candidate for the abnormal scene. In this case, the operator may check the image of the candidate of the abnormal scene to confirm whether it is an abnormal scene or a normal scene.
For example, if the image of a candidate for an abnormal scene contains a gun or other violent or cruel object, it can be confirmed as an abnormal scene, while if it contains an object such as outdoor fireworks, it is a normal scene. Can be confirmed.

As described above, in the present embodiment, the abnormal scene is semi-automatically detected in a plurality of stages by using the multimodal information of the audio and the image of the video data. Specifically, the candidate of the abnormal scene is automatically detected from the sound of the video data, the candidate of the detected abnormal scene is presented to the operator, and it is confirmed whether the abnormal scene or the normal scene is from the image of the candidate of the abnormal scene. I'm letting you. As a result, the load on the operator in monitoring the delivered video is significantly reduced.

<Processing procedure for abnormal scene detection processing>
FIG. 2 is a flowchart showing an example of a processing procedure of an abnormality scene detection process executed by the abnormality detection device 1 according to the present embodiment.
Each step in FIG. 2 is realized by the CPU reading and executing a program stored in a storage device such as an HDD of the abnormality detection device 1. Further, at least a part of the flowchart shown in FIG. 2 may be realized by hardware. When it is realized by hardware, for example, by using a predetermined compiler, a dedicated circuit may be automatically generated on FPGA (Field Programmable Gate Array) from a program for realizing each step. Further, a Gate Array circuit may be formed in the same manner as the FPGA and realized as hardware. Further, it may be realized by ASIC (Application Specific Integrated Circuit).

In S1, the feature extraction unit 12 of the abnormality detection device 1 separates the video data supplied from the data acquisition unit 11 into audio data and image data, and extracts the audio features and the image features, respectively.
In FIG. 3, the feature extraction unit 12 is separated from the video data (for example, a multimedia format such as mp4 or m3u8) supplied from the data acquisition unit 11 as a preprocessing for feature extraction, and is segmented in units of one second, for example. An example of the audio signal waveform of the generated audio data (for example, wav format) is shown. In FIG. 3, the vertical axis represents the amplitude of voice and the horizontal axis represents time.
The feature extraction unit 12 may normalize the separated audio data by upsampling or the like as appropriate according to a sound source or the like that can be assumed in the delivered video data.

In the present embodiment, the feature extraction unit 12 may extract voice features from the voice data shown in FIG. 3 as voice features expressed by, for example, a mel frequency spectrogram.
Spectrogram refers to the result of calculating the frequency spectrum by passing an audio signal through a window function, and has three dimensions with the time, frequency, and strength (amplitude) of the signal components as the X-axis, Y-axis, and Z-axis, respectively. It is represented by a graph. The spectrogram corresponds to a so-called voiceprint in which the spectra of each voice data segment (frame) obtained by extracting the frequency component and the amplitude component of the voice signal by, for example, Fourier transform are arranged along the time axis. A mel spectrogram is a spectrogram converted by a mel scale for weighting in consideration of human pitch perception (frequency perception characteristic).

FIG. 4 is an example of an audio feature represented by a mel spectrogram extracted from audio data separated from video data by the feature extraction unit 12, and is detected as an abnormal scene candidate by the abnormal scene candidate detection unit 13. Is shown. In FIGS. 4 and 5, the X-axis indicates time, the Y-axis indicates frequency, and the Z-axis indicates amplitude, that is, the intensity of the audio signal. Further, in FIGS. 4 and 5, cells with higher signal strength are shown with a lighter pattern, and cells with lower signal strength are shown with a darker pattern.
The spectrogram shown in FIG. 4 shows a pattern in which the volume is loud, the signal intensity distribution has a peak, and the audio signal is attenuated in a short time. FIG. 4 shows an example of a spectrogram of shooting a gun, and for example, a person's cry or the sound of hitting something is considered to show the same or similar pattern.

On the other hand, FIG. 5 shows an audio feature expressed by a melspectogram extracted from an audio data segment separated from the video data by the feature extraction unit 12, and is determined to be a normal scene by the abnormal scene candidate detection unit 13 (abnormality). An example of audio features (not detected as scene candidates) is shown. The spectrogram shown in FIG. 5 shows a pattern in which the volume is low or medium and the signal intensity distribution is uniform on the time axis.
The feature extraction unit 12 separates the foreground voice (for example, a person's utterance voice, screaming voice, etc.) and the background voice (music, crowd sound, etc.) from the voice data, and uses the spectrogram of either voice as the voice feature. It may be supplied to the abnormal scene candidate detection unit 13. In this case, for example, a sound that temporarily appears on the time axis and is not repeated can be separated as a foreground sound.

In the present embodiment, the feature extraction unit 12 further calculates a mel frequency cepstrum coefficient (MFCC) from the voice data, and concatenates the calculated MFCC with the mel spectrogram shown in FIGS. 4 to 5. Audio features may be extracted.
The cepstrum is a spectrum obtained by multiplying an amplitude spectrum obtained by Fourier transforming an audio signal by a logarithm to obtain a logarithmic spectrum, and then applying the Fourier transform again to the logarithmic spectrum. By obtaining the cepstrum of the logarithmic spectrum (logarithmic cepstrum), it is possible to separate the sound source component that fluctuates with a high period from the convoluted vocal tract characteristic component.

The low-order components of the log cepstrum represent the spectral envelope of speech (frequency characteristics derived from the vocal tract components). It is possible to remove pitch components with large individual differences and extract only the acoustic characteristics of the vocal tract, which are important for phonological identification. The feature amount given by applying the Mel scale to the low-order component of the logarithmic cepstrum in consideration of the human frequency perception characteristic is MFCC.

Specifically, the amplitude spectrum is applied to a plurality of filter banks at equal intervals on the Mel scale, the spectral components of each band are extracted, the sum of the amplitude spectra of each band is summed, and the amplitude spectrum is compressed into a multidimensional amplitude spectrum. , The log of this compressed amplitude spectrum is taken to obtain the log amplitude spectrum.
The Mel frequency spectrum obtained in this way (a logarithmic amplitude spectrum compressed by the Mel scale) is converted into a Mel frequency cepstrum by performing a Fourier transform (for example, Discrete Fourier Transform (DFT)). The MFCC can be obtained by taking out the low-order component (voiceway component of the spectrum) of cepstrum and performing normalization processing as necessary.

FIG. 6 shows an example of a voice feature in which an MFCC 61 obtained by slicing in a unit time (for example, 1 second) and taking an average value and a melspectogram 62 having an average amplitude on the time axis are connected. show.
By supplying the voice feature as shown in FIG. 6 to the abnormal scene candidate detection unit 13 to detect the candidate of the abnormal scene, the information of the frequency component of the voice data, particularly the frequency component important for human hearing is not lost. , Audio features can be properly compressed.

The audio features extracted from the audio data of the video data by the feature extraction unit 12 are not limited to the expressions shown in FIG. 6, and the feature extraction unit 12 extracts the audio features by any method and expression other than the above. You can do it.
The feature extraction unit 12 may also extract image features from all or part of the image data separated from the video data by using, for example, a convolutional neural network (CNN) or the like. However, the method for extracting image features from image data by the feature extraction unit 12 is not limited to CNN, and any method can be used.

Returning to FIG. 2, in S2, the abnormality scene detection unit 13 of the abnormality detection device 1 classifies by unsupervised learning based on the audio characteristics of the audio data separated from the video data supplied from the feature extraction unit 12. Detects candidates for abnormal scenes.
The abnormal scene detection unit 13 separates audio features having abnormal values in the feature space of audio features by, for example, an isolation forest (IF), and the video scene corresponding to the separated abnormal audio features. Is detected as a candidate for an abnormal scene.

Isolation forest is one of unsupervised learning that directly separates features with outliers without modeling (profiling) features with normal values to generate a normal model. Since it is a high-speed algorithm and consumes less memory, it is suitable for monitoring video delivered in real time, and since it is not necessary to model a normal model, it is unlikely that the accuracy will decrease even with a small number of samplings.

FIG. 7 is a schematic diagram illustrating an algorithm in which the isolation forest separates features having outliers in the feature space. The isolation forest creates repeating partitions (partitions) as shown by the dashed line in FIG. 7 until each feature point placed on the feature space can be separated from all other feature points. With reference to FIG. 7, the leftmost feature point and the rightmost feature point each require fewer partitions than the feature points located near the center. In FIG. 7, since the leftmost and rightmost feature points can be separated from all other feature points in one partition, they can be detected as feature points having abnormal values.

FIG. 8 is a conceptual diagram in which the iterative process of partition generation in FIG. 7 is represented by a binary tree structure (isolation tree: Isolation Tree). The number of partitions in FIG. 7 can be expressed by the path length from the root node to the terminal node of the tree structure in FIG.
The abnormality scene candidate detection unit 13 calculates the abnormality score of each feature point by the following equation 1 based on the path length of the feature points of each voice feature.

（式１）

(Equation 1)

Here, E (h (x)) of the right-hand side exponent part is the average path length, and c (n) is a normalization factor depending on the number of instances n of the data set. The abnormal score S (x, n) of each feature point approaches 1 as the average path length is shorter, and approaches 0 as the average path length is longer.

With reference to FIG. 8, the bar on the left corresponds to the value of the anomaly score from 0 to 1 from the bottom to the top. The abnormal scene detection unit 13 compares the threshold value θ of the abnormal score between 0 and 1 with the abnormal score S (x, n) of each feature point, and the abnormal score S (x, n) sets the threshold value θ. The voice features of the feature points that exceed are determined as abnormal values (outliers), while the voice features of the feature points whose abnormal score S (x, n) is within the threshold value θ are determined as normal values.

The abnormal scene detection unit 13 detects a video scene including an audio feature for which an abnormal score exceeding the threshold value θ is calculated and a corresponding image feature as a candidate for the abnormal scene.
The unsupervised learning algorithm used by the abnormal scene candidate detection unit 13 to detect an abnormal scene candidate is not limited to the isolation forest. The abnormal scene candidate detection unit 13 detects a candidate for an abnormal scene by calculating a reconstruction score of a voice feature using a variational autoencoder (VAE) instead of the isolation forest. You may also use any other unsupervised learning.

Returning to FIG. 2, in S3, the abnormality scene candidate detection unit 13 of the abnormality detection device 1 compares the number of learning data stored in the learning data DB 2 with a predetermined threshold value. When the number of learning data stored in the learning data DB 2 exceeds a predetermined threshold value (S3: Y), the process proceeds to S4, and the abnormal scene candidate detection unit 13 sends the detected abnormal scene candidates to the abnormality determining device 14. Supply. On the other hand, when the number of learning data stored in the learning data DB 2 is within a predetermined threshold value (S3: N), the process of the abnormality determination device (S4 to S6) is bypassed and the process proceeds to S7.

In the present embodiment, when the abnormality determining device 14 does not sufficiently accumulate the learning data for determining the abnormal scene in the learning data DB 2, it is determined that the accuracy (reliability) of the abnormal scene determination by the abnormality determining device 14 is not sufficient. , Bypassing the processing in the abnormality determining device 14. Then, the scene presentation unit 15 directly presents the candidate of the abnormal scene detected by the abnormal scene candidate detection unit 13 to the operator, and accepts the confirmation input of the operator. As a result, while the number of samples of learning data is small, the processing load of machine learning execution by the abnormality determining device 14 can be reduced by always requesting the operator's confirmation judgment from the candidate of the abnormality scene.

As described above, in the present embodiment, the control of how to determine the abnormal scene from the detected abnormal scene candidates is autonomously optimized. Specifically, while the number of samples of learning data in the abnormality determining device 14 is small, the operator gives priority to the determination of the abnormal scene and prevents the accuracy of the determination of the abnormal scene from deteriorating. On the other hand, when the number of samples of learning data to the abnormality determining device 14 exceeds a predetermined threshold value, only the candidate of the abnormal scene that the abnormality determining device 14 could not classify as either an abnormal scene or a normal scene is used as an operator. By presenting and requesting confirmation input, the load on the operator can be further reduced.

In S4, the abnormality determination device 14 of the abnormality detection device 1 classifies the abnormality scene candidates supplied from the abnormality scene candidate detection unit 13 into one of a normal scene, an abnormality scene, and a scene requiring the operator's judgment. Therefore, the abnormality of the candidate of the abnormal scene is determined, and the determination result is stored in the learning data DB 2.
Specifically, the abnormality determiner 14 uses the k-nearest neighbor algorithm (k-NN), for example, as supervised learning, on a feature space in which audio features and image features are integrated. By searching for the nearest neighbor solution, the candidate of the abnormal scene supplied from the abnormal scene candidate detection unit 13 is classified.

FIG. 9 is a conceptual diagram illustrating an example of a classification algorithm based on the k-nearest neighbor method.
With reference to FIG. 9, a group of objects indicated by circle marks is arranged in the feature space. Each object is represented by a position vector in a multidimensional feature space, and the correct classification class is known. The star mark in the center of the concentric circle is the position vector of the classification target whose classification class is unknown, and in the present embodiment, it is the position vector of the abnormal scene candidate to be determined. In the k-nearest neighbor method, the distance between the new position vector indicated by the star mark and the existing position vector group indicated by the circle mark is calculated, and k nearest neighbor samples are selected. The distance between the position vectors may be calculated as the Euclidean distance, but may also be calculated as another distance such as the Manhattan distance.

With reference to FIG. 9, when k = 3, since two dark circle marks and one light circle mark are arranged as the three nearest neighbors in the inner concentric circle, the determination target is obtained. The position vector of is classified into the class of dark circle marks. On the other hand, when k = 6, in the outer concentric circle, four light circle marks are arranged for two dark circle marks as six objects in the nearest neighbor, so that the position vector to be determined is the position vector to be determined. Classified as a light circle mark class. The class may be determined by weighting the distance from the new position vector among the k nearest neighbor objects.

The anomaly determination device 14 maps a candidate for an abnormal scene whose correct class is unknown on a feature space in which audio features and image features of video data are integrated as a position vector, and k objects (samples) in the nearest vicinity. ), By comparing the number of samples classified as abnormal scenes with the number of samples classified as normal scenes, the candidates for the abnormal scenes to be determined can be determined by the abnormal scene, the normal scene, and the operator. Classify into one of the required scenes.

Specifically, the anomaly determination device 14 is a feature space in which audio features and image features are integrated, and the number of samples classified as anomalous scenes among k nearest neighbor samples for anomalous scene candidates. However, when the number of samples is sufficiently larger than the number of samples classified as normal scenes, the candidate of the abnormal scene to be determined is determined to be an abnormal scene. In the feature space, the abnormality determination unit 14 also has a sufficiently larger number of samples classified as normal scenes among the k nearest neighbor samples with respect to the candidate of abnormal scenes, which is sufficiently larger than the number of samples classified as abnormal scenes. In this case, the candidate of the abnormal scene to be determined is determined to be a normal scene.

On the other hand, in the abnormality determination device 14, the difference between the number of samples classified as abnormal scenes and the number of samples classified as normal scenes among the k nearest neighbor samples with respect to the candidate of abnormal scenes on the feature space. Is small and is within a predetermined threshold value, it is determined that the candidate of the abnormal scene to be determined is a scene that requires the operator's judgment.
Alternatively, the abnormality determining device 14 sets the candidate of the abnormal scene to be determined as either an abnormal scene or a normal scene depending on the size of the number of samples of k nearest abnormal scenes and the number of samples of the normal scene. It may be classified automatically. In particular, when a sufficient number of sample numbers of training data are accumulated in the training data DB 2, it is possible to eliminate the need for operator intervention in detecting abnormal scenes.
The algorithm for the abnormality determination device 14 to determine the abnormality of the candidate of the abnormality scene is not limited to the above-mentioned k-nearest neighbor method. The anomaly determination device 14 may determine an abnormality scene using, for example, another supervised learning machine learning algorithm including a neural network such as CNN, a support vector machine (SVM), or the like.

Returning to FIG. 2, when the abnormality determining device 14 determines in S4 that the candidate for the abnormal scene is an abnormal scene, the process proceeds to S5, the distribution of the video content including the abnormal scene is stopped, and the account of the distribution source of the video content. Processing such as deletion of the abnormal scene or deletion of the determined abnormal scene is executed to end the processing.
When the abnormality determining device 14 determines that the candidate for the abnormal scene is a normal scene, the process proceeds to S6, continues distribution of the video content including the candidate for the abnormal scene, and ends the process. On the other hand, if the abnormality determination device 14 determines that the candidate for the abnormality scene is a scene that requires the operator's judgment, the process proceeds to S7.

In S7, the scene presentation unit 15 of the abnormality detection device 1 is a candidate for an abnormality scene supplied from the abnormality scene candidate detection unit 13, or a candidate for an abnormality scene determined by the abnormality determination device 14 to be a scene that requires the operator's judgment. The video (audio data and image data) is presented to the operator.
In S8, the scene presentation unit 15 of the abnormality detection device 1 accepts the input of tagging either the abnormal scene or the normal scene as the operator's confirmation input for the video of the candidate of the abnormal scene presented in S7.
In S9, the operator determines that the candidate for the abnormal scene tagged as the abnormal scene is processed for the abnormal scene, that is, the distribution of the video content including the abnormal scene is stopped, the account of the distribution source of the video content is deleted, or the determination is made. Execute processing such as deleting abnormal scenes. On the other hand, the operator causes the video distribution to continue for the candidate of the abnormal scene tagged as the normal scene without executing the processing for the abnormal scene.

In S10, the scene presentation unit 15 of the abnormality detection device 1 uses the tag (label) of the operator's abnormal scene or normal scene input to the scene presentation unit 15 in S8 as a voice feature and an image of the presented abnormal scene candidate. It is stored in the learning data DB 2 as learning data which is a determination result of an abnormal scene by the operator in association with the feature. As a result, the learning data DB 2 is updated with the new learning data.

In S11, the abnormality determination device 14 of the abnormality detection device 1 executes re-learning based on the learning data DB 2 updated in S10. Whether or not the abnormality determination device 14 needs to be relearned depends on the abnormality determination algorithm of the abnormality determination device 14. For example, as described above, when the abnormality determination device 14 uses the k-nearest neighbor method, k-nearest neighbor samples are selected with reference to the learning data DB 2 each time an abnormality scene is determined, so that S11 It is not necessary to execute re-learning in S10, it is sufficient to update the learning data DB2 in S10, and the processing of S11 may be omitted.

On the other hand, when the abnormality determination device 14 uses a neural network, SVM, or the like, it is necessary to relearn the abnormality determination device 14 at any timing to update the parameters of the abnormality determination device 14.
As for the timing of re-learning, the abnormality determination device 14 may be re-learned every time the learning data DB 2 is updated, or the abnormality determination device 14 may be re-learned every time the learning data DB 2 is updated a predetermined number of times. good. Alternatively, the abnormality determination device 14 can be re-learned immediately before the abnormality determination device 14 is used in S4, but when an abnormality scene is to be detected in real time from the video delivered in real time, the re-learning execution is performed. It should be taken into consideration that the real-time property may be reduced.

<Modification example>
FIG. 10 is a diagram showing a modified example of the abnormality scene detection process executed by the abnormality detection device 1 according to the present embodiment.
As a modified example, the abnormal scene candidate detection unit 13 omits the processing of S3 and uniformly detects the abnormal scene regardless of the number of learning data stored in the learning data DB 2. Can be supplied to the abnormality determination device 14. This makes it possible to further reduce the load of the operator's confirmation processing of abnormal scene candidates in video monitoring.

As described above, according to the present embodiment, the anomaly detection device extracts audio features and image features in the acquired video data, and classifies the extracted audio features by unsupervised learning. Detects abnormal scene candidates from the data. The anomaly detector also comprises an anomaly determiner that determines any of the detected anomalous scene candidates as anomalous, normal, or otherwise based on the audio and image features of the video data. When it is determined that the candidate for the abnormal scene belongs to others, the candidate for the abnormal scene is presented via the user interface, and the user interface is used to input information to be added to the presented candidate for the abnormal scene. Accept through.

As a result, in the first stage, candidates for abnormal scenes are detected as the first stage at high speed and low load by unsupervised learning based on the audio characteristics in the video data, and in the second stage, the audio characteristics and the audio characteristics in the video data are detected. The judgment of the abnormal scene by the abnormality judgment device based on the image feature and the judgment of the abnormal scene by the operator's visual observation based on the video presentation are used in a complementary manner.
Therefore, it is possible to detect various abnormalities from the video with high accuracy while reducing the load on the operator.
As a result, it is possible to realize high-speed and high-precision multimodal detection of abnormal scenes that are delivered in real time and sufficiently follow the video data that may include various abnormal scenes.

(Embodiment 2)
Hereinafter, the second embodiment will be described in detail only with reference to FIGS. 11 to 14 and different from the first embodiment.
In this embodiment, in addition to the first embodiment described above, the emotions of a person who is an object in the video are analyzed from the image features of the video data, and the emotion analysis results are used to detect candidates for abnormal scenes and to detect abnormal scenes. Used for judgment.

FIG. 11 is a block diagram showing an example of the functional configuration of the abnormality detection device 1 according to the present embodiment.
In the block diagram of FIG. 11, in addition to the functional configuration of the abnormality detection device 1 of the first embodiment shown in FIG. 1, an emotion analysis unit 16 is provided.
In FIG. 11, the functional configurations of the data acquisition unit 11, the feature extraction unit 12, the abnormality scene candidate 13, the abnormality determination device 14, and the scene presentation unit 15 are the same as those of the corresponding units shown in FIG.
With reference to FIG. 11, the feature extraction unit 12 supplies the image features extracted from the image data in the video data to the emotion analysis unit 16.

The emotion analysis unit 16 analyzes the emotion of the human face, which is an object in the image, based on the image features of the video data supplied from the feature extraction unit 12.
The emotion analysis unit 16 may estimate the emotion of the person's face in the image by analyzing the face of the person in the image by using, for example, supervised learning such as CNN. The emotions of a person's face estimated from the image of the person's face may include, for example, anger, disgust, fear, happiness, sadness, surprise, and other (neutral) emotions.
The emotion analysis unit 16 may also determine the emotion of a person's face in the target image based on the estimated average reliability of emotions calculated between a plurality of image frames adjacent in time. ..

The emotion analysis unit 16 may further analyze the movement of a person's body or limbs in an image, an object held by the person (for example, a microphone, a musical instrument, etc.), or a background (for example, indoor or outdoor). The feature extraction unit 12 may extract the features of the target object to be analyzed by the emotion analysis unit 16 and supply them to the emotion analysis unit 16.

In the present embodiment, the emotions of the human face estimated by the emotion analysis unit 16 from the image features of the image of the human face are detected by the abnormality scene candidate detection unit 13 and the abnormality determination device 14. Complements the anomaly determination process of candidate abnormal scenes executed by.
Specifically, the emotion analysis unit 16 estimates the context of the image from the emotion of the human face estimated from the image features of the human face, and detects the abnormal scene candidate with respect to the abnormal scene candidate detection unit 13. A processing queue (trigger) may be given. For example, when the emotion analysis unit 16 analyzes an image of a person's face and detects, for example, anger, fear, surprise, etc. as the emotion of the person's face, the image including the image may be an abnormal scene. Therefore, the emotion analysis unit 16 may give a queue to the abnormal scene analysis unit 13 to execute a process of detecting a candidate for an abnormal scene from the audio characteristics of the video.

The emotion analysis unit 16 also supplies the emotional features of the human face estimated from the image features of the human face to the abnormality determination device 14, and the abnormality determination unit 14 supplies the human face from the emotion analysis unit 16. The emotional features of the above may be integrated into the feature space, and the candidate for the abnormal scene may be abnormally determined by the k-near method. For example, the abnormality determining device 14 may use features such as anger, fear, and surprise as emotions on a person's face as positive factors for determining an abnormal scene.
The emotion analysis unit 16 further supplies the analysis result of the emotion of the human face estimated from the image features of the human face to the scene presentation unit 15, and the scene presentation unit 15 is supplied from the emotion analysis unit 16. The analysis result of the emotion of the face may be displayed together with, for example, a superimposed display or a separate window display in the presented image.

FIG. 12 is a flowchart showing an example of a processing procedure of an abnormality scene detection process executed by the abnormality detection device 1 according to the second embodiment.
In the flowchart of FIG. 12, the process of S12 is added between S1 and S2 with respect to the abnormal scene detection process executed by the abnormality detection device 1 of the first embodiment shown in FIG.
The processing of S1 is the same as that of the first embodiment shown in FIG. That is, as in the first embodiment, the feature extraction unit 12 of the abnormality detection device 1 extracts audio features and image features from the video data supplied by the data acquisition unit 11, respectively.

In S1, when the feature extraction unit 12 of the abnormality detection device 1 extracts the audio feature and the image feature from the video data, the process proceeds to S12.
In S12, the emotion analysis unit 16 of the abnormality detection device 1 detects a candidate for an abnormal scene from the image features extracted by the feature extraction unit 12. Specifically, the emotion analysis unit 16 estimates a person's emotions from the image features of the person's face in the image, and when, for example, emotions such as anger, fear, and surprise are estimated, the image is used. The included video scene may be detected as a candidate for an abnormal scene.
When the emotion analysis unit 16 detects a candidate for an abnormal scene from the image feature, it queues up for the detection process of the abnormal scene candidate based on the audio feature of the video executed by the abnormal scene candidate detection unit 13 executed in the subsequent S2. Trigger) is given.

Following S12, in S2, the abnormal scene candidate detection unit 13 of the abnormality detection device 1 is supplied from the emotion analysis unit 16 when the emotion analysis unit 16 detects an abnormal scene candidate from the image feature and gives a trigger. By classifying the voice features corresponding to the abnormal scene candidates using unsupervised learning, the abnormal scene candidates are detected.

Alternatively, the abnormal scene candidate detection unit 13 constantly detects an abnormal scene candidate from the audio characteristics of the video data regardless of whether or not a trigger is given from the emotion analysis unit 16, and the emotion analysis unit 16 displays an image. When a trigger for detecting an abnormal scene candidate based on a feature is given, it may be confirmed from the detected voice feature of the abnormal scene candidate whether or not it should be supplied to the abnormality detector 14 as a candidate for the abnormal scene.
The processing from S2 to S11 is the same as that of the first embodiment shown in FIG.
As in FIG. 10, the abnormality detection device 1 according to the present embodiment omits the determination and branch processing of S3, and uses the abnormality determination device 14 of S4 regardless of the number of learning data stored in the learning data DB 2. You may proceed to the determination process of the abnormal scene.

FIG. 13 is a flowchart showing an example of a processing procedure of a modified example of the abnormality scene detection processing executed by the abnormality detection device 1 according to the second embodiment.
In the flowchart of FIG. 13, the process of S13 is added between S2 and S3 with respect to the abnormal scene detection process executed by the abnormality detection device 1 of the first embodiment shown in FIG.
The processing of S1 and S2 is the same as that of the first embodiment shown in FIG. That is, as in the first embodiment, the feature extraction unit 12 of the abnormality detection device 1 extracts audio features and image features from the video data supplied by the data acquisition unit 11, and the abnormality scene candidate detection unit 13 extracts the features. A candidate for an abnormal scene is detected based on the audio characteristics of the video data supplied from the extraction unit 12.

Next, in S13, the emotion analysis unit 16 of the abnormality detection device 1 determines the human face from the image features of the human face included in the image among the image features of the image data supplied from the feature extraction unit 12. Estimate the emotions of.
The processing from S3 to S11 is the same as that of the first embodiment shown in FIG. 2, but in S4, the abnormality determining device 14 voices the emotional features of the human face in the image supplied from the emotion analysis unit 16. And may be integrated into the feature space of the image. Further, in S7, the scene presentation unit 15 may present the analysis result of the emotion analysis unit 16 together with the image of the candidate of the abnormal scene.
In addition, in FIG. 13, S2 and S13 may be executed in parallel, and S13 may be executed before S2 in chronological order.
Further, the abnormality detection device 1 according to the present embodiment omits the determination and branching processing of S3 as in FIG. 10, and uses the abnormality determination device 14 of S4 regardless of the number of learning data stored in the learning data DB 2. You may proceed to the determination process of the abnormal scene.

FIG. 14 is a diagram showing an output example of an emotion analysis result presented by the scene presentation unit 15 after the emotion analysis unit 16 of the abnormality detection device 1 analyzes an image of video data.
With reference to FIG. 14, a bounding box 131 is displayed around the human face in the image, indicating that the object has been detected as a human face object. The reliability of emotions estimated from the face of the person in the bounding box 13 is displayed in the upper left of the output window.
In the example of FIG. 14, the reliability of anger is calculated to be the highest at 37.45%, but the facial expression of the person surrounded by the bounding box 131 is assumed to be neutral without showing anger. .. In this case, the operator presented with the image of FIG. 14 visually confirms the facial expression of the person in the presented image, and presents the confirmation result that the scene is not an abnormal scene (that is, a normal scene). It can be input to the unit 15. Alternatively, the emotion analysis unit 16 sets a predetermined threshold value for the reliability score, and when the anger reliability score is equal to or less than the threshold value, the emotion analysis unit 16 may not detect it as a candidate for an abnormal scene.

As described above, according to the present embodiment, the abnormality determination device of the abnormality detection device is based on multimodal information of both audio characteristics and image characteristics of video data, particularly emotional characteristics of a human face. , It suffices to execute the determination of the abnormal scene for the candidate of the abnormal scene determined to have a high probability of being an abnormal scene. Therefore, even when the number of samples of the training data is small, the abnormality determination process can be executed with high accuracy and low load.
Similarly, according to the present embodiment, the scene presentation unit of the abnormality detection device is based on multimodal information of both audio characteristics of video data and image characteristics, particularly emotional characteristics of a human face, to obtain an abnormal scene. It suffices to present to the operator a candidate for an abnormal scene that is determined to be highly probable. Therefore, the load on the operator in confirming the abnormal scene is further reduced.

<Hardware configuration of anomaly detection device>
FIG. 15 is a diagram showing a non-limiting example of the hardware configuration of the abnormality detection device 1 according to the present embodiment.
The anomaly detection device 1 according to the present embodiment can be implemented on a single or a plurality of any computer, mobile device, or any other processing platform.
Although an example in which the abnormality detection device 1 is mounted on a single computer is shown with reference to FIG. 15, the abnormality detection device 1 according to the present embodiment is mounted on a computer system including a plurality of computers. You can do it. A plurality of computers may be connected to each other so as to be able to communicate with each other by a wired or wireless network.

As shown in FIG. 15, the abnormality detection device 1 may include a CPU 21, a ROM 22, a RAM 23, an HDD 24, an input unit 25, a display unit 26, a communication I / F 27, and a system bus 28. The anomaly detection device 1 may also include an external memory. PC3 may also have the same configuration as in FIG.
The CPU (Central Processing Unit) 21 comprehensively controls the operation of the abnormality detection device 1, and controls each component (22 to 27) via the system bus 28, which is a data transmission path.
The abnormality detection device 1 may also include a GPU (Graphics Processing Unit). The GPU has a higher calculation function than the CPU 21, and by operating a plurality or a large number of GPUs in parallel, it provides higher processing performance, particularly for a video processing application using machine learning as in the present embodiment. do. The GPU typically includes a processor and shared memory. Each processor acquires data from a high-speed shared memory and executes a common program to execute a large amount of the same kind of calculation processing at high speed.

The ROM (Read Only Memory) 22 is a non-volatile memory for storing a control program or the like necessary for the CPU 21 to execute a process. The program may be stored in a non-volatile memory such as an HDD (Hard Disk Drive) 14 or SSD (Solid State Drive) or an external memory such as a removable storage medium (not shown).
The RAM (Random Access Memory) 23 is a volatile memory, and functions as a main memory, a work area, or the like of the CPU 11. That is, the CPU 21 loads a program or the like necessary from the ROM 22 into the RAM 23 when executing the process, and realizes various functional operations by executing the program or the like.

The HDD 24 stores, for example, various data and various information necessary for the CPU 21 to perform processing using a program. Further, the HDD 24 stores, for example, various data and various information obtained by the CPU 21 performing processing using a program or the like.
The input unit 25 is composed of a pointing device such as a keyboard and a mouse.
The display unit 26 is composed of a monitor such as a liquid crystal display (LCD). The display unit 26 is a GUI (Graphical User Interface) which is a user interface for instructing and inputting various parameters used in the abnormal scene detection process, communication parameters used in communication with other devices, and the like to the parameter adjustment device 1. ) May be provided.

The communication I / F 27 is an interface that controls communication between the abnormality detection device 1 and the external device.
The communication I / F 27 provides an interface with the network and executes communication with an external device via the network. Video, abnormal scene determination result, abnormal scene confirmation input, various parameters, etc. are transmitted and received to and from the external device via the communication I / F27. In the present embodiment, the communication I / F 27 may execute communication via a wired LAN (Local Area Network) or a dedicated line conforming to a communication standard such as Ethernet (registered trademark). However, the network that can be used in this embodiment is not limited to this, and may be configured by a wireless network. This wireless network includes a wireless PAN (Personal Area Network) such as Bluetooth®, ZigBee®, UWB (Ultra Wide Band) and the like. Further, it includes a wireless LAN (Local Area Network) such as Wi-Fi (Wiless Fidelity) (registered trademark) and a wireless MAN (Metropolitan Area Network) such as WiMAX (registered trademark). Further, it includes a wireless WAN (Wide Area Network) such as LTE / 3G, 4G, and 5G. The network may connect each device so as to be able to communicate with each other, and the communication standard, scale, and configuration are not limited to the above.

At least a part of the functions of each element of the abnormality detection device 1 shown in FIG. 1 can be realized by the CPU 21 executing a program. However, at least a part of the functions of each element of the abnormality detection device 1 shown in FIG. 1 may be operated as dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU 21.

Although specific embodiments have been described above, the embodiments are merely examples and are not intended to limit the scope of the present invention. The devices and methods described herein can be embodied in forms other than those described above. Further, without departing from the scope of the present invention, omissions, substitutions and modifications can be made to the above-described embodiments as appropriate. Such abbreviations, substitutions and modifications are included in the claims and equivalents thereof and fall within the technical scope of the invention.

1 ... Abnormality detection device, 2 ... Learning data DB, 3 ... PC, 11 ... Data acquisition unit, 12 ... Feature extraction unit, 13 ... Abnormal scene candidate detection unit, 14 ... Abnormality determination device, 15 ... Scene presentation unit, 16 ... Emotion analysis unit, 21 ... CPU, 22 ... ROM, 23 ... RAM, 24 ... HDD, 25 ... input unit, 26 ... display unit, 27 ... communication I / F, 28 ... bus

Claims (16)

The video acquisition unit that acquires video data and
A feature extraction unit that extracts audio features from the video data acquired by the video acquisition unit and extracts image features from the video data, and a feature extraction unit.
An abnormal scene candidate detection unit that detects an abnormal scene candidate from the video data based on the audio feature extracted by the feature extraction unit.
An abnormality determining device that determines the abnormal scene candidate detected by the abnormal scene candidate detection unit as abnormal, normal, or other based on the audio feature and the image feature.
When the abnormality determination device determines that the candidate for the abnormal scene belongs to another, the candidate for the abnormal scene is presented via the user interface, and information to be added to the presented candidate for the abnormal scene. An information processing apparatus including a scene presentation unit that receives input from the user interface via the user interface. The information processing apparatus according to claim 1, wherein the abnormal scene candidate detection unit detects a candidate for the abnormal scene from the video data by using unsupervised learning. The claim is characterized in that the abnormal scene candidate detection unit detects a candidate for the abnormal scene from the video data by directly separating the abnormal audio feature without generating a model of the normal audio feature group. Item 2. The information processing apparatus according to Item 1 or 2. The information processing according to claim 3, wherein the abnormal scene candidate detection unit separates the abnormal voice feature by calculating the path length in the isolation forest of each voice feature. Device. The information processing according to any one of claims 1 to 4, wherein the feature extraction unit extracts audio features represented by a Mel Frequency spectrogram of audio data in the video data. Device. The feature extraction unit is characterized in that the Mel Frequency cepstrum coefficient (MFCC) is calculated from the voice data, the calculated MFCC is connected to the Mel frequency, and the voice feature is extracted. The information processing apparatus according to claim 5. The scene presenting unit is characterized in that information input via the user interface is added to the voice feature and the image feature and stored in a storage device as learning data for the abnormality determining device. The information processing apparatus according to any one of claims 1 to 6. 7. The abnormality scene candidate detection unit is characterized in that, when the number of the learning data stored in the storage device exceeds a predetermined threshold value, the abnormality determination device determines the candidate of the abnormality scene. The information processing device described in. When the number of the learning data stored in the storage device is within a predetermined threshold value, the abnormal scene candidate detection unit bypasses the determination by the abnormal device and causes the scene presentation unit to display the abnormal scene. The information processing apparatus according to claim 7, wherein the candidate is presented. In the anomaly determination device, the difference between the number of abnormal samples located in the vicinity of the candidate for the abnormal scene and the number of normal samples in the feature space in which the audio feature and the image feature are integrated is within a predetermined threshold value. The information processing apparatus according to any one of claims 1 to 9, wherein the candidate for the abnormal scene is determined elsewhere. The information processing apparatus according to any one of claims 1 to 10, wherein the abnormality determining device determines a candidate for the abnormal scene by the k-nearest neighbor method. From the image features extracted by the feature extraction unit, supervised learning is used to analyze facial emotions contained in the video data, and the analyzed facial emotion features are supplied to the abnormality determination device. Further equipped with an emotion analysis department
The information processing apparatus according to any one of claims 1 to 11. When the emotion analysis unit detects a candidate for the abnormal scene from the video data based on the analyzed emotion of the face, the abnormal scene candidate detection unit detects the abnormal scene based on the audio feature. The information processing apparatus according to claim 12, wherein the information processing apparatus is to be executed. An information processing system including a server and at least one client device connected to the server via a network.
The server
The video acquisition unit that acquires video data and
A feature extraction unit that extracts audio features from the video data acquired by the video acquisition unit and extracts image features from the video data, and a feature extraction unit.
An abnormal scene candidate detection unit that detects an abnormal scene candidate from the video data based on the audio feature extracted by the feature extraction unit.
An abnormality determining device that determines the abnormal scene candidate detected by the abnormal scene candidate detection unit as abnormal, normal, or other based on the audio feature and the image feature.
When the abnormality determination device determines that the abnormality scene candidate belongs to another, the abnormality scene candidate is presented via the user interface, and information to be added to the presented abnormality scene candidate. The scene presentation unit that accepts the input of
It has a transmission unit that transmits a candidate for the abnormal scene to the client device, and has.
The client device is
A receiving unit that receives the candidate of the abnormal scene transmitted from the server, and a receiving unit.
The user interface that presents the candidate of the abnormal scene received by the receiving unit and accepts the input of the information to be added to the presented candidate of the abnormal scene, and the user interface.
An information processing system comprising: a transmission unit for transmitting information to be added to the candidate of the abnormal scene for which the user interface has received input to the server. It is an information processing method executed by an information processing device.
Steps to acquire video data and
A step of extracting audio features from the acquired video data and extracting image features from the video data,
A step of detecting a candidate for an abnormal scene from the video data based on the audio features extracted by unsupervised learning.
A step of determining the candidate of the abnormal scene detected by the abnormality determining device as abnormal, normal, or other based on the audio feature and the image feature.
When the abnormality determination device determines that the candidate for the abnormal scene belongs to another, the candidate for the abnormal scene is presented via the user interface, and information to be added to the presented candidate for the abnormal scene. An information processing method comprising a step of accepting an input of the above through the user interface. It is an information processing program for causing a computer to execute information processing, and the program causes the computer to execute information processing.
Video acquisition processing to acquire video data and
A feature extraction process that extracts audio features from the video data acquired by the video acquisition process and extracts image features from the video data.
Anomalous scene candidate detection processing that detects anomalous scene candidates from the video data based on the audio features extracted by the feature extraction processing, and
Abnormality determination processing for determining the abnormal scene candidate detected by the abnormal scene candidate detection process by the abnormality determination device as abnormal, normal, or other based on the audio feature and the image feature.
When the abnormality determination device determines that the candidate for the abnormal scene belongs to another, the candidate for the abnormal scene is presented via the user interface, and information to be added to the presented candidate for the abnormal scene. An information processing program for executing a process including a scene presentation process for receiving an input of the above via the user interface.

Images