JP2022153360A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device capable of reducing the load of an operator and highly accurately detecting various abnormalities from a video, an information processing method, and an information processing program.

SOLUTION: An abnormality detecting device 1 comprises: a data acquisition unit which acquires video data; a feature extracting unit which extracts audio features from the video data acquired by the data acquisition unit and extracts image features from the video data; an abnormality scene candidate detecting unit which detects a candidate of an abnormal scene from the video data on the basis of the audio features extracted by the feature extracting unit; an abnormality determination unit which determines whether the candidate of the abnormal scene detected by the abnormal scene candidate detecting unit is abnormal, normal, or others on the basis of the audio and image features; and a scene presentation unit which presents a candidate of the abnormality scene via a user interface and receives an input of information which have to be added to the candidate of the presented abnormality scene via the user interface when the abnormality determination unit determines that the candidate of the abnormality scene belongs to the others..

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to an information processing device, an information processing method, and a program, and more particularly to technology for analyzing video and detecting anomalies.

Recent video distribution services enable real-time distribution of not only video content created by content providers but also video content created by general users.
In such a video distribution service, the distributed video is monitored so that the video content to be distributed does not include so-called abnormal scenes that are inappropriate for viewing. In order to prevent this, it is necessary to delete abnormal scenes, stop distribution, delete distribution accounts, etc. Such abnormal scenes include, for example, Non-Family-Safe (NFS) scenes such as violent scenes and scenes not intended for children.

Patent Literature 1 discloses an elevator monitoring device that detects abnormal behavior of passengers from photographed data photographed by a security camera provided inside an elevator car.
Specifically, in the monitoring device of Patent Literature 1, the threshold value for judging agitation is determined according to a predetermined frequency band extracted from the result of frequency analysis of voice data of passengers collected by an intercom installed in the car. is set, and the amount of variation in the movement of the occupant is statistically calculated from the photographed data photographed by the security camera. The monitoring device of Patent Literature 1 further compares the calculated variation amount of the occupant's movement with a violent judgment threshold, and regards the movement of the passenger as an abnormal behavior when the fluctuation amount of the passenger's movement is greater than or equal to the violent judgment threshold. determine outrage. As a result, even if the occupant can only move slightly, abnormal behavior can be determined from the photographed data.

JP 2013-63810 A

However, with the technique of Patent Document 1, the detectable abnormalities are limited to the rampage of the passengers in the elevator, so it is difficult to appropriately detect various abnormal scenes that can be included in various video contents. .

In particular, video distribution services are segmented into a wide range of segments depending on the age group and preferences of the main target users, and the range of abnormal scenes that are inappropriate for viewing varies for each video distribution service. . Furthermore, since the frequency of appearance of abnormal scenes in video content is usually very low, it is not suitable for creating a general-purpose database of learning data required for supervised machine learning. On the other hand, if an attempt is made to detect abnormal scenes from video content by unsupervised machine learning, the detection accuracy will drop.

By the way, in the video created by the content provider, tag information such as whether or not the video content distributed by the content provider includes a violent scene, whether or not the content is for children, or whether or not there is an age limit is included. It is often added, and it is possible to have content providers tag the presence of anomalous scenes when content is created.
On the other hand, the video content created by general users, which has been increasing in recent years, often does not have such tag information of abnormal scenes added, or even if it does, the tag information is not necessarily attached to the video distribution service. may not be appropriate for

For this reason, conventionally, depending on the video distribution service, the operator constantly monitors the video content to be distributed, and if an abnormal scene is found in the video content, the operator sets an age limit for the video content, or In some cases, distribution of content is stopped, which increases the time and work load of the operator who monitors the video, as well as the psychological load. At the same time, there is a risk that abnormal scenes may be overlooked due to manual monitoring of video content.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and an object of the present invention is to provide an information processing apparatus and an information processing method capable of detecting various abnormalities from images with high accuracy while reducing the burden on the operator. and to provide programs.

In order to solve the above problems, one aspect of an information processing apparatus according to the present invention is a video acquisition unit that acquires video data; an abnormal scene candidate detection unit for detecting abnormal scene candidates from the video data based on the audio features extracted by the feature extraction unit; and the abnormal scene candidate detection unit an anomaly determiner for determining, based on the audio feature and the image feature, the candidate for the abnormal scene detected by is determined to belong to others, the candidate for the abnormal scene is presented via a user interface, and input of information to be added to the candidate for the presented abnormal scene is accepted via the user interface. and a scene presenter.

The abnormal scene candidate detection unit may detect the abnormal scene candidates from the video data using unsupervised learning.

The abnormal scene candidate detection unit may detect the abnormal scene candidates from the video data by directly separating abnormal audio features without generating a model of normal audio features.

The abnormal scene candidate detector may isolate the abnormal audio features by calculating a path length in an isolation forest of each audio feature.

The feature extraction unit may extract an audio feature represented by a Mel frequency spectrogram of audio data in the video data.

The feature extracting unit may calculate Mel Frequency Cepstrum Coefficients (MFCC) from the audio data, connect the calculated MFCC to the Mel frequency, and extract the audio feature.

The scene presenting unit may add information input via the user interface to the audio feature and the image feature, and store the information in a storage device as learning data for the abnormality determiner.

The abnormal scene candidate detection unit may cause the abnormality determiner to determine the candidate of the abnormal scene when the number of the learning data stored in the storage device exceeds a predetermined threshold.

When the number of pieces of learning data stored in the storage device is within a predetermined threshold, the abnormal scene candidate detection unit bypasses the determination by the abnormal device and instructs the scene presentation unit to display the abnormal scene. Candidates can be presented.

The abnormality determiner determines that a difference between the number of abnormal samples and the number of normal samples located near the candidate for the abnormal scene is within a predetermined threshold in a feature space in which the audio features and the image features are integrated. In this case, other abnormal scene candidates may be determined.
The abnormality determiner may determine the candidate for the abnormal scene by a k-nearest neighbor method.

From the image features extracted by the feature extraction unit, using supervised learning, analyze facial emotion included in the video data, and supply the analyzed facial emotion features to the abnormality determiner. An emotion analysis unit may be further provided.

The emotion analysis unit causes the abnormal scene candidate detection unit to detect an abnormal scene based on the audio feature when the candidate for the abnormal scene is detected from the video data based on the analyzed emotion of the face. You can let it run.

One aspect of an information processing system according to the present invention is an information processing system comprising a server and at least one client device connected to the server via a network, wherein the server acquires video data. an acquisition unit, a feature extraction unit that extracts audio features from the video data acquired by the video acquisition unit, extracts image features from the video data, and based on the audio features extracted by the feature extraction unit, an abnormal scene candidate detection unit that detects abnormal scene candidates from the video data; and the abnormal scene candidates detected by the abnormal scene candidate detection unit are determined as abnormal, normal, and an anomaly determiner that determines one of the other, and when the anomaly determiner determines that the candidate for the abnormal scene belongs to the other, the candidate for the abnormal scene is presented via a user interface, and presented a scene presenting unit that receives input of information to be added to the abnormal scene candidate received via the user interface; and a transmission unit that transmits the abnormal scene candidate to the client device, The client device presents a receiving unit for receiving the abnormal scene candidates transmitted from the server, presents the abnormal scene candidates received by the receiving unit, and adds to the presented abnormal scene candidates. and a transmitting unit configured to transmit to the server information to be added to the abnormal scene candidate whose input is received by the user interface.

One aspect of an information processing method according to the present invention is an information processing method executed by an information processing apparatus, comprising: acquiring video data; extracting audio features from the acquired video data; a step of extracting features; a step of detecting abnormal scene candidates from the video data based on the extracted audio features by unsupervised learning; , a step of determining one of abnormal, normal, and others based on the audio features and the image features; presenting scene candidates via a user interface, and accepting input of information to be added to the presented abnormal scene candidates via the user interface.

One aspect of the information processing program according to the present invention is an information processing program for causing a computer to execute information processing, wherein the program causes the computer to perform image acquisition processing for acquiring image data, and the image acquisition processing. a feature extraction process for extracting an audio feature from the video data acquired by and extracting an image feature from the video data; and an abnormal scene candidate from the video data based on the audio feature extracted by the feature extraction process. an abnormal scene candidate detection process for detecting an abnormal scene candidate detection process for detecting an abnormal scene candidate detection process for detecting an abnormal scene candidate detection process for detecting an abnormal scene candidate detected by the abnormal scene candidate detection process for detecting an abnormal scene candidate based on the audio feature and the image feature by an abnormality determiner; an abnormality determination process for determining one of them; Input/output processing for accepting input of information to be added to scene candidates via the user interface.

ADVANTAGE OF THE INVENTION According to this invention, various abnormalities can be detected from an image|video with high precision, reducing an operator's load.
The objects, aspects and effects of the present invention described above and the objects, aspects and effects of the present invention not described above can be understood by a person skilled in the art to carry out the following invention by referring to the accompanying drawings and the description of the claims. can be understood from the form of

FIG. 1 is a block diagram showing an example of the functional configuration of an abnormality detection device according to Embodiment 1 of the present invention. FIG. 2 is a flowchart illustrating an example of a procedure of abnormal scene detection processing executed by the abnormality detection device according to the first embodiment; FIG. 3 is a diagram showing an example of a segment of audio data separated from video data by the feature extractor of the anomaly detection device. FIG. 4 is a diagram showing an example of mel-spectogram features of anomalous scene candidates extracted from audio data by the feature extraction unit of the anomaly detection device. FIG. 5 is a diagram showing an example of mel-spectogram features of a normal scene extracted from audio data by the feature extraction unit of the anomaly detection device. FIG. 6 is a diagram showing an example of speech features obtained by connecting mel-frequency cepstrum and mel-spectogram output by the feature extraction unit of the anomaly detection device to the anomalous scene candidate detection unit. FIG. 7 is a diagram for explaining separation of abnormal feature points in an isolation forest used in unsupervised learning by the abnormal scene candidate detection unit of the abnormality detection device. FIG. 8 is a schematic diagram showing an example of a decision tree of an isolation forest used in unsupervised learning by the abnormal scene candidate detection unit of the abnormality detection device. FIG. 9 is a schematic diagram for explaining an example of the k-nearest neighborhood method used by the anomaly determiner of the anomaly detection device to determine an anomaly of an anomalous scene candidate. 10 is a flowchart illustrating an example of a procedure of abnormal scene detection processing executed by the abnormality detection device according to the modification of the first embodiment; FIG. FIG. 11 is a block diagram showing an example of the functional configuration of an abnormality detection device according to Embodiment 2 of the present invention. FIG. 12 is a flowchart illustrating an example of a procedure of abnormal scene detection processing executed by the abnormality detection device according to the second embodiment; FIG. 13 is a flowchart illustrating an example of a procedure of abnormal scene detection processing executed by the abnormality detection device according to the modification of the second embodiment; FIG. 14 is a schematic diagram showing an example of a face emotion recognition result recognized from image data of video data by the emotion analysis unit of the abnormality detection device according to the second embodiment. FIG. 15 is a block diagram showing an example of the hardware configuration of an abnormality detection device according to each embodiment of the present invention.

Embodiments for carrying out the present invention will be described in detail below with reference to the accompanying drawings. Among the constituent elements disclosed below, those having the same functions are denoted by the same reference numerals, and descriptions thereof are omitted. The embodiments disclosed below are examples of means for realizing the present invention, and should be appropriately modified or changed according to the configuration of the device to which the present invention is applied and various conditions. is not limited to the embodiment of Also, not all combinations of features described in the present embodiment are essential for the solution means of the present invention.

(Embodiment 1)
The anomaly detection apparatus according to the present embodiment extracts features of audio data and image data from video data, and uses multimodal features of these audio data and image data to detect anomalous scenes from video data in multiple stages. Detect semi-automatically.
In the following, the anomaly detection device first detects anomalous scene candidates by unsupervised learning based on the features of audio data extracted from video data streamed in real time, and then detects anomalous scene candidates. is judged by an anomaly judgment device as one of an abnormal scene, a normal scene, and a scene requiring operator judgment, and video data of candidate abnormal scenes judged to require operator judgment are presented. An example will be described in which confirmation input as to whether or not is added to the features of the video data and accumulated as learning data for the anomaly determiner.

However, this embodiment is not limited to this. For example, the anomaly detection device may detect an anomalous scene ex post facto from recorded video data. Further, for example, abnormal scene detection is variably controlled according to the number of stored learning data, and all of the detected abnormal scene candidate video data is presented to the operator bypassing the abnormality determiner. Alternatively, the anomaly determiner may automatically detect an anomalous scene based on the audio and image features of the detected anomalous scene candidate video data. In the latter case, the threshold for abnormality determination may be set relatively low, and only abnormal scenes near the threshold may be appropriately presented to the operator for confirmation.

<Functional configuration of abnormality detection device>
FIG. 1 is a block diagram showing an example of the functional configuration of an abnormality detection device 1 according to this embodiment.
The abnormality detection device 1 shown in FIG.
The abnormality detection device 1 may be communicably connected to a client device 3 configured by a PC (Personal Computer) or the like via a network. In this case, the abnormality detection device 1 is mounted on a server, and the client device 3 may provide a user interface when the abnormality detection device 1 executes input/output of information with the outside. Some or all of the components 11 to 15 including the presentation unit 15 may be provided.

The data acquisition unit 11 acquires video data streamed in real time and supplies the acquired video data to the feature extraction unit 12 . The video data is moving image data including audio data and visual data, but the data acquisition unit 11 acquires still image data including audio data instead of the moving image data. It may be supplied to the feature extraction unit 12 .
The data acquisition unit 11 may acquire video data pre-recorded in a non-volatile storage device such as an HDD of the abnormality detection device 1 instead of streaming-delivered video data. may be received from via the communication I/F.
The data acquisition unit 11 also receives input of various parameters necessary for executing abnormal scene detection processing in the abnormality detection device 1 . The data acquisition unit 11 may receive input of various parameters through the user interface of the client device 3 communicably connected to the abnormality detection device 1 .

The feature extraction unit 12 separates audio data from the video data supplied from the data acquisition unit 11 and extracts audio features from the separated audio data.
The feature extraction unit 12 also separates image data from the video data supplied from the data acquisition unit 11 and extracts image features from the separated image data.
The feature extraction unit 12 supplies the extracted audio features and image features to the abnormal scene candidate detection unit 13 together with the video data.

The abnormal scene candidate detection unit 13 detects abnormal scene candidates from the video data based on the audio features supplied from the feature extraction unit 12 , and supplies the detected abnormal scene candidates to the abnormality determiner 14 . The abnormal scene candidate detection unit 13 may also supply the detected abnormal scene candidates to the scene presentation unit 15 by bypassing the abnormality determiner 14 .

Abnormal scenes include, but are not limited to, scenes inappropriate for family viewing (Non-Family-Safe: NFS), such as violent scenes and scenes not for children. Abnormal scenes are scenes or content that are stipulated that they should not be distributed via the video distribution service according to the terms or rules of each video distribution service, and other operators should not ultimately distribute audio and images of video data. shall broadly include any scene or content manually determined to be
Details of the feature extraction processing performed by the feature extraction unit 12 and the abnormal scene candidate detection processing performed by the abnormal scene candidate detection unit 13 will be described later with reference to FIGS. 3 to 8. FIG.

In this embodiment, the abnormal scene candidate detection unit 13 detects abnormal scene candidates by classifying audio features by unsupervised learning. In the video data distributed by streaming, the frequency of applications for abnormal scenes is very low, and the criteria for determining whether or not an abnormal scene should be included vary from video distribution service to video distribution service. is changed, it is difficult to prepare appropriate teacher data in advance, and it is difficult to achieve highly accurate classification by supervised learning. In this embodiment, by classifying the audio features from only the audio data in the video data by unsupervised learning, it is possible to detect candidates for abnormal scenes with high accuracy and low load even with a small number of samples. .

The abnormal scene candidate supplied from the abnormal scene candidate detection unit 13 is input to the abnormality determiner 14, and the image data of the input abnormal scene candidate is classified into an abnormal scene, a normal scene, or a scene that requires an operator's judgment. classified as The anomaly determiner 14 adds the anomalous scene candidates classified into either an anomalous scene or a normal scene out of the abnormal scene candidate classification results and stores them in the learning data DB (database) 2 with the classification results. To go. In addition, the abnormality determiner 14 supplies to the scene presenting unit 15 an abnormal scene candidate classified as a scene requiring operator's judgment among the abnormal scene candidates.

In this embodiment, the abnormality determiner 14 uses a feature space in which the audio features and image features of the video data extracted by the feature extraction unit 12 are integrated, and uses supervised learning to identify candidates for input abnormal scenes. , an abnormal scene, a normal scene, and a scene requiring operator's judgment. The abnormality determiner 14 may perform learning using the abnormal scene candidate classification results accumulated in the learning data DB 2 as teacher data.
The details of the abnormal scene determination process executed by the abnormality determiner 14 will be described later with reference to FIG.

The scene presenting unit 15 presents abnormal scene candidates, which are classified as scenes requiring operator judgment, supplied from the abnormality determiner 14 to the outside via a display device or the like, and receives confirmation input from the operator. The abnormality detection device 1 may also present the abnormal scene candidates supplied from the abnormal scene candidate detection unit 13 to the outside, and may receive confirmation input from the operator.
The abnormality detection device 1 may use its own display device or the like as a user interface. , or an operator confirmation input may be accepted.
In this case, the abnormality detection device 1 further includes a transmitting/receiving unit that transmits the abnormal scene candidates supplied from the abnormal scene candidate detection unit 13 to the client device 3 and receives the operator's confirmation input transmitted from the client device 3. Be prepared. The client device 3 includes a transmitting/receiving unit that receives anomalous scene candidates transmitted from the anomaly detection device 1 and transmits an operator's confirmation input for the anomaly scene candidates presented via the user interface to the anomaly detection device 1. you can

The operator confirms that the presented abnormal scene candidate is either an abnormal scene or a normal scene by comparing the video data image of the abnormal scene candidate presented by the scene presentation unit 15 with the sound. , the confirmation result is input to the scene presentation unit 15 . The operator can take a predetermined action on the abnormal scene candidate confirmed as an abnormal scene. For example, the confirmed abnormal scene may be deleted from the video data to be distributed, the distribution of the video data may be stopped, or the account of the distribution source user of the video data may be suspended.
The scene presenting unit 15 adds the confirmation result (annotation of an abnormal scene or a normal scene) confirmed and input by the operator to the sound feature and image feature of the candidate of the abnormal scene presented, and stores the result in the learning data DB 2 as learning data. Store.

As a specific example, it is assumed that the abnormal scene candidate detection unit 13 detects a gunshot-like sound from the audio features extracted from the video data, and detects the scene containing the sound as an abnormal scene candidate. In this case, the operator may check the image of the candidate for the abnormal scene to confirm whether it is an abnormal scene or a normal scene.
For example, if an abnormal scene candidate image contains a gun or other violent or cruel object, it can be identified as an abnormal scene. can be confirmed.

Thus, in this embodiment, multimodal information of audio and image of video data is used to semi-automatically detect an abnormal scene in a plurality of stages. Specifically, candidates for abnormal scenes are automatically detected from the sound of the video data, the candidates for the detected abnormal scenes are presented to the operator, and the image of the candidate for the abnormal scene is checked to see if it is an abnormal scene or a normal scene. I am letting As a result, the operator's burden in monitoring the video to be distributed is greatly reduced.

<Procedure of Abnormal Scene Detection Processing>
FIG. 2 is a flowchart showing an example of the abnormal scene detection process performed by the abnormality detection device 1 according to the present embodiment.
Each step in FIG. 2 is implemented by the CPU reading and executing a program stored in a storage device such as an HDD of the abnormality detection device 1 . Also, at least part of the flowchart shown in FIG. 2 may be realized by hardware. When implemented by hardware, for example, by using a predetermined compiler, a dedicated circuit may be automatically generated on an FPGA (Field Programmable Gate Array) from a program for implementing each step. Also, a Gate Array circuit may be formed in the same manner as the FPGA and implemented as hardware. Also, it may be realized by an ASIC (Application Specific Integrated Circuit).

In S1, the feature extraction unit 12 of the abnormality detection device 1 separates the video data supplied from the data acquisition unit 11 into audio data and image data, and extracts audio features and image features, respectively.
FIG. 3 shows that the feature extraction unit 12 separates from the video data (for example, multimedia format such as mp4 or m3u8) supplied from the data acquisition unit 11 as preprocessing for feature extraction, for example, segments in units of 1 second 1 shows an example of an audio signal waveform of audio data (for example, wav format). In FIG. 3, the vertical axis indicates the amplitude of the voice, and the horizontal axis indicates time.
The feature extraction unit 12 may normalize the separated audio data by appropriate upsampling or the like in accordance with the sound source that can be assumed in the video data to be distributed.

In the present embodiment, the feature extracting unit 12 may extract speech features from the speech data shown in FIG. 3 as speech features expressed by, for example, a mel spectrogram (Mel Frequency Spectrogram).
A spectrogram is the result of passing an audio signal through a window function to calculate the frequency spectrum. represented graphically. A spectrogram corresponds to a so-called voiceprint in which the spectrum of each audio data segment (frame) obtained by extracting frequency components and amplitude components of an audio signal by Fourier transform, for example, is arranged along the time axis. A mel spectrogram is a spectrogram converted by the mel scale for weighting considering human pitch perception (frequency perception characteristics).

FIG. 4 shows an example of audio features expressed by a mel-spectrogram extracted from audio data separated from video data by the feature extraction unit 12 and detected as abnormal scene candidates by the abnormal scene candidate detection unit 13. FIG. indicates 4 and 5, the X-axis indicates time, the Y-axis indicates frequency, and the Z-axis indicates amplitude, ie strength of the audio signal. In addition, in FIGS. 4 and 5, a cell with a higher signal strength is indicated by a lighter pattern, and a cell with a lower signal strength is indicated by a darker pattern.
The spectrogram shown in FIG. 4 shows a pattern in which the sound volume is large, the signal strength distribution has peaks, and the audio signal is attenuated in a short period of time. FIG. 4 shows an example of a spectrogram of a gun shot, but it is conceivable that, for example, a person's shouting or the sound of hitting something would exhibit a similar or similar pattern.

On the other hand, FIG. 5 shows audio features represented by mel-spectograms extracted from audio data segments separated from video data by the feature extraction unit 12, which are determined as normal scenes by the abnormal scene candidate detection unit 13 (abnormal scenes). 4 shows an example of audio features that are not detected as scene candidates. The spectrogram shown in FIG. 5 shows a pattern of low or medium volume and uniform distribution of signal strength over time.
The feature extraction unit 12 separates foreground audio (e.g., human utterances, shouts, etc.) and background audio (e.g., music, crowd noise, etc.) from audio data, and extracts the spectrogram of one of the audio as an audio feature. It may be supplied to the abnormal scene candidate detection unit 13 . In this case, for example, a sound that appears temporarily on the time axis and is not repeated can be separated as a foreground sound.

In this embodiment, the feature extraction unit 12 further calculates Mel Frequency Cepstrum Coefficients (MFCC) from the audio data, connects the calculated MFCC to the Mel spectrogram shown in FIGS. Audio features may be extracted.
The cepstrum is obtained by multiplying the logarithm of the amplitude spectrum obtained by Fourier-transforming an audio signal to obtain a logarithmic spectrum, and applying the Fourier transform again to the logarithmic spectrum to convert it into a spectrum. By obtaining the cepstrum of the logarithmic spectrum (logarithmic cepstrum), it is possible to separate the sound source component that fluctuates at a high frequency from the convoluted vocal tract characteristic component.

The low-order components of the logarithmic cepstrum represent the spectral envelope of speech (frequency characteristics derived from vocal tract components). It is possible to remove pitch components with large individual differences and extract only the acoustic characteristics of the vocal tract, which are important for identifying phonemes. The MFCC is a feature amount obtained by applying a weighting that considers the human frequency perception characteristic to the low-order components of the logarithmic cepstrum by applying the Mel scale.

Specifically, the amplitude spectrum is applied to multiple filter banks that are evenly spaced on the Mel scale, the spectral components of each band are extracted, the sum of the amplitude spectra of each band is taken, and the amplitude spectrum is compressed into a multi-dimensional amplitude spectrum. , taking the logarithm of this compressed amplitude spectrum to obtain the logarithmic amplitude spectrum.
The obtained Mel frequency spectrum (the logarithmic amplitude spectrum compressed on the Mel scale) is subjected to Fourier transform (for example, Discrete Fourier Transform: DFT) to convert it into a Mel frequency cepstrum. The MFCC can be obtained by extracting the low-order components of the cepstrum (vocal tract components of the spectrum) and performing normalization processing as necessary.

FIG. 6 shows an example of speech features in which an MFCC 61 obtained by slicing in a unit time (for example, one second) and taking an average value, and a mel-spectogram 62 obtained by taking an average amplitude of the mel spectrum on the time axis are connected. show.
By supplying the audio features as shown in FIG. 6 to the abnormal scene candidate detection unit 13 to detect abnormal scene candidates, the frequency component information of the audio data, especially the frequency components that are important for human hearing, are not lost. , the speech features can be adequately compressed.

Note that the audio features extracted from the audio data of the video data by the feature extraction unit 12 are not limited to the expressions shown in FIG. You can
The feature extraction unit 12 may also extract image features from all or part of the image data separated from the video data using, for example, a convolutional neural network (CNN). However, the method for extracting image features from image data by the feature extraction unit 12 is not limited to CNN, and any method can be used.

Returning to FIG. 2, in S2, the abnormal scene detection unit 13 of the abnormality detection device 1 performs classification by unsupervised learning based on the audio features of the audio data separated from the video data supplied from the feature extraction unit 12. to detect candidates for abnormal scenes.
The abnormal scene detection unit 13, for example, by isolation forest (Isolation Forest: IF), separates an audio feature having an abnormal value on a feature space of audio features, and detects a video scene corresponding to the separated abnormal audio feature. is detected as an abnormal scene candidate.

Isolation forest is one of unsupervised learning that directly isolates features with abnormal values without modeling (profiling) a group of features with normal values to generate a normal model. Since it is a high-speed algorithm and consumes little memory, it is suitable for monitoring video that is distributed in real time. In addition, since modeling of a normal model is unnecessary, it is difficult to cause a decrease in accuracy even with a small number of samples.

FIG. 7 is a schematic diagram illustrating an algorithm for isolating features having outliers in the feature space by the isolation forest. The isolation forest repeatedly generates partitions as indicated by broken lines in FIG. 7 until each feature point placed on the feature space can be separated from all other feature points. Referring to FIG. 7, the leftmost and rightmost feature points each require fewer partitions than the feature points located near the center. In FIG. 7, the leftmost and rightmost feature points can be separated from all other feature points by one partition, so they can each be detected as feature points with outliers.

FIG. 8 is a conceptual diagram expressing the iterative processing of partition generation in FIG. 7 in a binary tree structure (isolation tree). The number of partitions in FIG. 7 can be represented by the path length from the root node to the end node of the tree structure in FIG.
The abnormal scene candidate detection unit 13 calculates an anomaly score for each feature point using Equation 1 below, based on the path length of the feature point for each audio feature.

（式１）

(Formula 1)

where E(h(x)) of the exponent on the right side is the average path length, and c(n) is a normalization factor that depends on the number of instances n in the data set. The abnormality score S(x, n) of each feature point approaches 1 as the average path length becomes shorter, and approaches 0 as the average path length becomes longer.

Referring to FIG. 8, the left bar corresponds to anomaly score values from 0 to 1 from bottom to top. The abnormal scene detection unit 13 compares the abnormal score threshold θ between 0 and 1 with the abnormal score S(x, n) of each feature point, and the abnormal score S(x, n) exceeds the threshold θ. The audio features of the feature points exceeding the score are determined as abnormal values (outliers), and the audio features of the feature points whose abnormal scores S(x, n) are within the threshold value θ are determined as normal values.

The abnormal scene detection unit 13 detects video scenes including audio features and corresponding image features for which abnormal scores exceeding the threshold θ are calculated as abnormal scene candidates.
The unsupervised learning algorithm used by the abnormal scene candidate detection unit 13 to detect abnormal scene candidates is not limited to the isolation forest. The abnormal scene candidate detection unit 13 uses a Variational AutoEncoder (VAE) instead of the isolation forest to calculate the reconstruction score of the speech feature, thereby detecting the candidate of the abnormal scene. may be used, or any other unsupervised learning may be used.

Returning to FIG. 2, in S3, the abnormal scene candidate detection unit 13 of the abnormality detection device 1 compares the number of learning data stored in the learning data DB2 with a predetermined threshold. If the number of learning data stored in the learning data DB 2 exceeds the predetermined threshold (S3: Y), the process proceeds to S4. supply. On the other hand, when the number of learning data stored in the learning data DB2 is within the predetermined threshold (S3: N), the processing (S4 to S6) in the abnormality determiner is bypassed and the process proceeds to S7.

In the present embodiment, when the abnormality determiner 14 does not accumulate enough learning data for abnormal scene determination in the learning data DB 2, it is determined that the accuracy (reliability) of abnormal scene determination by the abnormality determiner 14 is not sufficient. , bypasses the processing in the abnormality determiner 14 . Then, the scene presenting unit 15 directly presents the abnormal scene candidates detected by the abnormal scene candidate detecting unit 13 to the operator, and receives confirmation input from the operator. As a result, while the number of samples of learning data is small, it is possible to reduce the processing load of the machine learning execution in the abnormality determiner 14 by always requesting the operator's confirmation judgment for the candidate of the abnormal scene.

As described above, in the present embodiment, the control of how to determine an abnormal scene from detected abnormal scene candidates is optimized autonomously. Specifically, while the number of samples of learning data to the abnormality determiner 14 is small, priority is given exclusively to the operator's determination of an abnormal scene, thereby preventing a decrease in the accuracy of the determination of an abnormal scene. On the other hand, when the number of samples of learning data to the abnormality determiner 14 exceeds a predetermined threshold, only abnormal scene candidates that the abnormality determiner 14 could not classify as either an abnormal scene or a normal scene are sent to the operator. By presenting and requesting confirmation input, the operator's load can be further reduced.

In S4, the anomaly determiner 14 of the anomaly detection device 1 classifies the anomalous scene candidates supplied from the anomalous scene candidate detector 13 into normal scenes, anomalous scenes, and scenes requiring operator judgment. , the abnormality of the candidate of the abnormal scene is determined, and the determination result is stored in the learning data DB2.
Specifically, the abnormality determiner 14 uses, for example, k-nearest neighbor algorithm (k-NN) as supervised learning, on a feature space in which audio features and image features are integrated The abnormal scene candidates supplied from the abnormal scene candidate detection unit 13 are classified by searching for the nearest neighbor solution.

FIG. 9 is a conceptual diagram illustrating an example of a classification algorithm based on the k-nearest neighbor method.
Referring to FIG. 9, object groups indicated by circle marks are arranged in the feature space. Each object is represented by a position vector in a multidimensional feature space, and the correct classification class is known. The star mark in the center of the concentric circles is the position vector of the classification target whose classification class is unknown, and in this embodiment, the position vector of the abnormal scene candidate to be determined. In the k-nearest neighbor method, the distance between a new position vector indicated by a star mark and the existing position vector group indicated by a circle mark is calculated, and k nearest samples are selected. The distance between position vectors may be calculated as Euclidean distance, but may be calculated as other distances such as Manhattan distance.

Referring to FIG. 9, when k=3, two dark circle marks and one light circle mark are arranged as the three closest objects within the inner concentric circle. position vectors fall into the dark circle class. On the other hand, when k=6, two dark circle marks and four light circle marks are arranged as the six closest objects in the outer concentric circle, so the position vector to be determined is: It is classified into the thin circle mark class. Note that the class may be determined by weighting the distance from the new position vector among the k nearest objects.

The anomaly determiner 14 maps, as a position vector, candidates for anomalous scenes whose correct class is unknown on a feature space in which audio features and image features of video data are integrated, and k nearest objects (samples ), by comparing the number of samples classified as abnormal scenes with the number of samples classified as normal scenes, candidates for abnormal scenes to be determined are classified into abnormal scenes, normal scenes, and operator judgment. Classify it into one of the required scenes.

Specifically, the anomaly determiner 14 determines the number of samples classified as an anomalous scene among the k nearest samples to an anomaly scene candidate on the feature space in which the audio features and the image features are integrated. is sufficiently larger than the number of samples classified as normal scenes, the abnormal scene candidate to be determined is determined to be an abnormal scene. The abnormality determination unit 14 also determines whether the number of samples classified as normal scenes among the k nearest neighbor samples to the abnormal scene candidate is sufficiently larger than the number of samples classified as abnormal scenes in the feature space. In this case, the abnormal scene candidate to be determined is determined to be a normal scene.

On the other hand, the abnormality determiner 14 calculates the difference between the number of samples classified as abnormal scenes and the number of samples classified as normal scenes among the k nearest samples to the candidate of abnormal scenes in the feature space. is small and within a predetermined threshold, the candidate for the abnormal scene to be determined is determined to be a scene requiring operator's determination.
Alternatively, the anomaly determiner 14 selects either an anomalous scene or a normal scene as an anomaly scene candidate to be determined, depending on the magnitude of the number of samples of k nearest neighbor anomalous scenes and the number of samples of normal scenes. It may be classified automatically. In particular, when a sufficient number of samples of learning data are accumulated in the learning data DB2, it is possible to eliminate the need for operator intervention in abnormal scene detection.
Note that the algorithm used by the abnormality determiner 14 to determine the abnormality of the abnormal scene candidate is not limited to the k-neighborhood method described above. The anomaly determiner 14 may determine anomalous scenes using other supervised learning machine learning algorithms, including, for example, neural networks such as CNNs, Support Vector Machines (SVMs), and the like.

Returning to FIG. 2, when the abnormality determiner 14 determines in S4 that the candidate for the abnormal scene is an abnormal scene, the process proceeds to S5, where distribution of the video content including the abnormal scene is stopped, and the account of the distribution source of the video content is , or the determined abnormal scene is deleted, and the process ends.
When the abnormal scene candidate is determined to be a normal scene, the abnormality determiner 14 proceeds to S6, continues distribution of the video content including the abnormal scene candidate, and terminates the process. On the other hand, if the abnormality determiner 14 determines that the candidate for the abnormal scene is a scene that requires the operator's determination, the process proceeds to S7.

In S7, the scene presenting unit 15 of the abnormality detection device 1 presents the candidate of the abnormal scene supplied from the candidate of abnormal scene detection unit 13 or the candidate of the abnormal scene determined by the abnormality determiner 14 as the scene requiring the operator's determination. The video (audio data and image data) is presented to the operator.
In S8, the scene presenting unit 15 of the abnormality detection device 1 receives an input for tagging either an abnormal scene or a normal scene as an operator confirmation input for the image of the candidate for the abnormal scene presented in S7.
In S9, for the abnormal scene candidate tagged as an abnormal scene, the operator performs processing for the abnormal scene, that is, stop distribution of the video content containing the abnormal scene, delete the account of the distribution source of the video content, or Execute processing such as deletion of abnormal scenes. On the other hand, the operator allows the video distribution to continue for abnormal scene candidates tagged as normal scenes without executing processing for abnormal scenes.

In S10, the scene presentation unit 15 of the abnormality detection device 1 converts the tag (label) of the operator's abnormal scene or normal scene input to the scene presentation unit 15 in S8 into the audio features and images of the presented abnormal scene candidates. It is stored in the learning data DB 2 as learning data, which is the abnormal scene determination result by the operator, in association with the feature. As a result, the learning data DB2 is updated with the new learning data.

At S11, the abnormality determiner 14 of the abnormality detection device 1 performs re-learning based on the learning data DB2 updated at S10. It should be noted that whether re-learning of the abnormality determiner 14 is necessary depends on the abnormality determination algorithm of the abnormality determiner 14 . For example, as described above, when the abnormality determiner 14 uses the k-nearest neighbor method, each time an abnormal scene is determined, the learning data DB2 is referred to and k nearest samples are selected. It is not necessary to re-learn in , and it is sufficient to update the learning data DB2 in S10, and the processing of S11 may be omitted.

On the other hand, when the abnormality determiner 14 uses a neural network, SVM, or the like, it is necessary to relearn the abnormality determiner 14 at some timing to update the parameters of the abnormality determiner 14 .
As for the timing of relearning, the abnormality determiner 14 may be relearned each time the learning data DB2 is updated, or the abnormality determiner 14 may be relearned each time the learning data DB2 is updated a predetermined number of times. good. Alternatively, the abnormality determiner 14 can be re-learned immediately before using the abnormality determiner 14 in S4. Consideration should be given to the fact that real-time performance may be degraded.

<Modification>
FIG. 10 is a diagram showing a modification of the abnormal scene detection process executed by the abnormality detection device 1 according to this embodiment.
As a modified example, as shown in FIG. 10, the abnormal scene candidate detection unit 13 omits the processing of S3, and uniformly detects the detected abnormal scenes regardless of the number of learning data accumulated in the learning data DB 2. can be supplied to the abnormality determiner 14 . As a result, it is possible to further reduce the operator's burden of confirming abnormal scene candidates in video monitoring.

As described above, according to the present embodiment, the anomaly detection apparatus extracts audio features and image features from acquired video data, and classifies the extracted audio features by unsupervised learning to detect video data. Detect abnormal scene candidates from data. The abnormality detection device also includes an abnormality determiner that determines the detected abnormal scene candidate as abnormal, normal, or other based on the audio features and image features of the video data, and the abnormality determiner: If it is determined that the candidate for the abnormal scene belongs to Others, the candidate for the abnormal scene is presented via a user interface, and information to be added to the candidate for the presented abnormal scene is input through the user interface. accepted through

As a result, in the first step, abnormal scene candidates are detected at high speed and low load by unsupervised learning based on the audio features in the video data, and in the second step, the audio features in the video data and Abnormal scene determination by an abnormality determiner based on image features and abnormal scene determination by operator's visual observation based on image presentation are used in a complementary manner.
Therefore, various abnormalities can be detected from the video with high accuracy while reducing the burden on the operator.
As a result, high-speed and highly accurate multimodal detection of abnormal scenes can be realized, which sufficiently follows video data that is distributed in real time and can include various abnormal scenes.

(Embodiment 2)
Only the differences from the first embodiment will be described in detail below with reference to FIGS. 11 to 14. FIG.
In this embodiment, in addition to the above-described first embodiment, the emotion of a person, who is an object in the video, is analyzed from the image characteristics of the video data, and the result of the emotion analysis is used to detect candidates for abnormal scenes and to detect abnormal scenes. Used for judgment.

FIG. 11 is a block diagram showing an example of the functional configuration of the abnormality detection device 1 according to this embodiment.
The block diagram of FIG. 11 includes an emotion analysis unit 16 in addition to the functional configuration of the abnormality detection device 1 of the first embodiment shown in FIG.
In FIG. 11, the functional configurations of the data acquisition unit 11, the feature extraction unit 12, the abnormal scene candidate 13, the abnormality determination unit 14, and the scene presentation unit 15 are the same as the corresponding units shown in FIG.
Referring to FIG. 11, feature extractor 12 supplies image features extracted from image data in video data to emotion analyzer 16 .

The emotion analysis unit 16 analyzes the emotion of the human face, which is the object in the image, based on the image features of the video data supplied from the feature extraction unit 12 .
The emotion analysis unit 16 may estimate the emotion of the person's face in the image by analyzing the person's face in the image using, for example, supervised learning such as CNN. A person's facial emotion estimated from a person's facial image may include, for example, anger, disgust, fear, happiness, sadness, surprise, and other (neutral) emotions.
The emotion analyzer 16 may also determine the emotion of the person's face in the target image based on the estimated average confidence of the emotion calculated over a plurality of temporally adjacent image frames. .

The emotion analysis unit 16 may further analyze the movement of the person's body and limbs in the image, objects held by the person (eg, microphone, musical instrument, etc.), or the background (eg, indoors or outdoors, etc.). The feature extraction section 12 may extract features of the target object to be analyzed by the emotion analysis section 16 and supply them to the emotion analysis section 16 .

In this embodiment, the emotion of a person's face estimated by the emotion analysis unit 16 from the image features of the image of the person's face is detected by the abnormal scene candidate detection process executed by the abnormal scene candidate detection unit 13 and by the abnormality determiner 14 . supplements the abnormality determination process of the candidate of the abnormal scene executed by .
Specifically, the emotion analysis unit 16 estimates the context of the video from the emotion of the person's face estimated from the image features of the person's face, and instructs the abnormal scene candidate detection unit 13 to detect an abnormal scene candidate. Queues (triggers) for processing may be provided. For example, when the emotion analysis unit 16 analyzes an image of a person's face and detects, for example, anger, fear, surprise, etc., as the emotion of the person's face, there is a possibility that the video containing the image is an abnormal scene. is high, the emotion analysis unit 16 may give a cue to the abnormal scene analysis unit 13 to execute processing for detecting abnormal scene candidates from the audio features of the video.

The emotion analysis unit 16 also supplies the emotion features of the human face estimated from the image features of the person's face to the abnormality determiner 14, and the abnormality determiner 14 detects the human face supplied from the emotion analysis unit 16. may be integrated into the feature space, and abnormal scene candidates may be judged to be abnormal by the k-neighborhood method. For example, the abnormality determiner 14 may use features such as anger, fear, surprise, etc. as positive factors for determining an abnormal scene as emotions of a person's face.
The emotion analysis unit 16 further supplies an analysis result of the emotion of the person's face estimated from the image features of the person's face to the scene presentation unit 15 , and the scene presentation unit 15 receives the human facial emotion supplied from the emotion analysis unit 16 . For example, the analysis result of the emotion of the face may be displayed together with the presented image by superimposing it or displaying it in a separate window.

FIG. 12 is a flowchart showing an example of the procedure of abnormal scene detection processing executed by the abnormality detection device 1 according to the second embodiment.
In the flowchart of FIG. 12, the process of S12 is added between S1 and S2 to the abnormal scene detection process executed by the abnormality detection device 1 of the first embodiment shown in FIG.
The processing of S1 is the same as that of the first embodiment shown in FIG. That is, as in the first embodiment, the feature extraction unit 12 of the abnormality detection device 1 extracts audio features and image features from the video data supplied by the data acquisition unit 11 .

In S1, when the feature extraction unit 12 of the abnormality detection device 1 extracts the audio feature and the image feature from the video data, the process proceeds to S12.
In S<b>12 , the emotion analysis unit 16 of the abnormality detection device 1 detects abnormal scene candidates from the image features extracted by the feature extraction unit 12 . Specifically, the emotion analysis unit 16 estimates a person's emotion from the image features of the person's face in the image. A video scene including a video scene may be detected as a candidate for an abnormal scene.
When the emotion analysis unit 16 detects abnormal scene candidates from the image features, the emotion analysis unit 16 queues ( trigger).

Following S12, in S2, the abnormal scene candidate detection unit 13 of the abnormality detection device 1 is supplied from the emotion analysis unit 16 when the emotion analysis unit 16 detects an abnormal scene candidate from the image feature and gives a trigger. Abnormal scene candidates are detected by classifying audio features corresponding to the abnormal scene candidates using unsupervised learning.

Alternatively, the abnormal scene candidate detection unit 13 always detects abnormal scene candidates from the audio features of the video data regardless of whether or not the emotion analysis unit 16 gives a trigger, and detects the image from the emotion analysis unit 16. When the feature-based abnormal scene candidate detection is triggered, it may be confirmed from the audio features of the detected abnormal scene candidate whether it should be supplied to the abnormality detector 14 as an abnormal scene candidate.
The processing from S2 to S11 is the same as in the first embodiment shown in FIG.
10, the abnormality detection device 1 according to the present embodiment omits the determination and branching process of S3, and regardless of the number of learning data stored in the learning data DB 2, the abnormality detection device 14 of S4 You may proceed to the abnormal scene determination process.

FIG. 13 is a flowchart showing an example of a processing procedure of a modified example of abnormal scene detection processing executed by the abnormality detection device 1 according to the second embodiment.
In the flowchart of FIG. 13, the process of S13 is added between S2 and S3 to the abnormal scene detection process executed by the abnormality detection device 1 of the first embodiment shown in FIG.
The processing of S1 and S2 is the same as that of Embodiment 1 shown in FIG. That is, as in the first embodiment, the feature extraction unit 12 of the abnormality detection device 1 extracts audio features and image features from the video data supplied by the data acquisition unit 11, and the abnormal scene candidate detection unit 13 extracts the feature Abnormal scene candidates are detected based on the audio features of the video data supplied from the extraction unit 12 .

Next, in S13, the emotion analysis unit 16 of the abnormality detection device 1 extracts a human face from the image features of the image data supplied from the feature extraction unit 12, particularly the image features of the human face included in the image. to estimate the emotions of
The processing from S3 to S11 is the same as that of Embodiment 1 shown in FIG. and may be integrated into the feature space of the image. Further, in S7, the scene presentation unit 15 may present the analysis result of the emotion analysis unit 16 together with the image of the candidate for the abnormal scene.
In FIG. 13, S2 and S13 may be executed concurrently, and S13 may be executed before S2 in chronological order.
10, the abnormality detection device 1 according to the present embodiment omits the determination and branching process of S3, and regardless of the number of learning data stored in the learning data DB 2, the abnormality detection device 14 of S4 You may proceed to the abnormal scene determination process.

FIG. 14 is a diagram showing an output example of emotion analysis results presented by the scene presentation unit 15 after the emotion analysis unit 16 of the abnormality detection device 1 analyzes the image of the video data.
Referring to FIG. 14, a bounding box 131 is displayed around the human face in the image, indicating that the human face object has been detected. The reliability of emotion estimated from the human face within this bounding box 13 is displayed in the upper left of the output window.
In the example of FIG. 14, the reliability of anger is calculated to be the highest at 37.45%, but it is assumed that the facial expression of the person surrounded by the bounding box 131 does not indicate anger and is neutral. . In this case, the operator presented with the image of FIG. 14 visually confirms the expression of the person's face in the presented image, and presents the scene with the result of confirmation that it is not an abnormal scene (that is, the scene is normal). can be entered in section 15. Alternatively, the emotion analysis unit 16 may set a predetermined threshold for the reliability score, and if the reliability score for anger is equal to or less than the threshold, it may not be detected as an abnormal scene candidate.

As described above, according to the present embodiment, the anomaly determiner of the anomaly detection apparatus uses multimodal information of both audio features and image features of video data, particularly facial emotional features of people. , it is sufficient to execute the determination of the abnormal scene for the candidate of the abnormal scene determined to have a high probability of being an abnormal scene. Therefore, even if the number of samples of learning data is small, the abnormality determination process can be executed with high accuracy and low load.
Similarly, according to the present embodiment, the scene presenting unit of the abnormality detection device extracts an abnormal scene from multimodal information of both audio features and image features of the video data, especially the emotional features of a person's face. It is sufficient to present the operator with the candidate of the abnormal scene determined to have a high probability. Therefore, the burden on the operator in confirming abnormal scenes is further reduced.

<Hardware configuration of abnormality detection device>
FIG. 15 is a diagram showing a non-limiting example of the hardware configuration of the abnormality detection device 1 according to this embodiment.
The anomaly detection device 1 according to this embodiment can be implemented on any single or multiple computers, mobile devices, or any other processing platform.
Referring to FIG. 15, an example in which abnormality detection device 1 is implemented in a single computer is shown, but abnormality detection device 1 according to the present embodiment is implemented in a computer system including a plurality of computers. you can A plurality of computers may be interconnectably connected by a wired or wireless network.

As shown in FIG. 15, the abnormality detection device 1 may include a CPU 21, a ROM 22, a RAM 23, an HDD 24, an input section 25, a display section 26, a communication I/F 27, and a system bus . The anomaly detection device 1 may also include an external memory. PC3 may also have a configuration similar to that of FIG.
A CPU (Central Processing Unit) 21 comprehensively controls the operation of the abnormality detection device 1, and controls each component (22 to 27) via a system bus 28, which is a data transmission line.
The abnormality detection device 1 may also include a GPU (Graphics Processing Unit). The GPU has higher computational power than the CPU 21, and running multiple or multiple GPUs in parallel provides higher processing performance, especially for video processing applications that use machine learning as in the present embodiment. do. A GPU typically includes a processor and shared memory. Each processor obtains data from a high-speed shared memory and executes a common program to perform a large amount of the same kind of computation at high speed.

A ROM (Read Only Memory) 22 is a non-volatile memory that stores control programs and the like necessary for the CPU 21 to execute processing. The program may be stored in a non-volatile memory such as a HDD (Hard Disk Drive) 14 or an SSD (Solid State Drive) or an external memory such as a removable storage medium (not shown).
A RAM (Random Access Memory) 23 is a volatile memory and functions as a main memory of the CPU 11, a work area, and the like. That is, the CPU 21 loads necessary programs and the like from the ROM 22 to the RAM 23 when executing processing, and executes the programs and the like to realize various functional operations.

The HDD 24 stores, for example, various data and information necessary for the CPU 21 to perform processing using programs. Further, the HDD 24 stores various data, various information, and the like obtained by the CPU 21 performing processing using programs and the like, for example.
The input unit 25 is composed of a pointing device such as a keyboard and a mouse.
The display unit 26 is configured by a monitor such as a liquid crystal display (LCD). The display unit 26 is a GUI (Graphical User Interface), which is a user interface for inputting instructions to the parameter adjustment apparatus 1, such as various parameters used in abnormal scene detection processing and communication parameters used in communication with other devices. ) may be provided.

The communication I/F 27 is an interface that controls communication between the abnormality detection device 1 and an external device.
A communication I/F 27 provides an interface with a network and executes communication with an external device via the network. Via the communication I/F 27, images, abnormal scene determination results, abnormal scene confirmation inputs, various parameters, etc. are transmitted/received to/from external devices. In this embodiment, the communication I/F 27 may perform communication via a wired LAN (Local Area Network) conforming to a communication standard such as Ethernet (registered trademark) or a dedicated line. However, the network that can be used in this embodiment is not limited to this, and may be configured as a wireless network. This wireless network includes a wireless PAN (Personal Area Network) such as Bluetooth (registered trademark), ZigBee (registered trademark), and UWB (Ultra Wide Band). It also includes a wireless LAN (Local Area Network) such as Wi-Fi (Wireless Fidelity) (registered trademark) and a wireless MAN (Metropolitan Area Network) such as WiMAX (registered trademark). Furthermore, wireless WANs (Wide Area Networks) such as LTE/3G, 4G, and 5G are included. It should be noted that the network connects each device so as to be able to communicate with each other, and the communication standard, scale, and configuration are not limited to those described above.

At least some of the functions of the elements of the abnormality detection device 1 shown in FIG. 1 can be realized by the CPU 21 executing a program. However, at least some of the functions of the elements of the abnormality detection device 1 shown in FIG. 1 may operate as dedicated hardware. In this case, the dedicated hardware operates under the control of the CPU 21 .

It should be noted that although specific embodiments are described above, the embodiments are merely examples and are not intended to limit the scope of the invention. The apparatus and methods described herein may be embodied in forms other than those described above. Also, appropriate omissions, substitutions, and modifications may be made to the above-described embodiments without departing from the scope of the invention. Forms with such omissions, substitutions and modifications are included in the scope of what is described in the claims and their equivalents, and belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1... Anomaly detection apparatus 2... Learning data DB 3... PC 11... Data acquisition part 12... Feature extraction part 13... Abnormal scene candidate detection part 14... Abnormality determination device 15... Scene presentation part 16... Emotion analysis unit 21 CPU 22 ROM 23 RAM 24 HDD 25 input unit 26 display unit 27 communication I/F 28 bus

Claims (16)

a video acquisition unit that acquires video data;
a feature extraction unit that extracts audio features from the video data acquired by the video acquisition unit and extracts image features from the video data;
an abnormal scene candidate detection unit that detects abnormal scene candidates from the video data based on the audio features extracted by the feature extraction unit;
an abnormality determiner for determining whether the abnormal scene candidate detected by the abnormal scene candidate detection unit is abnormal, normal, or otherwise based on the audio feature and the image feature;
When the abnormality determiner determines that the candidate for the abnormal scene belongs to others, the candidate for the abnormal scene is presented via a user interface, and information to be added to the candidate for the presented abnormal scene. and a scene presenting unit that receives input of through the user interface. The information processing apparatus according to claim 1, wherein the abnormal scene candidate detection unit detects the abnormal scene candidate from the video data using unsupervised learning. The abnormal scene candidate detection unit detects the abnormal scene candidates from the video data by directly separating abnormal audio features without generating a model of normal audio features. Item 3. The information processing device according to Item 1 or 2. The information processing according to claim 3, wherein the abnormal scene candidate detection unit separates the abnormal audio features by calculating a path length in an isolation forest of each audio feature. Device. 5. The information processing according to any one of claims 1 to 4, wherein said feature extraction unit extracts an audio feature represented by a Mel frequency spectrogram of audio data in said video data. Device. The feature extraction unit calculates Mel Frequency Cepstrum Coefficients (MFCC) from the audio data, connects the calculated MFCC to the Mel frequency, and extracts the audio feature. 6. The information processing apparatus according to claim 5. The scene presentation unit adds information input via the user interface to the audio features and the image features, and stores the information in a storage device as learning data for the abnormality determiner. The information processing apparatus according to any one of claims 1 to 6. 8. The abnormal scene candidate detection unit causes the abnormality determiner to determine the candidate of the abnormal scene when the number of the learning data stored in the storage device exceeds a predetermined threshold. The information processing device according to . When the number of pieces of learning data stored in the storage device is within a predetermined threshold, the abnormal scene candidate detection unit bypasses the determination by the abnormal device and instructs the scene presentation unit to display the abnormal scene. 9. The information processing apparatus according to claim 7, wherein candidates are presented. The abnormality determiner determines that a difference between the number of abnormal samples and the number of normal samples located near the candidate for the abnormal scene is within a predetermined threshold in a feature space in which the audio features and the image features are integrated. 10. The information processing apparatus according to any one of claims 1 to 9, wherein the candidate for the abnormal scene is determined to be other when the abnormal scene is detected. The information processing apparatus according to any one of claims 1 to 10, wherein the abnormality determiner determines the candidate for the abnormal scene by a k nearest neighbor method. From the image features extracted by the feature extraction unit, using supervised learning, analyze facial emotion included in the video data, and supply the analyzed facial emotion features to the abnormality determiner. further comprising an emotion analysis unit,
12. The information processing apparatus according to any one of claims 1 to 11, characterized by: The emotion analysis unit causes the abnormal scene candidate detection unit to detect an abnormal scene based on the audio feature when the candidate for the abnormal scene is detected from the video data based on the analyzed emotion of the face. 13. The information processing apparatus according to claim 12, wherein the information processing apparatus is executed. An information processing system comprising a server and at least one client device connected to the server via a network,
The server is
a video acquisition unit that acquires video data;
a feature extraction unit that extracts audio features from the video data acquired by the video acquisition unit and extracts image features from the video data;
an abnormal scene candidate detection unit that detects abnormal scene candidates from the video data based on the audio features extracted by the feature extraction unit;
an abnormality determiner for determining whether the abnormal scene candidate detected by the abnormal scene candidate detection unit is abnormal, normal, or otherwise based on the audio feature and the image feature;
When the abnormality determiner determines that the candidate for the abnormal scene belongs to others, the candidate for the abnormal scene is presented via a user interface, and information to be added to the candidate for the presented abnormal scene. a scene presentation unit that receives the input of via the user interface;
a transmitting unit configured to transmit the abnormal scene candidate to the client device;
The client device
a receiving unit that receives the abnormal scene candidate transmitted from the server;
the user interface that presents the candidate for the abnormal scene received by the receiving unit and receives input of information to be added to the candidate for the presented abnormal scene;
An information processing system, comprising: a transmitting unit configured to transmit to the server information to be added to the abnormal scene candidate whose input is received by the user interface. An information processing method executed by an information processing device,
obtaining video data;
extracting audio features from the acquired video data and extracting image features from the video data;
detecting abnormal scene candidates from the video data based on the extracted audio features by unsupervised learning;
determining, by an abnormality determiner, the detected abnormal scene candidate as abnormal, normal, or otherwise based on the audio features and the image features;
When the abnormality determiner determines that the candidate for the abnormal scene belongs to others, the candidate for the abnormal scene is presented via a user interface, and information to be added to the candidate for the presented abnormal scene. and a step of accepting an input of via the user interface. An information processing program for causing a computer to execute information processing, the program causing the computer to:
a video acquisition process for acquiring video data;
A feature extraction process for extracting an audio feature from the video data acquired by the video acquisition process and extracting an image feature from the video data;
Abnormal scene candidate detection processing for detecting abnormal scene candidates from the video data based on the audio features extracted by the feature extraction processing;
an abnormality determination process for determining, by an abnormality determiner, the abnormal scene candidate detected by the abnormal scene candidate detection process as abnormal, normal, or other based on the audio feature and the image feature;
When the abnormality determiner determines that the candidate for the abnormal scene belongs to others, the candidate for the abnormal scene is presented via a user interface, and information to be added to the candidate for the presented abnormal scene. and a scene presentation process for receiving an input of through the user interface.

Images