KR102319583B1 - System for providing interactive content based on image or voice analysis - Google Patents

Abstract

Embodiments are an imaging device configured to shoot a scene having an image or sound, and a server configured to analyze the shot scene to generate interactive content for the scene, wherein the server includes: shooting information obtained from the imaging device a situation analysis module configured to analyze the scene in terms of sound or image to calculate scene information; a situation expression module configured to convert a scene text into a scene text expressing a corresponding scene based on at least one of the shooting information and the scene information; an interactive module configured to generate interactive content in response to the scene text; And it relates to a system for providing interactive content including a server, including an activation module for controlling the operation of the situation analysis module or the interactive module.

Description

System for providing interactive content based on video or audio analysis {SYSTEM FOR PROVIDING INTERACTIVE CONTENT BASED ON IMAGE OR VOICE ANALYSIS}

Embodiments of the present invention relate to a technology for providing interactive content through non-human dialogue, and more particularly, by analyzing images and/or voices obtained by shooting a scene through an imaging device, and It relates to a system and method for providing content suitable for a situation in the form of audio, video, projection, and a combination thereof.

In modern society, the demand for CCTV is increasing due to interest in the field of policing, such as security and criminal investigation, or interest in the field of law, such as litigation materials. For this reason, there is a need for establishing an efficient CCTV monitoring control system.

The conventional CCTV monitoring control system is configured to provide the video captured by the CCTV and the information analyzed thereto to the controller. In addition, there is a limitation in providing only one-way communication for a subject to be filmed, such as a one-way alert centered on a controller.

Registered Patent Publication No. 10-1998963

According to a number of aspects of the present invention, it is possible to provide a system and method for analyzing a voice or video in a shooting scene to determine a corresponding situation, and then providing interactive content suitable for the subject of the scene.

A system for providing interactive content according to an aspect of the present invention includes: an imaging device configured to shoot a scene having an image or sound, and a server configured to analyze the captured scene to generate interactive content for the scene may include. The server may include: a situation analysis module configured to calculate scene information by analyzing the scene in terms of sound or image from the shooting information obtained from the imaging device; a situation expression module configured to convert a scene text into a scene text expressing a corresponding scene based on at least one of the shooting information and the scene information; an interactive module configured to generate interactive content in response to the scene text; And it includes at least one of the activation module for controlling the operation of the situation analysis module or the interactive module.

In an embodiment, the situation analysis module analyzes an image of a scene through an image analysis model, and the image analysis model calculates at least one of detecting or identifying an object in the image, detecting an object's behavior, and a scene situation. machine learned

In an embodiment, the situation analysis module analyzes a sound of a scene through a sound analysis model, and the sound analysis model calculates at least one of a type of a sound signal, a sub-type, and a situation of a scene in which the sound signal is generated. machine learned to do

In an embodiment, the situation analysis module may be further configured to calculate a situation different from the normal situation as a situation of the shooting scene when the situation of the shooting scene does not match the normal situation as a result of the image or sound analysis.

In an embodiment, the situation analysis module may be configured to analyze the image of the scene when the sound of the scene is calculated as a dialog.

In an embodiment, the situation expression module may be further configured to generate a scene text having a context to represent an image based on the image-related information in the scene information.

In an embodiment, the scene text may be generated through a captioning model that receives object-related information of the image as an input and converts it into text for expressing the image.

In an embodiment, the interactive module may be further configured to generate interactive content associated with the scene text through a chatbot program, and convert the interactive content into sound.

In an embodiment, the interactive module may be further configured to convert the sound of the scene into scene text before generating the interactive content.

In an embodiment, the interactive module may generate the dialogue related to the scene text as interactive content based on a pre-stored dialogue rule of the chatbot program.

In an embodiment, the interactive module may generate interactive content for the scene through a chatbot program having a machine learning model that generates output text suitable for input text based on a training sample.

In an embodiment, the situation analysis module may be further configured to perform analysis on the scene after converting the sound of the scene into text.

In an embodiment, the activation module may perform analysis of the scene when the shooting information of the imaging equipment is acquired at a specified time or a specified place.

In an embodiment, the activation module may control the initiation of an operation of the interactive module based on the necessity of generating the interactive content.

In an embodiment, the need may be based on at least one of the subject's behavior, the subject's voice type, the shooting situation type, and a combination thereof.

In an embodiment, the interactive content may be provided through the video device in the form of voice, video, projection, or a combination thereof.

A system according to an aspect of the present invention provides intelligent and interactive content that predicts information desired by a target based on the target's voice/action through voice/video acquired from the target and provides it through conversation.

In particular, the system is not limited to detecting abnormal behavior based on the basic behavior pattern of the subject, and generates interactive content suitable for the situation of the scene by analyzing the shooting scene based on the video and audio in the shooting scene. .

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

In order to more clearly explain the technical solutions of the embodiments of the present invention or the prior art, drawings necessary for the description of the embodiments are briefly introduced below. It should be understood that the drawings below are for the purpose of explaining the embodiments of the present specification and not for the purpose of limitation. In addition, some elements to which various modifications such as exaggeration and omission have been applied may be shown in the drawings below for clarity of description.
1 is a conceptual diagram of an interactive CCTV system according to an embodiment of the present invention.
2A to 2C are diagrams illustrating an operation result of a situation expression module according to an embodiment of the present invention.
3 is a conceptual diagram of an interaction module according to an embodiment of the present invention.
4 is a diagram for explaining a process of generating interactive content when a voice signal is a non-dialog, according to an embodiment of the present invention.
5 is a diagram for explaining a process of generating interactive content when a voice signal is a dialog, according to an embodiment of the present invention.
6 and 7 are system conceptual diagrams modified from the interactive content providing system of FIG. 1 according to various embodiments of the present invention.

The terminology used herein is for the purpose of referring to specific embodiments only, and is not intended to limit the invention. As used herein, the singular forms also include the plural forms unless the phrases clearly indicate the opposite. The meaning of "comprising," as used herein, is intended to specify a particular characteristic, region, integer, step, operation, element, component, and/or part, and is intended to specify another characteristic, region, integer, step, operation, element, component and/or component. or the presence or addition of parts.

Although not defined otherwise, all terms including technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention belongs. Commonly used terms defined in the dictionary are additionally interpreted as having a meaning consistent with the related technical literature and the presently disclosed content, and unless defined, they are not interpreted in an ideal or very formal meaning.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

The interactive content providing system according to embodiments of the present invention analyzes an image and/or voice acquired from a photographing device to provide a communication (eg, alarm, alarm, information provision, etc.) suitable for the scene in a natural language. It can be expressed by various means (sound output device, display device, projection device, etc.).

1 is a conceptual diagram of an interactive CCTV system according to an embodiment of the present invention.

Referring to FIG. 1 , the interactive content providing system 1 includes a server 10 in electrical communication with a photographing device 5 .

The interactive content providing system 1 according to embodiments may have aspects that are entirely hardware, entirely software, or partially hardware and partially software. For example, the interactive content providing system 1 may collectively refer to hardware equipped with data processing capability and operating software for driving the same. As used herein, terms such as “unit”, “system” and “device” are intended to refer to a combination of hardware and software run by the hardware. For example, the hardware may be a data processing device including a central processing unit (CPU), a graphic processing unit (GPU), or another processor. In addition, software may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

The imaging equipment 5 is configured to receive a video signal and a sound signal. The imaging device 5 may acquire an image or sound (eg, a voice of a subject) constituting a scene (eg, a person appears). To this end, the imaging equipment 5, for example, a photographing unit for photographing an image or a moving picture; and a sound input unit (eg, a microphone unit, etc.) for receiving a sound.

In the present specification, the voice signal is not limited to only the signal output from the human oral structure. That is, the voice signal refers to a signal including other sound signals in addition to the human voice.

In addition, the imaging equipment 5 is further configured to transmit and receive data to and from the server 10 . The video equipment 5 transmits the input of the received video signal and sound signal to the server 10 . To this end, the imaging device 5 further includes a transmission/reception unit capable of electrical communication with the server 10 through a wired/wireless electrical connection. The transceiver unit accesses the server 10 through various network interfaces such as a wide area/local area network or a local access method through wired/wireless according to the communication protocol of the server 10 .

Also, the video device 5 receives and outputs interactive content (eg, a dialog) generated based on the video signal and the sound signal from the server 10 . The interactive content is output as sound or image. In an embodiment, information (ie, interactive content) may be provided by displaying a natural sound automatically generated by the server 10 or on a display device. Alternatively, the interactive content may be provided to a target located at a predetermined interval from the imaging device 5 in the form of projection, depending on circumstances.

To this end, the video equipment 5 includes, for example, a display unit for outputting interactive content as an image; and at least one of a sound unit (eg, a speaker, etc.) outputting sound. In some embodiments, the display unit may be a projection unit that outputs in a projection method.

Hereinafter, for clarity of explanation, a sound input unit; transceiver unit; The present invention will be described in more detail through embodiments having a CCTV including a sound unit as the video equipment 5 .

The imaging device 5 transmits shooting information indicating a situation, such as a scene within a shooting range, to the server 10 . The photographing information includes image (or video) and sound information based on a physical signal within a photographing range. In addition, the photographing information further includes an identifier of the imaging device 5 that photographed an image, a sound, and the like, a photographing time, and the like. When the shooting information relates to a moving picture, the shooting information further includes related information (frame identifier, etc.) for each frame.

The server 10 is a data processing device including a processor, which receives an image signal and/or a sound signal in a shooting scene from the imaging device 5, processes the signal, and analyzes the scene at the time the signal was obtained. Then, interactive content suitable for the situation shown in the scene is generated.

In an embodiment, the server 10 includes at least one of a context analysis module 100 , a context expression module 300 , an interactive module 500 , and an activation module 700 .

After the server 10 receives an input from the video equipment 5, the interactive content providing system 1 includes a situation analysis module 100, a situation expression module 300, an interactive module 500, and an activation module ( 700), when interactive content is derived through at least one of the interactive content can be provided.

On the other hand, the interactive content of the interactive content providing system 1 is not necessarily limited to the results of data processing through all modules, and data processing results by at least some of a plurality of modules may be provided.

However, each of the units 100 , 300 , 500 , and 700 constituting the server 10 is not necessarily intended to refer to physically distinct separate components. Accordingly, in FIG. 1 , each of the parts 100 , 300 , 500 , and 700 is shown as a separate block that is distinguished from each other, but according to an embodiment, each of the parts 100 , 300 , 500 , and 700 includes some or all of them. may be integrated in the same device. In addition, each of the units 100 , 300 , 500 , and 700 merely functionally classifies the devices according to the operations performed by the computing devices in which they are implemented, and does not necessarily mean separate elements separated from each other.

Accordingly, the system architecture of the interactive content providing system 1 of FIG. 1 is only an exemplary description of a process of generating and outputting interactive content by analyzing video and audio, and various modifications Even in the case of having a structured structure, interactive content can be generated and output. The modified system architecture of the interactive content providing system 1 of FIG. 1 will be described in more detail with reference to FIGS. 6 and 7 below.

The situation analysis module 100 is configured to analyze the image and/or audio in the shooting scene to determine the situation of the shooting scene.

The context analysis module 100 is configured to perform at least one of an image-based analysis and a voice-based analysis. For example, the situation analysis module 100 is configured to simultaneously utilize image-based analysis and voice-based analysis, or is configured to selectively utilize it. For example, as shown in FIG. 1 , the situation analysis module 100 may include the image analysis module 110 and the audio analysis module 130 to analyze image information and audio information.

The image analysis module 110 uses the image signal received by the server 10 from the image equipment 5 to perform object detection, object recognition, and abnormality detection (object detection) of a scene related to the image signal. and to perform image-based computer vision or image processing operations, such as anormaly detection. Here, object detection includes detection of a designated object (eg, detection of a face) and detection of a user-defined object, and object recognition includes detection of a face, identification, and the like.

The image analysis module 110 may perform an image capturing location, detection or identification of an object in an image, and detection of an object's behavior. For this reason, the image analysis module 110 includes object identification information (person, car, animal, etc.) in the image, the number of objects, the behavior of the object, the appearance of the object (clothes, accessories, color, material, etc.), the location of the object, etc. can be calculated as an image analysis result.

In addition, the image analysis module 110 is further configured to classify the situation of the shooting scene. The photographing situation includes, for example, a normal situation and an abnormal situation (dangerous, emergency, etc.). For this reason, the image analysis module 110 includes an abnormal situation, abnormal situation identification information (danger, violence, wandering, falling of the target), and the like.

The image analysis module 110 performs an image processing operation through the image analysis model. Here, the image analysis model is a machine learning model, such as a deep learning model. The image analysis model consists of, for example, a machine learning model structure including a Convolution Neural Network (CNN), Recurrent Neural Networks (RNN), Long Short Term Memory (LSTM), Generative Adversarial Network (GAN) structure, and the like.

Through such an image analysis model, the image analysis module 110 may detect and recognize an object in an image, and may detect and predict an abnormality (eg, abnormal behavior of an object, etc.) in an image.

The image analysis module 110 has a model trained to perform a plurality of image analysis operations, or has a plurality of models (eg, a model for abnormal situation detection, a model for face recognition, etc.) for a specific analysis operation. can

In an alternative embodiment, the image analysis module 110 may be configured to perform an analysis suitable for the purpose of the imaging device 5 .

In one example, when the interactive content providing system 1 has the image equipment 5 for providing information, the image analysis module 110 is configured to perform analysis for object detection. In another example, when the interactive content providing system 1 has the video equipment 5 for an alert alarm, the situation analysis module 100 is configured to perform an analysis for object recognition.

The voice analysis module 130 is configured to perform voice-based analysis on some or all of the voices related to the shooting scene, including human voices. That is, the voice analysis module 130 mainly analyzes the human voice among the input voices, but also analyzes other voices (eg, background sound).

The voice related to the shooting scene includes a human voice and an object voice. Here, the human voice includes a conversation of a specific person and a plurality of loud voices. The object's voice includes an ambulance voice and the like. The analysis result of the voice related to the shooting scene may be used to determine the scene of the scene.

For such a voice-based analysis, the voice analysis module 130, the voice signal in the shooting scene received by the server 10 from the video equipment 5, the situation of the object in which the voice signal is generated (eg, dangerous situation) is configured to perform a voice processing operation to determine

In one embodiment, the voice analysis module 130 is further configured to calculate the voice characteristics of the target, which is the sound source, from the voice signal of the shooting information. The voice characteristics include, for example, the frequency, amplitude, and waveform of the voice signal, the interval between the voice signals, and the like.

Also, the voice analysis module 130 calculates the type of the voice signal based on the voice signal and/or the voice characteristics. The type of the voice signal includes a dialog in which the voice signal is a conversation for a specific purpose, or a non-dialog voice such as a background sound.

In some embodiments, the voice analysis module 130 may further calculate a specific type of subtype. When the type of the voice signal is a dialog, the voice analysis module 130 calculates the type of the corresponding dialog as a sub-type. The sub-type includes, for example, an inquiry under normal circumstances, a request under an emergency, and the like.

The voice analysis module 130 performs the above-described voice processing operation through the voice analysis model. Here, the voice analysis model has a structure similar to the above-described image analysis model, but is machine-learned using voice data as training data.

The situation analysis module 100 calculates a situation different from the normal situation as the situation of the shooting scene when the situation of the shooting scene does not match the normal situation as a result of video or audio analysis.

For example, when no abnormal situation is detected and the type of the shooting situation according to the image analysis is calculated as a normal situation, but the type of the shooting situation is calculated as an emergency situation as a result of the voice analysis, the situation analysis module 100 is different from the normal situation. The situation (that is, an emergency situation) is judged as the situation of the shooting scene.

Alternatively, if the behavior of the object is detected as “assault” and the type of shooting situation according to the image analysis is calculated as a dangerous situation, but the input voice is small, the type of the shooting situation according to the voice analysis is calculated as a normal situation, situation analysis The module 100 determines a situation different from the normal situation (ie, a dangerous situation) as the situation of the shooting scene.

The analysis result of the situation analysis module 100 may be provided to the situation expression module 300 or the interactive module 500 as scene information.

The situation expression module 300 is configured to perform a situation analysis operation of expressing a situation appearing in a scene as a text having a context based on an image or an analysis result of the image in the shooting scene. For example, as shown in FIG. 1 , the situation expression module 300 includes a video-text conversion module 310 and an audio-text conversion module 330 . The situation analysis operation of expressing the situation of the scene as a text having a context is distinguished from the situation determination operation of simply determining whether the situation of the scene is an abnormal situation.

In the situation analysis operation, the operation of converting image information into words or sentences may be referred to as captioning. The conversion module 310 is configured to convert image information into text, and interprets the scene in terms of the image. To this end, the conversion module 310 may be configured to apply an image captioning method for a still image or a video captioning method for a moving image to an image analysis result.

Image information for one still image is converted into text such as sentences by the image captioning method. Meanwhile, by the video captioning method, information about the video is converted into text based on continuous frames in the video. Here, the text includes a word or a sentence.

In an embodiment, the conversion module 310 converts image information into text through a captioning model. Here, the captioning model is configured of a machine learning model structure including, for example, a Convolution Neural Network (CNN), Recurrent Neural Networks (RNN), Long Short Term Memory (LSTM) structure, etc., but is not limited thereto. The captioning model is machine-learned in advance through a training sample including image-related information and/or text representing the image.

Also, in some embodiments, the captioning model may be machine-learned by an Attention mechanism that causes the corresponding model to focus only on important parts for text conversion. The important part includes, for example, information related to the object of the image (eg, object identification information, behavior, etc.).

Through the captioning model, the conversion module 310 may express a scene at the time of shooting as text having a context.

2A to 2C are diagrams illustrating an operation result of a situation expression module according to an embodiment of the present invention.

When the image of FIG. 2A is input to the server 10 , the conversion module 310 converts the analysis result of the image of FIG. 2A into text having a context of “a man in a black shirt plays a guitar”.

When the image of FIG. 2B is input to the server 10, the conversion module 310 converts the image analysis result of FIG. 2B into text "a worker in an orange safety vest is working on the road".

When the image of FIG. 2C is input to the server 10, the conversion module 310 converts the image analysis result of FIG. 2C into text "two people play with Lego".

In the video captioning method, information on a video may be converted into text based on information on each frame corresponding to a still image.

In an alternative embodiment, the context expression module 300 may express a scene in text in one or more languages (eg, Korean, English, etc.). For example, with respect to the image of FIGS. 2A to 2C , the situation expression module 300 is “man in black shirt is playing guitar”, “worker in orange safety vest in working on road”, “two people are playing with lego”, respectively. toy” respectively.

In some embodiments, the interactive content providing system 1 may be configured to use a deep learning model for individual image interpretation for each language.

In some other embodiments, the interactive content providing system 1 may be configured to use a deep learning model configured to simultaneously learn multiple languages and output multiple languages as one model.

For this reason, the interactive content providing system 1 can interact with each other in response to various languages.

In an embodiment, the conversion module 310 may be configured to perform a captioning operation suitable for the purpose of the imaging device 5 . In one example, when the interactive content providing system 1 has the video equipment 5 for providing information, the conversion module 310 is configured to perform a captioning operation on a still image.

The voice-to-text conversion module 330 is configured to express a situation appearing in a scene as text based on a voice signal or analysis result of the shooting scene. The conversion module 330 is configured to convert voice information into text, and interprets the scene in terms of voice. To this end, the conversion module 330 is configured to convert the speech of the target received by the server 10 through a speech-to-text program (STT, Speech-to-text) into text.

The conversion module 330 directly receives the voice signal received by the server 10 and converts it into text, or one of the voice signal received by the server 10 and the voice analysis result received from the voice analysis module 130 . Based on the above, a text representing a scene is generated.

The situation analysis result (eg, text conversion result) of the situation expression module 300 may be transmitted to the interactive module 500 .

The interactive module 500 generates interactive content suitable for a shooting scene (eg, a shooting scene) based on shooting information (input image or voice), a situation analysis result, an analysis result, and a combination thereof. Thereby the server 10 is configured to perform an interaction with the scene (eg, an object in the scene).

3 is a conceptual diagram of an interactive module according to an embodiment of the present invention.

Referring to FIG. 3 , the interactive module has different data processing paths according to the type of input.

In an embodiment, when receiving the scene text generated by the situation expression module 500 , the interactive module 500 may generate interactive content in response to the received text.

The interactive module 500 may execute one or more computer programs for generating interactive content. The computer program may be, for example, a natural language program (including, for example, a Nature Langague Processing (NLP) and/or a Nature Language Understanding (NLU) program) supporting machine learning or artificial intelligence or various non-humanoid chat programs. . For example, in the interactive module 500, a chatbot program 530 is installed to generate and provide interactive content.

Upon receiving the text from the image, the interactive module 500 generates a response suitable for the text from the received image, for example, through the chatbot program 530 . The response is text data with context for interactive content (“interactive content”).

In one embodiment, the interactive module 500 generates interactive content via a rule-based chatbot 530 . The rule-based chatbot is configured to, when a text specified in a preset rule is input, output a result corresponding to the input according to the rule. The rule-based chatbot may be configured of, for example, a Google DialogFlow API, an Amazon LEX API, or the like.

For example, when a predetermined scene text is generated in the situation expression module 300 and received by the interactive module 500 , the chatbot program generates an output according to a predetermined rule.

In another embodiment, the interactive module 500 generates interactive content through a machine learning based chatbot (Machine Learning based Chatbot). Machine-learning-based chatbots consist of a dialog model that produces output text suitable for input text based on a large-scale training sample. The training sample includes a dialog, which is a conversation for a specific purpose. The machine learning-based chatbot may be configured of, for example, a seq2seq model, pLSA, or the like.

In alternative embodiments, interactive module 500 may have various algorithm-based chatbots that output output text suitable for input text.

The interactive content generated by the chatbot program 530 is converted into voice through a text-to-speech (TTS) program 550 to generate interactive content.

In another embodiment, the interactive module 500 may receive a voice signal and generate interactive content in response to the received voice signal.

3 , the interactive module 500 may further include a voice-to-text conversion module 510 . In this case, the interactive module 500 converts the voice signal received by the server 10 into text, and then inputs the converted text into the chatbot program to generate interactive content suitable for the received voice signal. . In some embodiments, the transform module 510 may perform at least some operations of the transform module 330 . In addition, the conversion module 510 may be further configured to detect a context of a voice or an intent of an object appearing in the voice.

As an example, when a human voice is received through the text-based intention analysis model, the conversion module 510 may calculate the human intention indicated in the corresponding voice. Here, the intention analysis model is a pre-trained machine learning model to designate a specific intention (path guidance, weather, danger, time, etc.) in advance through a training sample obtained by converting speech into text. Then, when a human voice is received, the conversion module 510 may convert the voice into text and then apply it to the intention analysis model to output the intention of the person indicated in the voice.

Meanwhile, the transformation module 510 may have another machine learning model. For example, it is possible to have a machine-learned model to output the context or intent of speech by a topic modeling method or a text classification method.

For this reason, if the input voice is calculated with a specified specific intention, interactive content may be provided as a dialogue text previously related to the calculated intention. Alternatively, the calculated intention and the received voice input may be transmitted to the situation analysis module 100 .

The processes of analyzing the context or intention of the voice and converting the conversation text previously related to the analyzed context or voice into voice are Apple's Siri, Amazon's Echo, or Korean Patent Application Laid-Open No. 10-2019- Since it is well known to those skilled in the art through No. 0099061 and the like, a detailed description will be omitted herein in order to clarify the gist of the present invention.

As such, the interactive module 500 may generate interactive content by itself for at least a voice signal. When this interactive module 500 receives a predetermined input (eg, text converted to speech), it determines that interactive content needs to be generated, and then a predetermined output associated with the received input. to create interactive content.

In some embodiments, the operation of converting the voice signal into text in the server 10 may be performed only by the conversion module 510 . Here, the situation expression module 300 converts only image information into text.

In addition, the interactive module 500 generates a subsequent post-interactive content when it is necessary to continue the conversation after generating the pre-interactive content and transmits the post-interactive content to the video device 5 . send.

In one embodiment, when it is necessary to continue the conversation, the converted text based on the voice of the received subject within a predetermined time (eg, several seconds or tens of seconds) from the time the system 1 output the previous interactive content Including the case of a preset input of the interactive module 500 .

Meanwhile, the interactive module 500 may provide other data other than the interactive content converted into voice to other components in the server 10 .

In an embodiment, the interactive module 500 may provide the speech text of the scene in which the input speech is converted into text by the conversion module 510 to the context analysis module 100 . In this case, in generating the interactive content, the interactive module 500 generates interactive content based on the situation analysis result and the scene text obtained from the input voice instead of the text output from the situation expression module 300 . create

The activation module 700 controls the operation of at least one of the above-described modules 100 , 300 , or 500 . In the activation module 700, a command for a control operation is stored in advance.

When a situation analysis is required, the activation module 700 analyzes the situation based on the shooting information.

Cases in which the situation analysis is necessary include a case in which shooting information is acquired at a specified time, a case in which shooting information is acquired at a specified place, or a case where re-analysis of the situation is required.

For example, the activation module 700 causes the situation analysis module 100 to operate when the shooting information (eg, audio or video signal) of the video equipment 5 is obtained at a designated place. When the shooting information includes the identifier of the video equipment 5 that was photographed, the activation module 700 obtains the identifier of the video equipment 5 together with the video signal or sound signal (eg, voice signal), and the identifier stored in advance - It is determined whether the shooting information received by the server 10 through the location table is acquired at a designated place.

The activation module 700 causes the situation analysis module 100 to operate when the shooting information (eg, audio or video signal) of the imaging device 5 is acquired within a specified time range. When the shooting information includes a shooting time, the activation module 700 determines whether an image is acquired at a specified time based on the shooting time acquired together with the signal of the imaging device 5 .

The activation module 700 causes the situation analysis module 100 to re-analyze the situation of the shooting information when the interactive module 500 fails to generate interactive content for the input. If it is not possible to generate interactive content for the input even as a result of the re-analysis, the system 1 may end the conversation or induce input of other shooting information.

In this way, the activation module 700 receives the shooting information and controls whether to start the situation analysis operation.

In some embodiments, the activation module 700 is set to control the start of the operation of the situation analysis module 100 only for a specific signal of the shooting information. For example, for the voice signal, the activation module 700 determines whether the situation analysis module 100 (eg, the voice analysis module 130) determines whether the voice signal is a voice signal, or a dialog when the voice signal is a voice signal, and furthermore, a sub of the dialog. Let's analyze what the type is. If the audio signal input through the video equipment 5 is a dialog, the activation module 700 causes the situation analysis module 100 (eg, the video analysis module 110) to perform an image analysis operation. , when it is determined that the input voice is a human voice, the activation module 700 transmits the input voice and the type determination result (eg, the voice is a human voice and a dialog) to the interactive module 500 . Then, it is possible to interact with the object uttering the voice centering on the interactive module 500 . For example, an interaction with a target uttering a voice may be performed only by the interactive module 500 (or the activation module 700 ) without using the operation result of the situation analysis module 100 . This case will be described in more detail with reference to FIG. 5 below.

In addition, the activation module 700 controls whether to start the creation of interactive content based on the analysis result of the shooting information. The activation module 700 determines whether to start the operation of the interactive module 500 based on the necessity of generating interactive content.

The necessity of generating the interactive content is based on the behavior of the target, the type of the target's voice, the type of shooting situation, and the like. When the server 10 obtains at least one of the elements through data processing, it may be determined that there is a need.

In one example, the activation module 700 determines that generation is necessary when the type of the shooting situation is a non-normal situation such as an emergency situation.

In another example, when the voice signal is calculated as a dialog, the activation module 700 determines that generation is necessary. In addition, the activation module 700 may determine that there is a need for generation further based on a sub-type of the dialog, such as an inquiry or a request.

As such, when it is determined that there is a need to generate interactive content, the context expression module 300 generates text that interprets the context. Then, interactive content may be generated based on the text.

The activation module 700 prevents unnecessary data processing (eg, image analysis, etc.) for all input signals, thereby minimizing excessive resource use and enabling efficient system operation.

The server 10 transmits the generated interactive content to the video device 5, and the video device 5 outputs the received interactive content.

The interactive content providing method according to an aspect of the present invention may be performed by a computing device including a processor. In an embodiment, the interactive content providing method may be performed by some or all of the system 1 .

4 is a diagram for explaining a process of generating interactive content when a voice signal is a non-dialog, according to an embodiment of the present invention.

The system 1 can create interactive content provision, for example, when the voice of the shooting information is non-dialog, such as situational awareness.

In FIG. 4 , it is assumed that the operation of the situation analysis module 100 , the situation expression module 300 , and the interactive module 500 is started by the activation module 700 .

The server 10 analyzes the shooting information acquired by the imaging device 5 and calculates scene information (S410).

From the shooting information of the imaging device 5A of FIG. 4 , scene information such as “playground” as the shooting location, “person” and “3 people” as the object type and number, and “assault” as the behavior of the object is calculated.

From the shooting information of the imaging device 5B, the shooting location is “roadside”, the detected object type is “person”, and the number of “car” objects is “1 person” and “7 units”, respectively, and the appearance of the object (person) Scene information such as “black clothes” for the characteristic and “lorowing next to the vehicle” for the behavior of the object (person) is calculated.

From the shooting information of the imaging equipment (5C), the shooting location is “in front of the store”, the shooting time is “dawn”, the type and number of detected objects are “person”, “1 person”, and the appearance characteristics of the object are “semi-nude” , the behavior of the object, such as "roaming" scene information is calculated.

From the shooting information of the imaging equipment (5D), the shooting location is “crosswalk”, the detected object type is “person”, “traffic light”, the number of objects is “2 people”, the recognized object is “child”, For each object's behavior, scene information such as “dangerous behavior” and “red light on” is calculated.

Then, the calculated scene information is converted into text having a context for expressing the captured scene (S430).

The situation photographed from the video equipment 5A of FIG. 4 is converted into text “two people are assaulting one person at the playground”.

The situation photographed by the video equipment 5B is converted into text saying “one person is wandering next to a car on the roadside”.

The situation captured by the video equipment (5C) is converted to the text “A semi-nude person is roaming in front of the store at dawn”.

The situation filmed by the video equipment (5D) is converted into text that says, “Two children are acting dangerously at the red light at the crosswalk.”

Then, in response to the scene text generated in step S300, interactive content for the corresponding situation is generated (S450). In FIG. 4 , interactive content is generated by interacting with at least a captured image.

Regarding the situation photographed from the imaging device 5A in Fig. 4, "There! You must not assault people! The police have arrived!”, interactive content with interactive content as the context is created.

Regarding the situation filmed by the video equipment (5B), “The person in black! Do you have your own vehicle? Do you have any problems?” or “The one in black! CCTV is being filmed. If it is not your vehicle, it will be considered suspicious behavior.” Interactive content with interactive content as the context is created.

Regarding the situation filmed by the video equipment (5C), “Currently, your attire and behavior are not normal. What's going on?” or “You are acting suspiciously. If there is no response to the action, the police will be dispatched.” Interactive content with interactive content as the context is created.

Regarding the situation filmed from the video equipment (5D), interactive content is created with interactive content as the context, “Please do not cross the crosswalk, please step back from the roadside and refrain from dangerous actions because the crosswalk is a red light.”

The interactive content generated in step S450 is provided to the target through the video device 5 (S460). When the interactive content is converted into voice and provided to the video device 5 , the video device 5 provides the interactive content through the sound unit.

Also, following the interactive content generated in step S450, an interaction (eg, a conversation between the object and the system 1) may be continued (S470). In step S470, at least one of the above-described steps S410 to S450 is performed.

5 is a diagram for explaining a process of generating interactive content when a voice signal is a dialog, according to an embodiment of the present invention.

The system 1 may generate interactive content provision when, for example, the voice of the shooting information is a dialog, such as a conversation situation.

In FIG. 5 , similarly to FIG. 4 , it is assumed that the operation of the situation analysis module 100 , the situation expression module 300 , and the interactive module 500 is started by the activation module 700 . For example, the activation module 700 determines whether the received voice signal is a human voice, for example, and transmits the signal and/or the determination result to the interactive module 500 or the situation analysis module 100 .

The voice acquired by the video equipment 5 may be converted into text before analysis (S501).

For example, the input voice of the video equipment 5E is converted into text “How do I go to the city hall?” by the conversion module 510 .

The input voice of the video equipment 5F is converted into text “Help!” by the conversion module 510 . Next, the photographing information for the voice of step S501 is analyzed (S510). In some embodiments, after the audio signal is first analyzed, when the audio signal is a dialog, additional analysis (eg, image signal analysis) of the shooting information may be performed.

Based on the audio signal in the shooting information of the video equipment 5E, the audio type is analyzed as a dialog, and the sub type is “inquiry”. Then, based on the shooting information other than the voice signal, the shooting location is “City Hall Station Exit 10”, the detected object type and number are “person”, “1 person”, and the appearance characteristic of the object is “black clothes” The result is produced.

Based on the audio signal in the shooting information of the video equipment 5F, the audio type is analyzed as a dialog, and the sub type is “request”. Then, based on the shooting information other than the voice signal, the shooting location is “Wolgok Station Exit 10”, the detected object types and number are “person”, “2 people”, and the appearance characteristics of each object are “black clothes”, respectively. ”, “blue clothes”, and the situation type is “critical”.

Then, interactive content is generated based on at least one of the scene text converted from the input voice and the analysis result of the corresponding situation ( S550 ). Here, since the scene text is an input voice, a captioning operation for expressing the situation as text may not be performed.

Regarding the situation photographed from the video equipment 5E, interactive content is created by converting the interactive content into voice saying “I am 100m ahead on the right across the street”.

Regarding the situation filmed from the video equipment (5F), “The police have arrived”, “There is a police station 100m to the right”, or “CCTV is being filmed! The interactive content is created with the interactive content as the context, “Stop tracking the person in black!”

The interactive content generated in step S550 is provided to the target through the video equipment 5 (S560). When the interactive content is voice-converted and provided to the video device 5 , the video device 5 provides the interactive content through the sound unit.

Alternatively, although not shown in FIG. 5 , the interactive module 500 itself may generate and provide interactive content before step S510 , and further generate and provide subsequent interactive content.

The activation module 700 is illustrated as a single component in FIG. 1 , but is not limited thereto. In other embodiments, the system 1 may include a plurality of activation modules 700 according to an input, a control target, and the like.

6 and 7 are system conceptual diagrams modified from the interactive content providing system of FIG. 1 according to various embodiments of the present invention.

The interactive content providing system of FIG. 1 may be configured to include a plurality of activation modules. For example, as shown in FIG. 6 , the system 1 includes a plurality of activation modules 710 , 730 , and 750 for each control target. Here, the system 1 includes a first activation module 710 that controls the operation of the situation analysis module 100 , a second activation module 730 that controls the operation of the situation expression module 300 , and interactive content It may include a third activation module 750 for controlling the operation of the interactive module 500 for generation of .

The interactive content providing system 1 of FIG. 1 partially integrates the functions of the activation module 700 in each component 100 , 300 , and 500 , so that the control target controls the activation module 700 . It can be configured to do it on its own. For example, as shown in FIG. 7 , the system 1 may be modified such that each activation module 710 , 730 , and 750 is integrated within each control object.

It will be apparent to a person skilled in the art that the system 1 may include other components not described herein. For example, it may further include a data input device, an output device such as a display and printing, a storage device such as a memory, a transceiver for transmitting and receiving data through electrical communication, a network, a network interface and a protocol, and the like.

In this way, the system 1 can provide appropriate interactive content such as a guide for the scene by analyzing the sound and image in the shooting scene.

An operation by the interactive content providing system 1 and method according to the embodiments described above may be at least partially implemented as a computer program and recorded in a computer-readable recording medium. For example, embodied with a program product consisting of a computer-readable medium containing program code, which may be executed by a processor for performing any or all steps, operations, or processes described.

The computer may be any device that may be incorporated into or may be a computing device such as a desktop computer, laptop computer, notebook, smart phone, or the like. A computer is a device having one or more alternative and special purpose processors, memory, storage, and networking components (either wireless or wired). The computer may run, for example, an operating system compatible with Microsoft's Windows, Apple OS X or iOS, a Linux distribution, or an operating system such as Google's Android OS.

The computer-readable recording medium includes all types of recording devices in which computer-readable data is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage identification device, and the like. In addition, the computer-readable recording medium may be distributed in a network-connected computer system, and the computer-readable code may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment may be easily understood by those skilled in the art to which the present embodiment belongs.

Although the present invention as described above has been described with reference to the embodiments shown in the drawings, it will be understood that these are merely exemplary and that various modifications and variations of the embodiments are possible therefrom by those of ordinary skill in the art. However, such modifications should be considered to be within the technical protection scope of the present invention. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

Claims (16)

A system for providing interactive content, comprising: an imaging device configured to photograph a scene having an image or sound; and a server configured to analyze the captured scene to generate interactive content for the scene, the server comprising:
a situation analysis module configured to calculate scene information by analyzing the scene in terms of sound or image from the shooting information obtained from the imaging equipment, wherein the scene information includes an analysis result for the scene;
a situation expression module configured to convert a scene text into a scene text expressing a corresponding scene based on at least one of the shooting information and the scene information;
an interactive module configured to generate interactive content in response to the scene text; and
Comprising at least one of the activation module for controlling the operation of the situation analysis module or the interactive module,
The interactive module,
Based on at least one of an analysis result for the scene and a scene text based on the analysis result for the scene, the interactive content is generated as a response to the scene through a chatbot program,
the chatbot program is configured to receive the scene text as input text and produce an output text that is a response to the scene text;
The analysis result for the scene includes identification information of the object in the scene, the number of objects, the behavior of the object, the appearance of the object or the location of the object obtained through the image analysis model of the situation analysis module,
The interactive content is provided through the video device in at least one of audio, video, projection, and a combination thereof.
delete According to claim 1,
The situation analysis module analyzes the sound of the scene through the sound analysis model,
The sound analysis model is
A system according to any one of the preceding claims, wherein the system is machine-learned to calculate at least one of a type, a sub-type of a sound signal, and a situation of a scene in which the sound signal occurs.
According to claim 3, wherein the situation analysis module,
The system further configured to calculate the situation of the shooting scene as a situation different from the normal situation when the situation of the shooting scene is not classified as the normal situation as a result of the image or sound analysis.
According to claim 3, wherein the situation analysis module,
and perform analysis on the image of the scene when the sound of the scene is calculated as a dialog.
delete According to claim 1,
The scene text is generated through a captioning model that receives an image analysis result of the image as an input and converts it into text for expressing the image.
According to claim 1, wherein the interactive module,
The system further configured to convert the interactive content into sound.
The method of claim 8, wherein the interactive module,
before generating the interactive content, convert the sound of the scene into scene text;
The scene text used to create the interactive content is:
and a scene text from an input voice acquired by the situation expression module or a scene text from an input voice acquired by the interactive module.
According to claim 1, wherein the interactive module,
A system for generating a dialogue associated with the scene text as interactive content based on a pre-stored dialogue rule of the chatbot program.
According to claim 1, wherein the interactive module,
A system for generating interactive content for the scene through a chatbot program having a machine learning model that produces output text corresponding to input text as a response to input text based on training samples.
The method of claim 9, wherein the situation analysis module,
After converting the sound of the scene to text, the system further configured to perform analysis on the scene.
According to claim 1, wherein the activation module,
A system for performing analysis of the scene when the shooting information of the imaging equipment is acquired at a specified time or a specified place.
According to claim 1, wherein the activation module,
A system for controlling the initiation of an operation of an interactive module for generating interactive content based on at least one of a behavior of a target, a type of a voice of the target, a type of a shooting situation, and a combination thereof.
delete delete

Images