KR101849092B1 - Method and Apparatus for Detecting Picture Breaks for Video Service of Real Time - Google Patents

Abstract

A method and an apparatus for detecting image breaking in a real-time image service are disclosed. The method comprises the steps of: generating learning data including a plurality of patch regions based on real-time or sample image data; generating a determination reference model by enabling a deep learning algorithm to learn the generated learning data; and detecting an image breaking region by applying the determination reference model to a patch region on a real-time image.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and a device for detecting a screen break of a real-time image service,

The present embodiment relates to a method for detecting a screen break of a real-time image service and an apparatus therefor.

The contents described in this section merely provide background information on the present embodiment and do not constitute the prior art.

The Internet Protocol Television (IPTV) system provides services such as real-time broadcast, VOD service, and electronic commerce content through a display device based on a network.

In the IPTV system, a content providing apparatus compresses (encodes) a video image and transmits the video image to an IP network. A set-top box (STB) decodes the compressed video image received from the IP network, (Decoded) and outputs it to the display device.

Generally, as a method of measuring the image quality of IPTV, there is a method of using a high performance instrument based on a network packet. In this case, the instrument receives the MPEG-2 packet from the network (IP network), demultiplexes it, decodes the demultiplexed video signal, and measures the frame quality. However, a method using a high-performance measuring instrument requires additional equipment in addition to a set-top box, so that additional cost is incurred and it is difficult to detect an error of an image frame in real time. Therefore, there is a need for a method for measuring frame loss in a set-top box without a high-performance instrument.

In general, as a method of measuring an image error using an image frame, a normal image may be set as a reference image, and a different image portion may be determined as an image having an error by comparing with a reference image. However, in the case of using the reference image, there is a problem that it is difficult to detect the error in the image being serviced in real time.

On the other hand, in the case of the No Reference method for analyzing a service image, it is possible to use a method of creating a rule for detecting an error and applying an algorithm, but the detection rate is very low and a continuous upgrade is required.

Accordingly, in order to judge a video error of the current real-time image service, a person is monitoring by looking at the screen. However, such a monitoring method can not accurately detect a video error, and it is practically impossible for a person to perform only monitoring continuously.

In this embodiment, learning data including a plurality of patch areas is generated based on real-time or sample image data, a learning model is generated by learning processing the generated learning data to a Deep Learning algorithm, A method for detecting a screen break in a real-time image service and a device therefor, which detects a screen break area by applying a model to a patch area for a real-time image.

According to an aspect of the present invention, there is provided a method of detecting a screen break in an image error detection apparatus in a real-time image service, the method comprising: acquiring a decision criterion model generated based on a machine learning algorithm; A real-time image acquisition process for acquiring a real-time image frame from an external device; The real-time image frame is divided into a plurality of detection patch areas having a patch size set based on the determination reference model, and a detection area setting process of setting all or a part of the plurality of detection patch areas as an error detection area ; And a screen break detection step of detecting a screen break area in the error detection area using the determination criterion model.

According to another aspect of the present invention, there is provided an apparatus for detecting a screen break in a real-time image service, the apparatus comprising: a model acquisition unit for acquiring a decision criterion model generated based on a machine learning algorithm; A real-time image acquisition unit for acquiring a real-time image frame from an external device; Wherein the real-time image frame is divided into a plurality of detection patch areas having a patch size set based on the determination criterion model and all or a part of the plurality of detection patch areas is set as a detection area, ; And a picture break detection unit for detecting a picture break area in the error detection area using the judgment reference model.

According to another aspect of the present invention, there is provided a data processing apparatus including: a model acquiring step of acquiring a decision criterion model generated based on a machine learning algorithm; A real-time image acquisition process for acquiring a real-time image frame from an external device; The real-time image frame is divided into a plurality of detection patch areas having a patch size set based on the determination reference model, and a detection area setting process of setting all or a part of the plurality of detection patch areas as an error detection area ; And a screen break detection step of detecting a screen break area in the error detection area using the determination criterion model. The computer readable medium having recorded thereon the program for realizing the screen break detection method of the real time image service, Thereby providing a recording medium.

As described above, according to the present embodiment, video errors of a video service provided in real time can be detected.

In addition, it is possible to reduce a waste of manpower for monitoring a real-time image and to detect a screen break automatically using a deep-running algorithm.

In addition, there is an effect that the accuracy of error detection for a real-time image can be improved.

1 is a block diagram schematically showing a video service providing system according to the present embodiment.
2 is a block diagram schematically showing an image error detecting apparatus according to the present embodiment.
3 is a flowchart illustrating a method for detecting a screen break of a real-time image according to the present embodiment.
4 is a block diagram schematically showing a preprocessing unit included in the video error detection apparatus according to the present embodiment.
5 is a flowchart illustrating a preprocessing process for detecting a screen break of a real-time image according to the present embodiment.
6 is an exemplary diagram for explaining an operation of determining a video error in a preprocessing process for detecting a screen break of a real-time video according to the present embodiment.
7A and 7B are diagrams showing learning data for generating a determination criterion model according to the present embodiment.
8A to 8C are views illustrating an operation of detecting a screen break of a real-time image according to the present embodiment.
FIG. 9 is an exemplary view schematically showing an entire operation procedure for detecting screen breakage of a real-time image according to the present embodiment.

Hereinafter, the present embodiment will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically showing a video service providing system according to the present embodiment.

The video service providing system 100 according to the present embodiment includes an image providing server 110, a set-top box 120, a video output apparatus 130, and a video error detecting apparatus 140.

The image providing server 110 provides a real-time or non-real-time image data service based on a network (IP network). More specifically, the uncompressed video image is compressed (encoded) and converted into a transport stream (TS) packet, and then transmitted through a network (IP network) . Here, the uncompressed video image may be composed of a plurality of pictures, and the plurality of images are converted into I frames, B frames and P frames through a compression (encoding) process. The TS packet may be a signal in which a video signal, an audio signal, and a data signal are multiplexed. For example, it may be an MPEG-2 TS in which an MPEG-2 standard video signal, a Dolby AC-3 standard sound signal, etc. are multiplexed. And the image providing server 110 may mean an IPTV headend device.

Meanwhile, the image providing server 110 receives a plurality of pieces of broadcast content information from a content provider, transmits the received content information to a device such as the set-top box 120 and the terminal 130, . ≪ / RTI > That is, the image providing server 110 receives a plurality of pieces of broadcast content information transmitted from the content provider, distributes the received broadcast content information to the set-top box 120 and the subscriber terminal 130, (Video On Demand) and so on.

The set-top box 120 receives image data (packets) transmitted from the image providing server 110 via a relay node (not shown), and demultiplexes and decodes the received image data. The set-top box 120 transmits a video signal obtained by demultiplexing and decoding the video data to the video output device 130 to output a video image.

The video output device 130 is a device for receiving and displaying video signals from the set-top box 120. The video output device 130 may be a portable display device, a smart phone, a monitor, a TV, or the like. The video output device 130 may be connected to the set-top box 120 through a wired connection, but may also be connected to a wireless LAN signal, a Bluetooth communication,

Although the video output apparatus 130 is described as a separate apparatus from the set-top box 120, the video output apparatus 130 is not limited thereto and may be implemented as a single display apparatus having a set-top box function.

The image error detection apparatus 140 performs an operation of detecting an error of a real-time image based on a machine learning algorithm. The video error detection apparatus 140 is connected to the video providing server 110, the set-top box 120, and the like, and detects an error of the real-time video output from the video output apparatus 130. Here, the error of the real-time image may be a screen break, but the present invention is not limited thereto, and may be various types of errors occurring on the screen.

The image error detection apparatus 140 receives sample image data from the image providing server 110 or an external device and generates learning data, and generates a determination reference model using deep learning (Deep Learning) . Here, it is preferable that the video error detection apparatus 140 generates a determination criterion model using the Alexnet technique. However, the present invention is not limited thereto. For example, the video error detection apparatus 140 may employ a deep learning technique such as GoogleNet, VGG (Covolutional Neural Net) To generate a criterion reference model.

Hereinafter, the operation of generating the learning data and the judgment reference model in the video error detecting apparatus 140 will be briefly described.

The video error detecting apparatus 140 generates abnormal video data by inducing a packet error in the sample video data, and compares the abnormal video data with the normal video data to extract an abnormal patch including the abnormal pixel. Here, the patch means a certain image area of a predetermined size.

The video error detection apparatus 140 extracts a normal patch corresponding to the abnormal patch, and classifies each of the normal patch and the abnormal patch into classes to generate learning data. Here, the image error detection apparatus 140 can additionally extract a text patch from the image data, and the extracted text patch can be classified into a separate class and included in the learning data.

The video error detection apparatus 140 generates a determination reference model for detecting screen breakage of an image by applying learning data including a normal patch class, an abnormal patch class, and a text patch class to a deep learning algorithm.

The image error detection apparatus 140 receives sample image data from the image providing server 110 or an external device and generates learning data for application to the deep learning algorithm. However, the present invention is not limited thereto. For example, the image error detection apparatus 140 may acquire real-time image data transmitted from the image providing server 110 to the set-top box 120 to generate learning data, and apply the generated learning data to the deep learning algorithm .

Hereinafter, an operation of detecting a screen break in a real-time image frame using the judgment reference model in the video error detecting apparatus 140 will be briefly described.

The video error detection apparatus 140 acquires a real-time video frame transmitted from the video providing server 110 to the set-top box 120, and generates a real-time video frame by using a determination reference model generated based on the deep- .

The video error detection device 140 divides the real-time image frame into a plurality of detection patch areas having a predetermined patch size and sets an error detection area including a plurality of detection patch areas. Here, the video error detection device 140 may set all or part of the plurality of detection patch areas as error detection areas.

The video error detection apparatus 140 detects a screen break area in the error detection area using the determination reference model. The video error detection apparatus 140 calculates the pixel state information for each patch region included in the error detection region by applying a judgment reference model within the error detection region, .

The image error detection apparatus 140 calculates pixel state information including a normal pixel detection rate, an abnormal pixel detection rate, a text pixel detection rate, and the like for each patch region by applying a judgment criterion model, If the detection rate is equal to or greater than the predetermined reference detection rate, it is detected as a screen broken area in the real-time image frame.

2 is a block diagram schematically showing an image error detecting apparatus according to the present embodiment.

The video error detection apparatus 140 according to the present embodiment includes a preprocessor 210, a model acquisition unit 220, a real-time image acquisition unit 230, a detection area setting unit 240, and a screen break detection unit 250 do.

The preprocessing unit 210 generates learning data to be input to the machine learning algorithm based on the image data, and applies the generated learning data to the machine learning algorithm to generate a determination reference model, which is a learning result. Here, the image data for generating the learning data may be sample image data received from the image providing server 110 or an external device, but is not limited thereto. The image data may be received from the image providing server 110 or the set- Real-time image data.

The preprocessing unit 210 generates the abnormal image data by inducing a packet error in the image data, and compares the abnormal image data with the normal image data to extract the abnormal patch including the abnormal pixel. The preprocessing unit 210 extracts a normal patch corresponding to the abnormal patch and a text patch corresponding to the text pixel, and classifies the normal patch, the abnormal patch, and the text patch into classes to generate learning data.

The preprocessing unit 210 applies a learning data including a normal patch class, an abnormal patch class, and a text patch class to a deep learning algorithm to generate a judgment criterion model for detecting screen breakage of an image. The configuration and operation of the preprocessing unit 210 will be described concretely in FIG.

The model acquiring unit 220 acquires the determination reference model generated from the preprocessing unit 210. Here, the model acquisition unit 220 transmits the acquired determination criterion model to the detection area setting unit 240 to set an error detection area, or transmits the detection reference area to the screen break detection unit 250 to detect a screen break area in the real- .

The model acquisition unit 220 may be omitted from the components of the image error detection apparatus 140 according to the needs of the user or the manufacturer. If omitted, the preprocessing unit 210 outputs the generated determination reference model to the detection region setting unit 240 or the screen breakage detecting unit 250 directly.

The real-time image acquisition unit 230 acquires a real-time image frame transmitted from the image providing server 110 to the set-top box 120. The real-time image acquisition unit 230 may acquire a real-time image frame from the set-top box 120. However, the real-time image acquisition unit 230 may receive the real-time image frame from the image providing server 110. Here, the real-time image frame may be an image frame for IPTV (Internet Protocol Television).

The detection area setting unit 240 sets an error detection area by dividing the real-time image frame into a plurality of detection patch areas. The detection area setting unit 240 divides the real-time image frame into a plurality of detection patch areas using a predetermined patch size based on the determination reference model. For example, in the case of the determination reference model generated based on the 227 * 227 patch size, the detection area setting unit 240 divides the real-time image frame into a plurality of detection patch areas having a lattice shape using the 227 * 227 patch size .

The detection area setting unit 240 sets all or a part of the plurality of detection patch areas as an error detection area. The detection area setting unit 240 can set the entire patch area of the plurality of detection patch areas as the error detection area so as to accurately detect the screen breaking area of the real-time image frame. On the other hand, the detection area setting unit 240 can set some patch areas of a plurality of detection patch areas as error detection areas in consideration of the frame retention time of the real-time image, quick detection processing, hardware load, and the like.

For example, when the real-time image frame is a full HD image frame having a size of 1920 * 1080, the detection area setting unit 240 divides 32 detection patch areas using a 227 * 227 patch size , All or some patch areas of the 32 detection patch areas can be set as error detection areas.

The screen breakage detector 250 detects a screen breakage area in the error detection area using the judgment reference model.

The screen breakage detection unit 250 calculates the pixel state information for each patch area included in the error detection area by applying a judgment reference model in the error detection area, and detects a screen break in the corresponding real- do.

The screen breakage detection unit 250 calculates pixel state information including a normal pixel detection rate, an abnormal pixel detection rate, and a text pixel detection rate for each patch region by applying a judgment reference model, and detects abnormal pixel detection If the rate is more than the predetermined reference detection rate, it is detected as a screen broken area in the real-time image frame.

When the entire patch area is set as the error detection area in the detection area setting part 240, the screen breakage detection part 250 changes the predetermined number of patch areas so as not to overlap each other to detect the screen breakage area with respect to the error detection area. Here, the predetermined number of patch areas means the number of patches that can be simultaneously analyzed.

On the other hand, when the detection area setting unit 240 sets some patch areas as error detection areas, the screen breakage detection unit 250 detects a screen breakage area for error detection areas of different areas for each image frame. In other words, the screen breakage detector 250 detects a screen break area within an error detection area where some patch areas are changed according to predetermined rules for every image frame or a patch area is randomly changed.

The screen breakage detector 250 sets the time at which the screen breakage (error) to be detected continues as the error validity hold time, and ends the analysis of the screen breakage for the error detection area within the set valid error hold time.

3 is a flowchart illustrating a method for detecting a screen break of a real-time image according to the present embodiment.

The video error detecting apparatus 140 generates learning data (S310), and generates a determination reference model by learning the learning data to the machine learning algorithm (S320). In other words, the video error detection apparatus 140 generates a determination reference model for detecting a screen break of a real-time image by applying learning data including a normal patch class, an abnormal patch class, and a text patch class to a deep learning algorithm .

The video error detection apparatus 140 obtains a real-time image frame from the set-top box 120 (S330). Here, the video error detection apparatus 140 may acquire a real-time video frame from the set-top box 120, but it is not limited thereto. The video error detection apparatus 140 may receive a real-time video frame from the video providing server 110. Here, the real-time image frame may be an image frame for IPTV (Internet Protocol Television).

The video error detection apparatus 140 divides the real-time image frame into a plurality of detection patch regions (S340). The video error detection apparatus 140 divides the real-time image frame into a plurality of detection patch areas using a preset patch size based on the determination reference model. For example, in the case of the determination reference model generated based on the 227 * 227 patch size, the video error detection apparatus 140 divides the real-time image frame into a plurality of detection patch regions having a lattice shape using the 227 * 227 patch size .

The video error detection device 140 sets an error detection area including a patch area of all or a part of the plurality of detection patch areas (S350).

The video error detecting apparatus 140 detects a screen breaking region in the error detecting region using the determination reference model (S360). The video error detection apparatus 140 calculates the pixel state information for each patch region included in the error detection region by applying a judgment reference model within the error detection region, . In other words, the video error detection apparatus 140 calculates pixel state information including a normal pixel detection rate, an abnormal pixel detection rate, a text pixel detection rate, and the like for each patch region by applying a judgment reference model, The abnormal pixel detection rate is greater than a predetermined reference detection rate.

3, it is described that steps S310 to S360 are sequentially executed. However, this is merely an exemplary description of the technical idea of the present embodiment, and it will be understood by those skilled in the art that, It will be understood that various changes and modifications may be made to the invention without departing from the essential characteristics thereof, or alternatively, by executing one or more of the steps S310 through S360 in parallel, But is not limited thereto.

As described above, the operation of the video error detection apparatus 140 according to the present embodiment described in FIG. 3 can be implemented by a program and recorded in a computer-readable recording medium. A program for implementing the operation of the video error detecting apparatus 140 according to the present embodiment is recorded and the computer readable recording medium includes all kinds of recording apparatuses in which data that can be read by the computer system is stored. Examples of such computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc., and also implemented in the form of a carrier wave (e.g., transmission over the Internet) . The computer readable recording medium may also be distributed over a networked computer system so that computer readable code is stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment can be easily inferred by programmers in the technical field to which the present embodiment belongs.

4 is a block diagram schematically showing a preprocessing unit included in the video error detection apparatus according to the present embodiment.

The preprocessing unit 210 according to this embodiment includes an image acquisition unit 410, a noise generation unit 420, a synchronization processing unit 430, a video error determination unit 440, a patch extraction unit 450, (460) and a criterion model generation unit (470).

The image acquiring unit 410 includes a first input port 412 and a second input port 414 and is connected to the same first image data through the first input port 412 and the second input port 414, Normal image data). Here, the first image data may be sample image data received from the image providing server 110 or an external device, but may be real-time image data received from the image providing server 110 or the set-top box 120.

When acquiring the sample image data received from the image providing server 110 or the external device, the first input port 412 and the second input port 414 are connected to a first set-top box (not shown) and a second set- Time).

The image acquiring unit 410 transmits the first image data input through the first input port 412 to the synchronization processing unit 430 via the noise generating unit 420 and transmits the first image data through the second input port 414 The input first image data is transmitted to the synchronization processing unit 430.

The noise generator 420 generates second image data (abnormal image data) including noise. The noise generator 420 generates a second image data by deriving a packet error of the first image data input through the first input port.

Although the noise generating unit 420 is described as being a separate module from the image obtaining unit 410, the noise generating unit 420 is not limited thereto and may be embodied as being included in the first input port 412.

The synchronization processing unit 430 synchronizes the image frame timing between the first image data input through the second input port 414 and the second image data including noise.

The synchronization processing unit 430 is a separate module for synchronizing the viewpoints of the video frames between the first video data and the second video data. However, the present invention is not limited thereto, (440). ≪ / RTI >

The image error determination unit 440 determines an image error by comparing the first image data and the second image data, and detects an abnormal pixel with respect to an image error. Here, the first image data and the second image data are compared with the image frames at the same time synchronized by the synchronization processing unit 430 to detect an abnormal pixel.

The video error determination unit 440 compares the pixels at the same position between the first video frame of the first video data and the second video frame of the second video data to determine pixels exceeding the reference pixel difference value as a video error Thereby detecting an abnormal pixel in the second image frame. Herein, pixels exceeding the reference pixel difference value can detect an abnormal pixel by applying a K-means clustering method, but the present invention is not limited thereto, and a hierarchical clustering technique, a non-hierarchical clustering technique , A model-based technique, and the like, any clustering technique can be applied as long as it is a clustering technique that can distinguish a plurality of pixels using a predetermined feature.

The patch extracting unit 450 extracts a negative patch including an abnormal pixel and extracts a normal patch corresponding to the abnormal patch. In other words, the patch extraction unit 450 extracts an abnormal patch having a predetermined patch size including an abnormal pixel in the second image frame, and extracts a normal patch corresponding to the abnormal patch in the first image frame. Here, it is described that the patch extracting unit 450 extracts a normal patch at a position corresponding to an abnormal patch, but the present invention is not limited thereto, and it is possible to extract the abnormal patch and the normal patch at separate image frame positions.

The patch extracting unit 450 extracts a patch area including the largest number of abnormal pixels among the areas of the predetermined patch size in the second image frame as the first abnormal patch and then extracts the remaining area excluding the area extracted as the first abnormal patch The patch region including the largest number of abnormal pixels in the region is extracted as the second abnormal patch. The patch extracting unit 450 repeats the patch extraction operation to extract a plurality of abnormal patches such as the first abnormal patch and the second abnormal patch. Here, the patch extracting unit 450 repeats the operation of extracting the abnormal patches until the abnormal pixel does not exist in the predetermined patch size area or there are less than predetermined number of abnormal pixels.

The patch extracting unit 450 further extracts a text patch for an area recognized as text in the first image frame. Here, the patch extracting unit 450 can recognize the text using an OCR (Optical Character Recognition) method.

Although the patch extracting unit 450 extracts a text patch of a predetermined patch size in the first image frame, it is not limited thereto. For example, the patch extracting unit 450 may extract the text patches in the remaining regions except for the abnormal patches in the second image frame in which the extraction of the abnormal patches is completed.

The sizes of the abnormal patch, the normal patch, and the text patch extracted by the patch extracting unit 450 are preferably the same.

The learning data generation unit 460 collects various kinds of patches generated by the patch extraction unit 450, and classifies the collected patches into respective classes to generate learning data. Here, the learning data is input data for inputting to the machine learning algorithm, and means the base (sample) data of the determination reference model generated through learning of the machine learning algorithm.

The learning data generation unit 460 collects a plurality of abnormal patches and normal patches, and divides each of a plurality of abnormal patches and normal patches into an abnormal class and a normal class to generate learning data.

The training data generation unit 460 may classify each of the abnormal patches and the normal patches corresponding to the abnormal patches into classes. However, the learning data generation unit 460 does not necessarily have to classify the normal patches and the abnormal patches separately from the first image frame and the second image frame, Learning data can be generated by dividing each patch into a normal class and an abnormal class.

The learning data generation unit 460 further collects a text patch when a text patch exists, and further generates a text class for the text patch to generate learning data.

The learning data generation unit 460 preferably keeps the difference in the number of patches included in each of the normal class, the abnormal class, and the text class within a predetermined error number range. The learning data generation unit 460 increases the accuracy of the criterion model generated based on the learning data by processing the difference in the number of normal patches, abnormal patches, and text patches included in each class so as to remain within the error number range have. For example, when the learning data includes a normal class, an abnormal class, and a text class, the learning data generation unit 460 generates a learning class for each of the 'normal' 100,000 normal patches, 300,000 abnormal abnormal patches, A normal class 100,000 abnormal patches, an abnormal class 110,000 abnormal patches, and a text class 9000 text patches, as compared with the first learning data including a text class of '200,000 text patches' And generates second learning data having an error number range of 20,000.

The determination criterion model generation unit 470 generates the criterion reference model by applying the learning data to the machine learning algorithm. Here, the criterion model generation unit 470 may generate a criterion model using Deep Learning.

The determination criterion model generation unit 470 preferably generates a criterion criterion model using the Alexnet technique. However, the criterion criterion model generation unit 470 is not limited to this. For example, the criterion criterion model generation unit 470 may use a deep learning technique such as GoogleNet, VGG (Covolutional Neural Net) A judgment criterion model can be generated.

The criterion reference model generation unit 470 generates a criterion criterion model for detecting screen breakage in each of the plurality of patch regions. The determination criterion model can calculate pixel state information including a normal pixel detection rate, an abnormal pixel detection rate, a text pixel detection rate, and the like for each patch area. Based on the determination criterion model, the abnormal pixel detection rate The patch area is detected as a screen breaking area.

5 is a flowchart illustrating a preprocessing process for detecting a screen break of a real-time image according to the present embodiment. Steps S510 to S590 described in Fig. 5 are described in detail with reference to steps S310 and S320 in Fig.

The video error detection apparatus 140 obtains the first video data through the first input port 412 and the second input port 414, respectively (S510). Here, the first image data may be sample image data received from the image providing server 110 or an external device, but may be real-time image data received from the image providing server 110 or the set-top box 120.

The video error detecting apparatus 140 generates second video data including noise by deriving a packet error of the first video data input through the first input port 412 in operation S520.

The video error detection apparatus 140 compares the first video frame of the first video data and the second video frame of the second video data acquired through the second input port 414 in operation S530, Abnormal pixels are detected (S540). Here, the first image data and the second image data are synchronized and the first image frame and the second image frame at the same time are compared to detect an abnormal pixel. More specifically, the video error detecting apparatus 140 compares the same positional pixels between the first video frame and the second video frame, determines pixels that exceed a predetermined reference pixel difference value as a video error, And detects an abnormal pixel within the pixel.

The image error detection apparatus 140 extracts a negative patch including an abnormal pixel (S550), and extracts a normal patch corresponding to the abnormal patch (S552). In other words, the video error detection apparatus 140 extracts an abnormal patch having a predetermined patch size including an abnormal pixel in the second video frame, and extracts a normal patch corresponding to the abnormal patch within the first video frame . Here, the video error detecting apparatus 140 describes extracting a normal patch at a position corresponding to an abnormal patch, but the present invention is not limited thereto, and the video error detecting apparatus 140 can extract the abnormal patch and the normal patch at separate image frame positions .

The image error detection apparatus 140 may further extract a text patch for an area recognized as text in the first image frame or the second image frame.

The video error detection apparatus 140 classifies the plurality of abnormal patches and the normal patch into respective classes (S560). Here, the video error detecting apparatus 140 may further form a text class for the text patch when the text patch exists.

The video error detection apparatus 140 determines whether the abnormal patch, normal patch, text patch, or the like included in each class has reached a predetermined number (S562). If the abnormal patch, normal patch, Learning data is generated (S570).

It is preferable that the image error detection apparatus 140 maintains the difference in the number of patches included in each of the normal class, the abnormal class, and the text class within a predetermined error number range. The video error detection apparatus 140 processes the differences in the number of normal patches, abnormal patches, and text patches included in each class so as to remain within the error number range, thereby increasing the accuracy of the determination reference model generated based on the learning data have.

The video error detection apparatus 140 applies the learning data to the machine learning algorithm (S580), and generates a determination reference model for detecting the screen breaking region of the real-time video frame (S590). The video error detection apparatus 140 can generate a determination criterion model by applying a deep learning algorithm such as Alexnet technique, GoogleNet, Covolutional Neural Net (VGG), and ResNet. The pixel state information including the normal pixel detection rate, the abnormal pixel detection rate, the text pixel detection rate, and the like is calculated for each patch area.

5, it is described that steps S510 to S590 are sequentially executed. However, this is merely an example of the technical idea of the present embodiment, and it should be understood that any person skilled in the art will be able to It is to be understood that variations and modifications may be made without departing from the spirit or scope of the inventive concept by varying the order described in FIG. 5 or by performing one or more of steps S510 through S590 in parallel, It is not.

As described above, the operation of the video error detecting apparatus 140 according to the present embodiment described in FIG. 5 can be implemented by a program and recorded on a computer-readable recording medium. A program for implementing the operation of the video error detecting apparatus 140 according to the present embodiment is recorded and the computer readable recording medium includes all kinds of recording apparatuses in which data that can be read by the computer system is stored. Examples of such computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc., and also implemented in the form of a carrier wave (e.g., transmission over the Internet) . The computer readable recording medium may also be distributed over a networked computer system so that computer readable code is stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment can be easily inferred by programmers in the technical field to which the present embodiment belongs.

6 is an exemplary diagram for explaining an operation of determining a video error in a preprocessing process for detecting a screen break of a real-time video according to the present embodiment.

FIG. 6A shows a 'normal frame' based on the first image data, and FIG. 6A shows an 'abnormal frame' based on the second image data.

The video error detecting apparatus 140 compares the pixels at the same position between the 'normal frame' and the 'abnormal frame', respectively, and determines pixels having a predetermined reference pixel difference value as a video error.

FIG. 6C shows an 'image error frame' indicating an abnormal pixel determined as a video error within the 'abnormal frame'. The image error detecting apparatus 140 may display an abnormal pixel exceeding the reference pixel difference value by applying a K-means clustering method as a red dot.

The image error detecting apparatus 140 extracts an abnormal patch having a predetermined patch size including the abnormal pixel in the frame of FIG. 6C, and detects the abnormal patch corresponding to the abnormal patch in the frame of FIG. 6 (a) The normal patch is extracted.

7A and 7B are diagrams showing learning data for generating a determination criterion model according to the present embodiment.

FIG. 7A shows an abnormal class including a plurality of abnormal patches, and FIG. 7B shows a normal class including a plurality of normal patches. The video error detection apparatus 140 extracts an abnormal patch including an abnormal pixel and extracts a normal patch corresponding to the abnormal patch. The video error detecting apparatus 140 repeats the operation of extracting the patch until a predetermined number of patches are included in each class.

Fig. 7B shows a text class containing a text patch. The image error detecting apparatus 140 recognizes text using an OCR (Optical Character Recognition) method, extracts patches of a predetermined size corresponding to the text, and classifies the text into text classes.

8A to 8C are views illustrating an operation of detecting a screen break of a real-time image according to the present embodiment.

8A is a diagram illustrating a result of dividing a real-time image frame into a plurality of detection patch regions in the video error detection apparatus 140. [ As shown in FIG. 8A, the video error detection apparatus 140 can distinguish 1920 * 1080 real time video frames with a plurality of detection patch regions having a size of 227 * 227.

The image error detecting apparatus 140 can detect a screen breakage using all or a part of a plurality of detection patch regions having a size of 227 * 227.

FIG. 8B is an exemplary view showing an operation of detecting a screen break of a real-time image frame using a part of a plurality of detection patch areas in the video error detection device 140. FIG.

The time at which the image error detection device 140 continues to detect a screen error (error) is referred to as an error validity hold time. For example, the time that a screen break may remain is 2 seconds. Therefore, the video error detection device 140 must terminate the analysis of a plurality of detection patch areas within the error validity holding time.

As shown in FIG. 8B, the video error detecting apparatus 140 divides the real-time image frame during the error validity hold time into 10 frames so that a plurality of detection patch regions can be analyzed even at least once during the error validity hold time . Here, the video error detection apparatus 140 analyzes the four simultaneous analysis processing patches 830 in one frame on the basis of the determination reference model within 180 ms. That is, the time for analyzing all of the plurality of real-time image frames during the error hold time may be 180 ms * 10 + 200 ms (image processing overhead time), and this time should be set smaller than 2000 ms . The operation of the video error detecting apparatus 140 described above is preferably set as shown in [Table 1].

Item value Remarks Number of patches per frame 32 8 (Number of Landscape) * 4 (Number of Landscape) Error Retention Time 2000 ms Error to detect is held for 2 seconds Number of simultaneous analysis processing patches per frame 4 Application position can be set randomly or regularly (overlapping x) Simultaneous analysis processing Patch processing time 180 ms Analysis time must not be exceeded

The condition information such as the error validity holding time described in FIG. 8B, the number of frames divided within the error validity holding time, the processing time of the simultaneous analysis processing patch, the number of simultaneous analysis processing patches, Environment, user setting input, or the like. On the other hand, the video error detecting apparatus 140 may detect a type of a real-time image (e.g., drama, news, movie, music, etc.), a size of a real time image frame (e.g., HD, Full HD, UHD, You can check the same analysis environment and change the condition information according to the analysis environment.

8C is an exemplary view illustrating an operation of detecting a screen break of a real-time image frame by regularly or randomly changing a partial area of a plurality of detection patch areas in the video error detection apparatus 140. FIG.

As shown in FIG. 8C, the image error detection apparatus 140 can detect a screen break by applying a judgment reference model based on a deep learning algorithm in a regularly set error detection region 840.

As shown in (b) of FIG. 8, the image error detection apparatus 140 can detect a screen break by applying a judgment reference model based on a deep learning algorithm in a randomly set error detection region 850.

FIG. 9 is an exemplary view schematically showing an entire operation procedure for detecting screen breakage of a real-time image according to the present embodiment.

The video error detection apparatus 140 according to the present embodiment automatically detects the difference between the normal frame and the abnormal frame and extracts the abnormal patch and the normal patch with a pair to generate learning data.

The video error detection apparatus 140 learns and processes the generated learning data to a Deep Learning algorithm to generate a judgment reference model. In other words, the video error detection apparatus 140 learns learning data including an abnormal patch and a normal patch using the latest deep learning learning models Alexnet, GoogleNet, VGG (Covolutional Neural Net), ResNet and the like , And a judgment criterion model which automatically learns a classification pattern for detecting screen breakage is generated.

The video error detection apparatus 140 detects and predicts screen breakage of a real-time image using the learned determination reference model. The video error detection apparatus 140 can detect a screen break in real time by distributing the screen break analysis area for each frame based on the time during which the screen break is maintained.

The foregoing description is merely illustrative of the technical idea of the present embodiment, and various modifications and changes may be made to those skilled in the art without departing from the essential characteristics of the embodiments. Therefore, the present embodiments are to be construed as illustrative rather than restrictive, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be construed according to the following claims, and all technical ideas within the scope of equivalents thereof should be construed as being included in the scope of the present invention.

100: video service providing system 110: video providing server
120: set-top box 130: video output device
140: Video error detection device
210: preprocessing unit 220: model acquisition unit
230: Real-time image acquisition unit 240: Detection area setting unit
250: screen break detector
410: image acquisition unit 420: noise generation unit
430: synchronization processing unit 440: video error determination unit
450: Patch extracting unit 460: Learning data generating unit
470:

Claims (16)

A method for detecting a screen break in an image error detection apparatus in a real-time image service,
A model acquisition process for acquiring a decision criterion model generated based on a machine learning algorithm;
A real-time image acquisition process for acquiring a real-time image frame from an external device;
The real-time image frame is divided into a plurality of detection patch areas having a patch size set based on the determination reference model, and a detection area setting process of setting all or a part of the plurality of detection patch areas as an error detection area ; And
And a screen break detection step of detecting a screen break area in the error detection area using the judgment reference model,
The screen break detection process includes:
Wherein the simultaneous analysis processing patch area that can simultaneously analyze within the predetermined error validity time is detected in the entire patch area using the determination criterion model in the error detection area including the entire patch area, And detecting the broken area of the screen image. The method according to claim 1,
The screen break detection process includes:
Detecting the screen broken area using the determination criterion model in the error detection area including a part of patch areas,
Wherein the patch region is a different region for each image frame input in real time. 3. The method of claim 2,
The screen break detection process includes:
And detecting the screen broken area in the error detection area in which the partial patch area is changed according to a predetermined rule for each of the image frames or the partial patch area is randomly changed. . delete The method according to claim 1,
A preprocessing step of generating learning data based on the first image data and applying the learning data to the machine learning algorithm to generate the determination criterion model
Further comprising a step of detecting a screen break of the real-time image service. 6. The method of claim 5,
The pre-
An image acquiring step of acquiring the first image data through each of a first input port and a second input port;
An image comparing step of deriving a packet error of the first image data to generate second image data, and comparing the first image data and the second image data to extract an abnormal pixel;
A data generation step of extracting an abnormal patch including the abnormal pixel and extracting a normal patch corresponding to the abnormal patch to generate learning data; And
A model generation step of applying the learning data to the machine learning algorithm to generate the determination criterion model
And detecting a screen break of the real-time image service. The method according to claim 6,
The image comparison process includes:
A noise generation step of deriving the packet error of the first image data acquired through the first input port and outputting the second image data including noise; And
An image error judging process of comparing the first image frame of the first image data with the second image frame of the second image data and extracting the abnormal pixel in a broken region exceeding a preset reference pixel difference value
And detecting a screen break of the real-time image service. 8. The method of claim 7,
The image comparison process includes:
Further comprising a synchronization processing step of synchronizing the viewpoints of the first image data and the second image data,
Wherein the image error determination step comprises comparing the first image frame and the second image frame at the same time point to extract the abnormal pixel. The method according to claim 6,
The data generation process includes:
A patch extracting step of extracting the abnormal patch of a predetermined size including the abnormal pixel in the second image frame and extracting the normal patch corresponding to the abnormal patch in the first image frame; And
A learning data generating step of collecting the abnormal patch and the normal patch and dividing the abnormal patch and the normal patch into classes,
And detecting a screen break of the real-time image service. 10. The method of claim 9,
The patch extracting process includes:
Further extracting a text patch for an area recognized as text in the first image frame,
Wherein the learning data generating step generates the learning data further including a class for the text patch. The method according to claim 6,
The model generation process includes:
Generating a determination criterion model using a Deep Learning algorithm,
The screen breaking detection process may include detecting a criterion model generated using at least one of deep processing techniques such as Alexnet, GoogleNet, Covolutional Neural Net (VGG), and ResNet to a predetermined patch area And detecting the broken area of the screen. 6. The method of claim 5,
Wherein the first image data comprises:
Wherein the real-time image data is the real-time image data or the separately input sample image data. The method according to claim 1,
The screen break detection process includes:
Wherein the error detection unit detects the screen breakage based on the pixel state information and calculates the pixel state information for each patch region included in the error detection region based on the determination criterion model, Way. 14. The method of claim 13,
Wherein the pixel state information comprises:
A normal pixel detection rate, an abnormal pixel detection rate, and a text pixel detection rate,
Wherein the screen breaking detection process detects the screen breaking area in the real time image frame when the abnormal pixel detection rate is equal to or greater than a predetermined reference detection rate. An apparatus for detecting a screen break in a real-time video service,
A model acquisition unit for acquiring a decision criterion model generated based on a machine learning algorithm;
A real-time image acquisition unit for acquiring a real-time image frame from an external device;
Wherein the real-time image frame is divided into a plurality of detection patch areas having a patch size set based on the determination criterion model and all or a part of the plurality of detection patch areas is set as a detection area, ; And
And a screen break detection unit for detecting a screen break area in the error detection area using the judgment reference model,
Wherein the screen breakage detection unit comprises:
Wherein the simultaneous analysis processing patch area that can simultaneously analyze within the predetermined error validity time is detected in the entire patch area using the determination criterion model in the error detection area including the entire patch area, And detects the screen breakage area by changing the screen breakage area. In the data processing device,
A model acquisition process for acquiring a decision criterion model generated based on a machine learning algorithm;
A real-time image acquisition process for acquiring a real-time image frame from an external device;
The real-time image frame is divided into a plurality of detection patch areas having a patch size set based on the determination reference model, and a detection area setting process of setting all or a part of the plurality of detection patch areas as an error detection area ; And
And a screen break detection step of detecting a screen break area in the error detection area using the judgment reference model,
The screen break detection process includes:
Wherein the simultaneous analysis processing patch area that can simultaneously analyze within the predetermined error validity time is detected in the entire patch area using the determination criterion model in the error detection area including the entire patch area, And detecting the screen broken area by changing the display area of the screen.

Images