KR102167730B1 - Apparatus and method for masking a video - Google Patents

Abstract

The present application relates to an image masking apparatus and an image masking method. An image masking method according to an embodiment of the present invention includes: generating a frame image in units of frames by decoding an input target video; Extracting a frame feature vector corresponding to each frame image from the frame image, and detecting a candidate object from the frame feature vector; Extracting an object feature vector corresponding to each candidate object from the frame feature vector using the position information of the detected candidate objects; Setting identification information for the candidate objects by using the object feature vector; Tracking candidate objects having the same identification information within a continuous frame image, and generating tracking information for the candidate objects for each identification information; And when receiving the masking input, extracting candidate objects identical to the identification information included in the masking input by using the tracking information, and performing masking on the extracted candidate objects.

Description

Image masking device and image masking method {Apparatus and method for masking a video}

The present application relates to an image masking device and an image masking method, and to an image masking device and an image masking method capable of masking a video for personal privacy protection.

Due to the recent spread of networks such as the Internet and high-functionalization of various devices (personal computers, smartphones, cameras, etc.) used by general users, it is easy to publish videos taken by individuals online or share them with others. I could do it. However, as the number of videos shared on the network increases, a problem of personal information exposure or privacy invasion is becoming serious, such as a video, etc. in which a person's face is photographed is disclosed on the network without permission. In order to solve this problem, conventionally, a de-identification technique for preventing the exposure of personal information by mosaicizing the face of a person included in a video has been proposed.

However, the conventional de-identification technology was difficult in that the user had to directly perform de-identification processing on the corresponding video. In particular, in the case of de-identifying only a specific person among various persons included in the video. There were difficulties such as having to manually check all the people included in the entire video.

On the other hand, as a method for exceptionally handling masking for a specific person, registration number 10-1215948, "A method of masking image information of a surveillance system based on body information and face recognition" has been suggested. That is, if a person's face can be recognized from the image information, the distance between the eyebrows and the person's face is detected, and based on this, the face of a specific person stored in the database is identified, and the masking process of the person is separately processed. Is being presented.

However, in the case of the technology, since each technology used to detect a person, recognize a face, and track a person in an image is all independently performed, a very large amount of computation is required to perform each process. In addition, since the performance of the entire system is determined by the lower limit of the performance of a specific technology, a problem of lowering the detection rate and recognition rate of a person occurs. In addition, since the distance between the two eyes and the person's middle is used as a feature point used for face recognition, there are problems such as a remarkable decrease in the face recognition rate in an actual system.

The present application is intended to provide an image masking apparatus and an image masking method capable of masking a video to protect personal privacy.

The present application is to provide an image masking apparatus and an image masking method capable of selectively masking objects included in a video.

The present application is to provide an image masking apparatus and an image masking method capable of efficiently masking by tracking objects included in a video.

The present application is intended to provide an image masking device and an image masking method that are superior to conventional technologies in detecting, tracking, and recognizing an object included in a moving picture.

An image masking method according to an embodiment of the present invention includes: generating a frame image in units of frames by decoding an input target video; Extracting a frame feature vector corresponding to each frame image from the frame image, and detecting a candidate object from the frame feature vector; Extracting an object feature vector corresponding to each candidate object from the frame feature vector using the position information of the detected candidate objects; Setting identification information for the candidate objects by using the object feature vector; Tracking candidate objects having the same identification information within a continuous frame image, and generating tracking information for the candidate objects for each identification information; And when receiving the masking input, extracting candidate objects identical to the identification information included in the masking input by using the tracking information, and performing masking on the extracted candidate objects.

The step of setting the identification information includes: searching for a registered feature vector corresponding to the object feature vector from an identification information database; When a registration feature vector corresponding to the object feature vector is searched, extracting identification information matched with the registration feature vector from the identification information database and setting it as the identification information of the candidate object; And if the registration feature vector corresponding to the object feature vector is not searched, newly generating identification information of the candidate object, and newly registering the object feature vector and the identification information in the identification information database. .

In the step of searching the identification information database, if the object feature vector matches the registered feature vector within a preset error range, it may be determined that the object feature vector corresponds to the registered feature vector.

In the generating of the tracking information, the same candidate object may be tracked using a difference value of object feature vectors of each of the candidate objects included in the continuous frame image and a change in position and size of the candidate objects. have.

Here, the step of generating the tracking information includes Error = (V1-V2) + a × (d1-d2) + b × (s1-s2) where Error is a tracking error value, and V1 is included in the first frame image. The object feature vector of the first candidate object, V2 is the object feature vector of the second candidate object included in the second frame image, d1 is the distance from the reference point to the center point of the first candidate object, and d2 is the second candidate object from the reference point. The distance to the center point, s1 is the area of the first candidate object, s2 is the area of the second candidate object, a and b are weights, and the second candidate object whose tracking error value is the minimum value is identified as the first candidate object. Can be identified as a candidate object having information.

Here, in the image masking method according to an embodiment of the present invention, when a candidate object having the same identification information is partially omitted in a continuous frame image, an error of determining that an error has occurred in setting identification information for the candidate object It may further include a sensing step.

Here, the tracking information may include at least one of identification information of the candidate object, frame information in which the candidate object appears, location information and size information of the candidate object included in the frame image.

Here, the image masking method according to an embodiment of the present invention comprises the step of generating an object tracking image by overlaying the candidate object and the identification information for each candidate object on the target video using the tracking information. It may contain more.

Here, the image masking method according to an embodiment of the present invention includes an identification information list in which identification information corresponding to candidate objects included in the target video is arranged, and candidate objects having the same identification information appear in the target video. The method may further include displaying a masking selection interface including appearance section information and a frame image in which the candidate object appears.

Here, the masking input may be generated using identification information input from a user, or generated by extracting the identification information according to a preset selection algorithm.

Here, the performing of the masking includes extracting frame images in which the candidate object corresponding to the identification information appears using the tracking information, and setting a masking area corresponding to the position of the candidate object in the extracted frame image. I can.

In the performing of the masking, the masking area may be masked by using blurring, mosaic processing, or image replacement.

In the performing of the masking, masking information for a masking area set in the target video and the target video are stored, respectively, and the target video is masked and played using the masking information when the target video is played. have.

Here, in the performing of the masking, an access right is requested when the target video is played, and if the access right is not provided, the target video may be masked and played.

According to an embodiment of the present invention, a computer program stored in a medium may exist in order to execute the above-described image masking method in combination with hardware.

An image masking apparatus according to an embodiment of the present invention includes: a frame input unit configured to generate a frame image in units of a plurality of frames by decoding an input target video; Extracting a frame feature vector corresponding to each frame image from the frame image, detecting a candidate object from the frame feature vector, and using the position information of the detected candidate objects, each candidate object from the frame feature vector A feature vector extracting unit for extracting an object feature vector corresponding to; An identification information setting unit for setting identification information for the candidate objects by using the object feature vector; A tracking information generator configured to track candidate objects having the same identification information within a continuous frame image, and generate tracking information for the candidate objects for each identification information; And a masking unit for extracting candidate objects identical to the identification information included in the masking input by using the tracking information and performing masking on the extracted candidate objects.

An image masking apparatus according to an embodiment of the present invention includes: a processor; And a memory coupled to the processor, wherein the memory includes at least one module configured to be executed by the processor, and the at least one module decodes an input target video to Create a frame image, extract a frame feature vector corresponding to each frame image from the frame image, detect a candidate object from the frame feature vector, and use the position information of the detected candidate objects to feature the frame After extracting an object feature vector corresponding to each candidate object from the vector, setting identification information for the candidate objects using the object feature vector, and tracking candidate objects having the same identification information within a continuous frame image Thus, tracking information for the candidate objects is generated for each identification information, and when a masking input is received, candidate objects identical to the identification information included in the masking input are extracted using the tracking information, and the extracted candidate objects It may include a command that performs masking on objects.

In addition, the solution to the above-described problem does not enumerate all features of the present invention. Various features of the present invention and advantages and effects thereof may be understood in more detail with reference to the following specific embodiments.

According to the image masking apparatus and the image masking method according to an embodiment of the present invention, since each object included in a video can be distinguished, an object selected by the user can be selectively masked to de-identify. That is, objects included in the video are not collectively de-identified, but a selected specific object can be selectively de-identified, thereby enhancing user convenience.

According to the image masking apparatus and the image masking method according to an embodiment of the present invention, since the positions of each object included in the video can be tracked, a specific object selected by the user can be easily extracted from the entire video and masked. have.

According to the image masking apparatus and the image masking method according to an embodiment of the present invention, it is efficient because it is possible to detect, track, and recognize a candidate object from one frame feature vector, and machine learning each detection, tracking, and recognition Performance and computation speed can be remarkably improved because it can be learned to perform simultaneously using an algorithm or the like.

According to the image masking apparatus and the image masking method according to an embodiment of the present invention, when an unspecified person without access rights reads a video, a masked video is provided, thereby preventing exposure of personal information and invasion of privacy.

However, the effects that can be achieved by the image masking apparatus and the image masking method according to the embodiments of the present invention are not limited to those mentioned above, and other effects not mentioned are the techniques to which the present invention belongs from the following description. It will be clearly understood by those of ordinary skill in the field.

1 is a schematic diagram showing an image masking system according to an embodiment of the present invention.
2 and 3 are block diagrams showing an image masking apparatus according to an embodiment of the present invention.
4 is a schematic diagram illustrating extraction of a candidate object and setting of identification information according to an embodiment of the present invention.
5 is a schematic diagram showing a masking selection interface according to an embodiment of the present invention.
6 is a flowchart illustrating an image masking method according to an embodiment of the present invention.

Hereinafter, exemplary embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but identical or similar elements are denoted by the same reference numerals regardless of the reference numerals, and redundant descriptions thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not have meanings or roles that are distinguished from each other by themselves. That is, the term'unit' used in the present invention means a hardware component such as software, FPGA or ASIC, and the'unit' performs certain roles. However,'part' is not limited to software or hardware. The'unit' may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors. Thus, as an example,'unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, Includes subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays and variables. The functions provided in the components and'units' may be combined into a smaller number of components and'units', or may be further divided into additional components and'units'.

In addition, in describing the embodiments disclosed in the present specification, when it is determined that a detailed description of related known technologies may obscure the subject matter of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention It should be understood to include equivalents or substitutes.

1 is a schematic diagram showing an image masking system according to an embodiment of the present invention.

Referring to FIG. 1, an image masking system according to an embodiment of the present invention may include an image capturing apparatus 1 and an image masking apparatus 100.

Hereinafter, an image masking system according to an embodiment of the present invention will be described with reference to FIG. 1.

The image capturing apparatus 1 may generate a video by photographing the surrounding environment. Here, the image capturing device 1 may correspond to any device such as a video camera or a camcorder capable of photographing a moving picture. The image capturing device 1 may stream or store the captured video in real time as a file, and the image capturing device 1 may provide wired or wireless communication to transmit the captured video.

The image masking apparatus 100 may receive a target video to be masked, and may perform masking on objects included in the received target video. Here, the image masking device 100 may receive a target video captured by the image capturing device 1 in a file or data format, and according to an embodiment, the target video is streamed from the image capturing device 1 in real time. It is also possible.

In FIG. 1, the image masking device 100 is shown to be provided separately from the image photographing device 1, but according to the embodiment, the image masking device 100 is an image processing device 1 such as a CCTV, a black box for a vehicle, a camera, etc. ), or may be provided in a separate computer or smartphone.

Meanwhile, the image masking apparatus 100 may distinguish objects included in the target video, and may perform masking on the distinguished objects. That is, in order to prevent exposure of personal information or invasion of privacy, some objects included in the video may be masked and de-identified. For example, in the case of an image photographing apparatus 1 such as a CCTV or a vehicle black box, a set photographing area is recorded at random, so that a face or a body part of another person, a license plate, etc. may be photographed. Here, when the video is disclosed through the Internet or the like, personal information such as the face of another person may be exposed, resulting in a problem such as invasion of privacy. Therefore, in order to prevent such problems, it is necessary to mask the face of another person included in the target video or the vehicle license plate to be processed so that it cannot be identified.

Conventionally, there has been a difficulty in that a user has to perform masking directly on a target video, and even when automatic masking is provided, it is common to collectively perform masking on all objects included in the video. Therefore, in the case of masking only a specific person among various people included in the video, or when masking is required except for a specific person, manually check whether all the people included in the video correspond to a specific person, and perform masking. Difficulties existed, such as having to be performed.

On the other hand, according to the image masking apparatus 100 according to an embodiment of the present invention, each object shown in a video can be identified, and only selected objects among objects included in the video are tracked to automatically perform masking. It is possible to do.

Hereinafter, an image masking apparatus according to an embodiment of the present invention will be described.

2 is a block diagram showing an image masking apparatus according to an embodiment of the present invention.

Referring to FIG. 2, an image masking apparatus 100 according to an embodiment of the present invention includes a frame input unit 110, a feature vector extraction unit 120, an identification information setting unit 130, and a tracking information generation unit 140. , An error detection unit 150, a masking unit 160, an object tracking image generation unit 170, and a masking selection interface display unit 180.

The frame input unit 110 may generate an image in units of frames by decoding the input target video. Specifically, the frame input unit 110 may receive a target video encoded by the Moving Picture Experts Group (MPEC) method from the image capturing device 1, and then decode the target video in the form of a compressed MPEC file, and By capturing each of the frames of, you can create a frame image. Here, the target video may include N frame images, and frame numbers from 1 to N may be assigned to each frame image. Here, N corresponds to a natural number.

The feature vector extraction unit 120 may extract a frame feature vector corresponding to each frame image from the frame image. The frame feature vector generated by the feature vector extractor 120 can be used to detect candidate objects included in the frame image afterwards, or to distinguish and recognize each candidate object, and can be used to recognize candidates included in successive frame images. It can also be used to track objects.

First, the feature vector extractor 120 may generate a frame feature vector using pixel information of a frame image, such as location information and pixel value information of each pixel included in the frame image. Here, in order to extract frame feature vectors using pixel information corresponding to each frame image, machine learning algorithms such as CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) are used, or Histogram of Oriented Gradient (HOG) , LBP (Local Binary Pattern), etc. can be used to extract statistical characteristics of the image. In addition, frame feature vectors can be generated in various ways, and the content of the present invention is not limited by the above-described method.

Thereafter, the feature vector extraction unit 120 may detect candidate objects included in each frame image by using the generated frame feature vector. Here, the candidate object is an object that can be a target of masking, and the candidate object may be set differently according to embodiments. For example, CCTV (Closed Circuit Television) can photograph various types of objects, such as people, animals, vehicles, park benches, and sports equipment passing through a photographing area. However, in relation to personal information protection, since the object requiring masking may be a person or a vehicle license plate, the feature vector extraction unit 120 extracts a person or vehicle license plate from the target image captured by CCTV and sets it as a candidate object. I can.

Here, features such as shape and luminance of candidate objects to be extracted can be learned in advance using a machine learning algorithm, and the feature vector extractor 120 utilizes this to Area can be detected. For example, it is possible to repeatedly learn the shapes of various people to extract the shape of a person included in the frame image as a candidate object. At this time, the machine learning algorithm is CNN, RNN, Principal Component Analysis (PCA), and Logistic. Regression, Decision Tree, etc. can be used.

Meanwhile, the feature vector extractor 120 may specify and display the extracted candidate object as a bounding box or a segmentation mask on the frame image. Here, the bounding box may be a rectangle displayed around a pedestrian that is a candidate object, as shown in FIG. 4(a), and a user can easily distinguish candidate objects through the bounding box. Depending on the embodiment, using the position coordinates (x,y) of the upper left vertex of the bounding box and four numbers (x,y,w,h) representing the width and height (w,h) of the bounding box , You can specify each bounding box.

In addition, when a segmentation mask is used, a foreground corresponding to a candidate object among a frame image may be displayed separately from a background serving as the remaining background in pixel units. That is, as shown in FIG. 4B, the pixel value of the pixels corresponding to the background is set to 0, and the pixel value of the candidate object corresponding to the foreground is set to 1. As shown in FIG.

After detecting the candidate object, the feature vector extractor 120 may generate an object feature vector corresponding to the candidate object. The feature vector extractor 120 may specify a region corresponding to a candidate object from among frame feature vectors, and extract a feature vector value corresponding to the specified region and set it as an object feature vector.

Here, since the object feature vector is set differently for each candidate object, candidate objects can be distinguished using the object feature vector. For example, when the same candidate object A appears consecutively on a plurality of frame images, the object feature vectors of the same candidate object A may be set identically or very similarly on each frame image. On the other hand, in the case of different candidate objects, the object feature vector has an error value of at least a set value or more. Accordingly, candidate objects having the same object feature vector may be determined to correspond to the same object, and candidate objects having different object feature vectors may be determined to correspond to different objects.

Meanwhile, the size of the candidate object may change for each frame image according to the position or movement direction of the candidate object, and accordingly, the size of the object feature vector may be set differently for each frame image. Accordingly, the feature vector extraction unit 120 may uniformly transform the size of each object feature vector to a predetermined size using an interpolation method.

As described above, the feature vector extractor 120 according to an embodiment of the present invention may perform detection of a candidate object and generation of an object feature vector of the candidate object, using the frame feature vector. That is, since candidate object detection and object feature vector generation can be performed by using the frame feature vector extracted once, efficient operation can be performed and the operation speed can be improved.

The identification information setting unit 130 may set identification information for candidate objects using an object feature vector. That is, the identification information setting unit 130 may distinguish each candidate object by using the object feature vector of the candidate objects, and may assign and display identification information to each of the identified candidate objects. For example, a plurality of candidate objects are included in FIG. 4(a), and each of the candidate objects corresponds to a different object. Accordingly, the identification information setting unit 130 can distinguish each candidate object and set "ID" as identification information for the identified candidate object as "138", "147", "128", and "153", respectively. have. In this case, the identification information setting unit 130 may set identification information for each candidate object using an identification information database.

Specifically, the identification information setting unit 130 may search the identification information database d for a registration feature vector corresponding to the object feature vectors of candidate objects extracted from each frame image. Here, in the identification information database d, a registration feature vector and identification information corresponding to each registration feature vector may be matched and stored. Accordingly, the identification information setting unit 130 may search the identification information corresponding to the object feature vector of the candidate object in the identification information database d.

Here, when the registration feature vector corresponding to the object feature vector is searched, the identification information matched with the registration feature vector may be extracted from the identification information database d and set as the identification information of the corresponding candidate object. On the other hand, there may be a case where the registered feature vector corresponding to the object feature vector is not searched, and in this case, the corresponding object feature vector first appears in the target video. Accordingly, the identification information setting unit 130 can newly generate identification information corresponding to the object feature vector, and register the newly created identification information and the object feature vector in the identification information database d, and the identification information database d ) Can be updated.

Here, if the object feature vector matches the registered feature vector within a preset error range, the identification information setting unit 130 may determine that the object feature vector corresponds to the registered feature vector. That is, even in the case of the same candidate object, the object feature vector for each frame image may contain some errors, and thus the identity may be determined in consideration of an error range.

Since the identification information setting unit 130 sets identification information of each candidate object by referring to the identification information database (d), all candidate objects having the same object feature vector among candidate objects included in the target video have the same identification information. Can be set to For example, if a candidate object with object feature vector A appears in frame image 3-10 and then appears in frame image 20-26 again, the candidate object is frame image 3-10 and frame image 20-26. It may be set to have the same identification information b in the frame image.

Additionally, the identification information database (d) may be provided for each target video, but according to an embodiment, a single identification information database (d) for all target videos received by the image masking device 100 is provided. It is also possible. In this case, if the object feature vector included in the newly provided target video is the same as the object feature vector included in the previous target video, the same identification information as the identification information set in the previous video may be set. That is, the same identification information can be set for the same candidate object even for different target videos.

The tracking information generating unit 140 may track candidate objects having the same identification information within a continuous frame image, and generate tracking information for candidate objects for each identification information. That is, tracking information can be generated by combining the location information and identification information of the candidate object extracted from each frame image, and the location change of the candidate object naturally continuing in the continuous frame image can be tracked using the tracking information. have. Here, the tracking information may include identification information of the candidate object, frame information such as a frame number in which the candidate object appears, location information and size information of the candidate object included in the frame image, and the like. In addition, depending on the embodiment, it is possible to further include information on a pose of a candidate object, facial feature points, clothing feature points, and the like in the tracking information. One target video may include a plurality of tracking information, and each tracking information may be generated for each identification information of a candidate object.

Depending on the embodiment, the tracking information generation unit 140 tracks the same candidate object by using the difference value of the object feature vector of each candidate object included in the continuous frame image and the change in position and size of the candidate objects. It is also possible to do it. For example, when trying to track a first candidate object included in a first frame image, tracking with a first candidate object for a plurality of candidate objects included in a second frame image consecutive to the first frame image You can calculate the error value. Thereafter, the candidate object having the minimum tracking error value may be determined as the same candidate object as the first candidate object.

Specifically, the tracking error value can be calculated using Error = (V1-V2) + a × (d1-d2) + b × (s1-s2). Here, Error is the tracking error value, V1 is the object feature vector of the first candidate object included in the first frame image, V2 is the object feature vector of the second candidate object included in the second frame image, and d1 is the first from the reference point. The distance to the center point of the candidate object, d2 is the distance from the reference point to the center point of the second candidate object, s1 is the area of the first candidate object, s2 is the area of the second candidate object, and a, b are weights and It corresponds.

In general, since the time difference between consecutive frame images is very short, it is difficult for the same candidate object to move a large distance between consecutive frame images or to rapidly increase or decrease the area. Accordingly, the smaller the difference value between the object feature vectors and the smaller the change in distance and area is, the higher the probability of corresponding to the same candidate object. Therefore, using the above-described tracking error value, it is possible to track a candidate object within a continuous frame image. In addition, according to an embodiment, if V1-V2 is equal to or greater than a set limit error, the second candidate object may be determined as a different candidate object without calculating a tracking error value. That is, since it corresponds to the case where the object feature vector is out of the set error range, the second candidate object may be determined to be different from the first candidate object.

The error detection unit 150 may detect an error in setting identification information for a candidate object. For example, when a candidate object having the same identification information is partially omitted in a continuous frame image, the error detection unit 150 may determine that an error has occurred in setting identification information for the candidate object.

In general, since frame images of a target video are photographed at very short time intervals, candidate objects that exist simultaneously in adjacent frame images and whose positions in each frame image are close to each other can be expected to have the same identification information. Therefore, in the case where the existing candidate object suddenly disappears within the adjacent frame image, it can be considered that the candidate object did not actually move, but that an error occurred in setting identification information for the candidate object.

For example, a first frame image at a time point t-1, a second frame image at a time point t, and a third frame image at a time point t+1 exist, and the (x1) of the first frame image and the third frame image , y1) There may be a case where a candidate object with {id=0} exists. At this time, if the candidate object with {id=0} does not exist in (x1, y1) of the second frame image, or if the object with {id=1} suddenly is located at (x1, y1), the identification It can be determined that an error has occurred in information setting, etc. Accordingly, the error detection unit 150 may display the occurrence of an error and notify the user or the like.

When receiving the masking input, the masking unit 160 may extract candidate objects identical to the identification information included in the masking input using the tracking information, and mask the extracted candidate objects. That is, the masking unit 160 may selectively perform masking on selected candidate objects, and at this time, candidate objects to be masked may be specified using identification information.

Here, the masking input may be generated using identification information input from a user, or may be generated using identification information extracted according to a preset selection algorithm. For example, the image masking apparatus 100 may provide a separate masking selection interface to the user, and the user may select identification information corresponding to a candidate object to be masked using the masking selection interface. In this case, the masking unit 160 may receive a masking input through the masking selection interface, and may mask candidate objects corresponding to the received masking input.

In addition, in the case of an embodiment in which candidate objects to be masked are automatically extracted using a separate selection algorithm set in the image masking apparatus 100, all candidate objects included in a specific section in the target video are identified to be masked. The information may be extracted, or identification information corresponding to a specific candidate object included in the target video may be extracted and set to mask. For example, by using a selection algorithm, identification information corresponding to a specific gender or age group may be extracted from among candidate objects included in the target video.

Meanwhile, the masking unit 160 may extract frame images in which a candidate object corresponding to the selected identification information appears, using the tracking information, and set a masking area corresponding to the position of the candidate object appearing in the extracted frame image. have. Thereafter, the set masking area may be masked and de-identified so that users cannot identify it. Here, the masking area may be limited to an area corresponding to a face among candidate objects and set.

In this case, the masking unit 160 may mask the masked area in a manner such as blurring or mosaic processing. Here, the blurring may be implemented using a low-pass filter, and according to an embodiment, masking may be performed using image substitution covering the masking area with a single color or a specific pattern, a separate image, animation, character, or the like.

Additionally, the masking unit 160 may encode an image obtained by synthesizing the masking, and may output the masked image through an image output unit (not shown). Meanwhile, according to an embodiment, the masking unit 160 may separately store masking information for a masking area set in the target video and the target video. That is, after separately storing the original file of the target video, the target video may be masked using masking information when the target video is played.

In addition, according to an exemplary embodiment, when the original file of the target video is to be played, access rights to the original file may be requested, and when there is no access right, masking for the target video may be applied and provided. Here, the access right may be checked through various types of authentication such as password, fingerprint recognition, and iris recognition.

Additionally, the image masking apparatus 100 according to an embodiment of the present invention may allow a user to select candidate objects to be masked, and in this case, may further include components for enhancing user selection convenience.

Specifically, the object tracking image generator 170 may generate an object tracking image by overlaying the candidate object and identification information for each candidate object on the target video using the tracking information. That is, as shown in Fig. 4(a), an object tracking image can be generated by displaying a bounding box representing a candidate object and identification information together with a target video.

In this case, the user may check the object tracking image and check the candidate object being tracked and identification information of each candidate object. Accordingly, the user may select an object to perform masking processing among a plurality of candidate objects by referring to the object tracking image.

In addition, the masking selection interface display unit 160 may provide a masking selection interface so that a user may select candidate objects to be masked. Specifically, the masking selection interface can be implemented as shown in FIG. 5. That is, the masking selection interface may display an identification information list in which identification information corresponding to candidate objects included in the target video is arranged, and candidate objects having the same identification information appear in the target video, appearing section information and the frame The number of images, etc. may be displayed. In addition, a frame image in which the candidate object is displayed may be further included so that the user can check candidate objects corresponding to each identification information.

Accordingly, the user may select candidate objects to be masked in the target video through the masking selection interface, and may select candidate objects to be masked by checking a checkbox displayed next to the identification information list. That is, instead of collectively performing masking on all objects included in the target video, it is possible for the user to selectively set candidate objects to be masked.

Meanwhile, as shown in FIG. 3, the image masking apparatus 100 according to an embodiment of the present invention may include a physical configuration such as a processor 10 and a memory 40, and the memory 40 One or more modules configured to be executed by the processor 10 may be included therein. Specifically, one or more modules may include a frame input module, a feature vector extraction module, an identification information setting module, a tracking information generation module, an error detection module, a masking module, an object tracking image generation module, and a masking selection interface display module. .

The processor 10 may execute various software programs and an instruction set stored in the memory 40 to perform various functions and perform a function of processing data. The peripheral interface unit 30 may connect input/output peripheral devices of the image masking device 100 to the processor 10 and the memory 40, and the memory controller 20 may be the processor 10 or the image masking device 100. When an element of accesses the memory 40, a function of controlling memory access may be performed. Depending on the embodiment, the processor 10, the memory controller 20, and the peripheral interface unit 30 may be implemented on a single chip, or may be implemented as separate chips.

The memory 40 may include a high-speed random access memory, one or more magnetic disk storage devices, a nonvolatile memory such as a flash memory device, and the like. In addition, the memory 40 may further include a storage device located away from the processor 10 or a network attached storage device that is accessed through a communication network such as the Internet.

Meanwhile, as shown in FIG. 3, the image masking apparatus 100 according to an embodiment of the present invention includes an operating system in the memory 40, a frame input module corresponding to an application program, a feature vector extraction module, and identification. It may include an information setting module, a tracking information generation module, an error detection module, a masking module, an object tracking image generation module, and a masking selection interface display module. Here, each of the modules is a set of instructions for performing the above-described functions, and may be stored in the memory 40.

Accordingly, in the image masking apparatus 100 according to an embodiment of the present invention, the processor 10 may access the memory 40 and execute a command corresponding to each module. However, the frame input module, feature vector extraction module, identification information setting module, tracking information generation module, error detection module, masking module, object tracking image generation module and masking selection interface display module include the above-described frame input unit, feature vector extraction unit, Since the identification information setting unit, the tracking information generation unit, the error detection unit, the masking unit, the object tracking image generation unit, and the masking selection interface display unit respectively correspond to each other, detailed descriptions are omitted here.

6 is a flowchart illustrating an image masking method according to an embodiment of the present invention.

6, the image masking method according to an embodiment of the present invention includes a frame input step (S10), a candidate object detection step (S20), an object feature vector extraction step (S30), an identification information setting step (S40), It may include a tracking information generation step (S50) and a masking step (S60). Here, each of the steps may be performed by the image masking apparatus according to an embodiment of the present invention.

Hereinafter, an image masking method according to an embodiment of the present invention will be described with reference to FIG. 6.

In the frame input step S10, a frame image in a frame unit may be generated by decoding the input target video. Specifically, the image masking device may receive an MPEC-encoded target video from an image capturing device. In this case, after decoding the target video in the form of a compressed MPEC file, each frame of the target video is captured to generate a frame image. I can.

In the candidate object detection step S20, a frame feature vector corresponding to each frame image may be extracted from the frame image, and a candidate object may be detected from the frame feature vector. Here, the image masking apparatus may generate a frame feature vector by using pixel information of a frame image, such as location information and pixel value information of each pixel included in the frame image. According to an embodiment, a convolutional neural (CNN) Network), a machine learning algorithm such as RNN (Recurrent Neural Network), or a method of extracting statistical characteristics of an image such as Histogram of Oriented Gradient (HOG) and Local Binary Pattern (LBP).

Thereafter, the image masking apparatus may detect candidate objects included in each frame image by using the generated frame feature vector. Here, the candidate object is an object that can be a target of masking, and features of the candidate objects, such as shape or luminance, may be learned in advance using a machine learning algorithm. That is, it can be learned using machine learning algorithms such as CNN, RNN, Principal Component Analysis (PCA), Logistic Regression, and Decision Tree, and by using this, a region corresponding to a candidate object can be detected from each frame image. . In this case, the extracted candidate objects may be specified and displayed on the frame image as a bounding box or a segmentation mask.

In the object feature vector extraction step (S30), an object feature vector corresponding to each candidate object may be extracted from the frame feature vector using the position information of the detected candidate objects. That is, a region corresponding to a candidate object may be specified from among frame feature vectors, and a feature vector value corresponding to the specified region may be extracted and set as an object feature vector. Here, since the object feature vector is set differently for each candidate object, candidate objects can be distinguished using the object feature vector.

Meanwhile, the size of the candidate object may change for each frame image according to the position or movement direction of the candidate object, and accordingly, the size of the object feature vector may be set differently for each frame image. Accordingly, the size of each object feature vector can be uniformly transformed into a predetermined size using interpolation.

In the identification information setting step S40, identification information for candidate objects may be set using an object feature vector. That is, each candidate object may be distinguished by using the object feature vectors of the candidate objects, and identification information may be assigned and displayed for each of the distinguished candidate objects. In this case, identification information for each candidate object may be set using an identification information database.

Specifically, a registration feature vector corresponding to the object feature vectors of candidate objects extracted from each frame image may be searched in the identification information database. Here, in the identification information database, the registration feature vector and the identification information corresponding to each registration feature vector may be matched and stored. Accordingly, identification information corresponding to the object feature vector of the candidate object can be searched in the identification information database.

Here, when the registration feature vector corresponding to the object feature vector is searched, the identification information matched with the registration feature vector may be extracted from the identification information database and set as the identification information of the candidate object. On the other hand, there may be a case where the registered feature vector corresponding to the object feature vector is not searched, and in this case, the corresponding object feature vector first appears in the target video. Accordingly, identification information corresponding to the object feature vector can be newly created, and the identification information database can be updated by registering the newly created identification information and the object feature vector in the identification information database.

Here, if the object feature vector matches the registered feature vector within a preset error range, it can be determined that the object feature vector corresponds to the registered feature vector. That is, even in the case of the same candidate object, the object feature vector for each frame image may contain some errors, and thus the identity may be determined in consideration of an error range.

In the tracking information generation step S50, candidate objects having the same identification information are tracked within a continuous frame image, and tracking information for candidate objects may be generated for each identification information. That is, tracking information can be generated by combining the location information and identification information of the candidate object extracted from each frame image, and the location change of the candidate object naturally continuing in the continuous frame image can be tracked using the tracking information. have. Here, the tracking information may include identification information of the candidate object, frame information such as a frame number in which the candidate object appears, location information and size information of the candidate object included in the frame image, and the like. In addition, depending on the embodiment, it is possible to further include information on a pose of a candidate object, facial feature points, clothing feature points, and the like in the tracking information. One target video may include a plurality of tracking information, and each tracking information may be generated for each identification information of a candidate object.

Depending on the embodiment, it is also possible to track the same candidate object by using the difference value of the object feature vector of each candidate object included in the continuous frame image and the position and size change of the candidate objects. Specifically, the tracking error value can be calculated using Error = (V1-V2) + a × (d1-d2) + b × (s1-s2). Here, Error is the tracking error value, V1 is the object feature vector of the first candidate object included in the first frame image, V2 is the object feature vector of the second candidate object included in the second frame image, and d1 is the first from the reference point. The distance to the center point of the candidate object, d2 is the distance from the reference point to the center point of the second candidate object, s1 is the area of the first candidate object, s2 is the area of the second candidate object, and a, b are weights and It corresponds.

On the other hand, although not shown, according to an embodiment, an error detection step may be further included. That is, in the error detection step, an error in setting identification information for a candidate object can be detected, and when an error in setting identification information is detected, the occurrence of an error can be notified to a user. For example, it may happen that some candidate objects having the same identification information are missing in a continuous frame image. In this case, since the interval between each frame image is very short, there is an error in setting identification information for the candidate object. It can be determined that it has occurred.

In the masking step (S60), upon receiving the masking input, candidate objects identical to identification information included in the masking input may be extracted using tracking information, and masking of the extracted candidate objects may be performed. That is, masking may be selectively performed on selected candidate objects, and candidate objects to be masked may be specified using identification information. Here, the masking input may be generated using identification information input from a user, or may be generated using identification information extracted according to a preset selection algorithm.

Meanwhile, in the masking step (S60), frame images in which a candidate object corresponding to the selected identification information appears may be extracted using the tracking information. Thereafter, a masking area may be set corresponding to a position of a candidate object appearing in the extracted frame image, and the set masking area may be masked to de-identify so that users cannot identify it. Here, the masking area may be limited to an area corresponding to a face among candidate objects and set.

In this case, the image masking apparatus may mask the masked area in a manner such as blurring or mosaic processing. Here, blurring can be implemented using a low-pass filter. In addition, depending on the embodiment, it is possible to mask the masking area using a single color, a specific pattern, a separate image, an animation, or an image substitution covering a character.

Additionally, the image masking apparatus may encode an image obtained by synthesizing the masking, and may output the masked image using an image output unit or the like. Meanwhile, depending on the embodiment, it is possible to separately store masking information for a masking area set in the target video and the target video. That is, after separately storing the original file of the target video, the target video may be masked using masking information when the target video is played.

In addition, according to an exemplary embodiment, when the original file of the target video is to be played, access rights to the original file may be requested, and when there is no access right, masking for the target video may be applied and provided. Here, the access right may be checked through various types of authentication such as password, fingerprint recognition, and iris recognition.

Additionally, the image masking method according to an embodiment of the present invention may allow a user to select candidate objects to be masked, and in this case, may further include configurations for enhancing user selection convenience.

Specifically, an object tracking image may be generated by further including an object tracking image generation step (not shown), and overlaying the candidate object and identification information for each candidate object on the target video using the tracking information. That is, an object tracking image may be generated by displaying a bounding box representing a candidate object and identification information together with a target video.

In this case, the user may check the object tracking image and check the candidate object being tracked and identification information of each candidate object. Accordingly, the user may select an object to perform masking processing among a plurality of candidate objects by referring to the object tracking image.

In addition, a masking selection interface may be provided so that the user may select candidate objects to be masked by further including a displaying step (not shown) of the masking selection interface. Specifically, the masking selection interface may display an identification information list in which identification information corresponding to candidate objects included in the target video is arranged, and candidate objects having the same identification information appear in the target video. The number of frame images that have appeared may be displayed. In addition, it may further include an image of a frame image in which the candidate object appears so that the user can check candidate objects corresponding to each identification information.

Accordingly, the user may select candidate objects to be masked in the target video through the masking selection interface, and may select candidate objects to be masked by checking a checkbox displayed next to the identification information list. That is, instead of collectively performing masking on all objects included in the target video, it is possible for the user to selectively set candidate objects to be masked.

The present invention described above can be implemented as a computer-readable code in a medium on which a program is recorded. The computer-readable medium may be one that continuously stores a program executable by a computer, or temporarily stores a program for execution or download. In addition, the medium may be a variety of recording means or storage means in a form in which a single piece of hardware or several pieces of hardware are combined, but is not limited to a medium directly connected to a computer system, and may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks, and And ROM, RAM, flash memory, and the like, and may be configured to store program instructions. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or storage medium managed by a server. Therefore, the detailed description above should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

The present invention is not limited by the above-described embodiments and the accompanying drawings. It will be apparent to those of ordinary skill in the art to which the present invention pertains, that components according to the present invention can be substituted, modified, and changed within the scope of the technical spirit of the present invention.

1: image recording device 100: image masking device
110: frame input unit 120: feature vector extraction unit
130: identification information setting unit 140: tracking information generation unit
150: error detection unit 160: masking unit
170: object tracking image generation unit 180: masking selection interface display unit
200: identification information database
S10: frame input step S20: candidate object detection step
S30: object feature vector extraction step S40: identification information setting step
S50: tracking information generation step S60: masking step

Claims (15)

Generating a frame image in units of frames by decoding the input target video;
Extracting a frame feature vector corresponding to each frame image from the frame image, and detecting a candidate object from the frame feature vector;
Extracting an object feature vector corresponding to each candidate object from the frame feature vector using the position information of the detected candidate objects;
Setting identification information for the candidate objects by using the object feature vector;
Tracking candidate objects having the same identification information within a continuous frame image, and generating tracking information for the candidate objects for each identification information; And
Upon receiving the masking input, extracting candidate objects identical to the identification information included in the masking input using the tracking information, and performing masking on the extracted candidate objects,
The step of setting the identification information
Searching an identification information database for a registered feature vector corresponding to the object feature vector;
When a registration feature vector corresponding to the object feature vector is searched, extracting identification information matched with the registration feature vector from the identification information database and setting it as the identification information of the candidate object; And
If the registration feature vector corresponding to the object feature vector is not searched, newly generating identification information of the candidate object, and newly registering the object feature vector and the identification information in the identification information database,
The step of performing the masking
And extracting frame images in which a candidate object corresponding to the identification information appears using the tracking information, and setting a position of the candidate object included in the extracted frame image as a masking area.
delete The method of claim 1, wherein the step of searching in the identification information database
And if the object feature vector matches the registered feature vector within a preset error range, it is determined that the object feature vector corresponds to the registered feature vector.
The method of claim 1,
The image masking method further comprising an error detection step of determining that an error has occurred in setting the identification information for the candidate object when a part of the candidate object having the same identification information is omitted in the continuous frame image.
The method of claim 1, wherein the tracking information
And at least one of identification information of the candidate object, frame information in which the candidate object appears, location information and size information of the candidate object included in the frame image.
The method of claim 1,
And generating an object tracking image in which the candidate object and the identification information for each candidate object are overlaid on the target video by using the tracking information.
The method of claim 6,
Masking including an identification information list in which identification information corresponding to candidate objects included in the target video is arranged, information about an appearance section in which candidate objects having the same identification information appear in the target video, and a frame image in which the candidate object appears The image masking method further comprising the step of displaying a selection interface.
The method of claim 1, wherein the masking input
An image masking method comprising generating by using identification information input from a user or by extracting the identification information according to a preset selection algorithm.
delete The method of claim 1, wherein performing the masking
An image masking method comprising masking the masked area by using blurring, mosaic processing, or image replacement.
The method of claim 1, wherein performing the masking
And storing masking information for a masking area set in the target video and the target video, respectively, and masking and playing the target video using the masking information when the target video is played.
The method of claim 11, wherein performing the masking
A video masking method comprising: requesting access rights when playing the target video, and masking and playing the target video when there is no access right.
A computer program stored in a medium to execute the image masking method of any one of claims 1, 3 to 8, and 10 to 12 in combination with hardware.
A frame input unit that decodes the received target video and generates a frame image in units of frames;
Extracting a frame feature vector corresponding to each frame image from the frame image, detecting a candidate object from the frame feature vector, and using the position information of the detected candidate objects, each candidate object from the frame feature vector A feature vector extracting unit for extracting an object feature vector corresponding to;
An identification information setting unit for setting identification information for the candidate objects by using the object feature vector;
A tracking information generator configured to track candidate objects having the same identification information within a continuous frame image, and generate tracking information for the candidate objects for each identification information; And
Upon receiving a masking input, comprising a masking unit for extracting candidate objects identical to the identification information included in the masking input by using the tracking information, and performing masking on the extracted candidate objects,
The identification information setting unit
When a registration feature vector corresponding to the object feature vector is searched in an identification information database, and when a registration feature vector corresponding to the object feature vector is searched, identification information matched with the registered feature vector is extracted from the identification information database, Set as the identification information of the candidate object, and if the registration feature vector corresponding to the object feature vector is not searched, the identification information of the candidate object is newly generated, and the object feature vector and the identification information are stored in the identification information database Register for a new one,
The masking part
And extracting frame images in which a candidate object corresponding to the identification information appears using the tracking information, and setting a position of the candidate object included in the extracted frame image as a masking area. delete

Images