JP2013098750A - Image processing apparatus, image processing method, image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which can mask a region to which a mask target is reflected even when time is required for determining whether being a mask target or not in a case where a subject requiring mask processing is framed in.SOLUTION: A mask candidate cutting part 222 cuts a region to be masked as a mask candidate region from a decoded image frame. A similarity determination part 234 confirms, when the latest mask candidate region is similar to a mask target, the mask candidate as a mask target. A mask supplement part 260 tracks temporally behind movement of the region confirmed as a mask target by the determination with the similarity determination part 234, and obtains movement locus. Subsequently, the movement locus is extrapolated temporally ahead and movement before detection of the mask target is estimated. A supplement mask to be applied to the region obtained by the extrapolation is generated.

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program. The present invention relates to an image processing apparatus and the like for automatically performing mask processing on an image area in which an object relating to privacy protection such as a face of a specific person is shown.

  In recent years, it has become possible to easily publish moving images taken by individuals from the personal computer terminals and portable terminals of the individuals via the Internet. However, a moving image may include information that can identify an individual, such as a person, a vehicle, and a place name or address indicating a captured area. It is not desirable from the viewpoint of privacy protection that a moving image including such information is disclosed as it is.

In view of this, there has been proposed a method of performing a process of automatically mosaicing a specific person or license plate in a moving image (for example, Patent Document 1 and Patent Document 2).
In the methods disclosed in Patent Literature 1 and Patent Literature 2, first, face information of a person who needs to perform mosaic processing is prepared in advance as reference information.
When the reference information matches the face in the image, a process for applying a mosaic to the face portion is performed.

  With this method, it is possible to perform a process for applying a mosaic to the face of a specific person, but it is not possible to apply a mosaic to an image before the face in the image matches the reference information. For example, if a person who is subject to mosaicing slowly slides in the screen, the mosaicing process starts only after the face is completely within the screen. Will be exposed to the screen although it is a part of the face.

To solve the above problem, Patent Document 3 (Japanese Patent Laid-Open No. 2010-233133) proposes the following method. In Patent Document 3, a video buffer is prepared, and a moving image is temporarily stored in the video buffer so as to go back several frames.
When a target to be mosaicked, that is, a mask target is detected, the frames in the video buffer are traced back in order to search again for a portion related to the mask target. Thereby, even before the mask target completely enters the screen, it is possible to apply a mask to a region where the mask target is partially reflected, that is, to apply a mosaic.

JP 2001-086407 A JP 2004-062560 A JP 2010-233133 A

  However, there is a limit to the capacity of the video buffer. Assuming that the moving image is 30 frames / second, it is the maximum that about 10 seconds can be stored at the longest (paragraph 0056 of Patent Document 3). Therefore, if you take a picture while slowly shaking the camera, or if a person passes slowly in front of a fixed camera, the amount of time that the person who needs masking slides in the screen is the capacity of the video buffer. It can happen that it becomes longer. As a result, a frame that has been discharged from the video buffer beyond the retention time of the video buffer is released with insufficient mask processing.

  Although it is conceivable to simply increase the capacity of the video buffer, there is a limit to the capacity (time) in which a moving image can be temporarily buffered no matter how large the capacity is. Also, a method of once detecting a mask target and then searching again from the first frame and picking up a portion related to the mask target can be considered, but this takes too much time and effort.

  Therefore, the object of the present invention is that even if it takes a long time to determine whether or not the subject is to be masked, such as when a subject requiring mask processing enters the frame, An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program that can reliably mask an imaged area.

Therefore, the present invention provides
An image processing apparatus (100) for searching image areas to be masked from moving image data and adding a mask to them,
A mask candidate cutout unit (222) that cuts out an area that can be a mask target from the image frames of the moving image data as a mask candidate area based on a pre-registered cutout reference list;
A similarity calculation unit (233) that compares the latest mask candidate region with the mask target reference list and calculates the similarity between the two based on a mask target reference list registered in advance;
A similarity determination unit (234) that compares the calculated similarity with a predetermined threshold and determines the mask candidate region as a mask target when the similarity is equal to or greater than the predetermined threshold;
The movement trajectory obtained by temporally tracking the movement of the region determined as the mask target by the determination by the similarity determination unit (234) is temporally extrapolated forward and applied to the region obtained by this extrapolation. A mask replenishment unit (260) for creating a replenishment mask;
A mask applying unit (240) for adding a mask to the region of the replenishment mask created by the mask replenishment unit (260) in addition to the region determined as a mask target by the determination by the similarity determination unit (234); An image processing apparatus is provided.

In the image processing apparatus (100),
The mask replenishment unit (260)
A coordinate information acquisition unit (261) that sequentially acquires coordinate information of areas determined as mask targets by the determination by the similarity determination unit (234);
An extrapolation data holding unit (262) that holds the coordinate information acquired by the coordinate information acquisition unit (261);
A motion extrapolation (263) that extrapolates the data held in the extrapolation data holding unit (262) in time and estimates a motion before the mask target is detected;
A moving mask creating unit (264) that creates a mask that moves so as to be applied to the motion obtained by extrapolation by the motion compensation exterior (263) may be provided.

In the image processing apparatus,
A position determination unit (250) for determining whether or not the distance between the region determined as the mask target by the determination by the similarity determination unit (234) and the image frame frame is equal to or less than a predetermined threshold;
When the distance is less than or equal to a predetermined threshold, the position determination unit (250) may activate the mask supplementation unit (260).

In the image processing apparatus,
A position determination unit (250) for determining whether or not the region determined as the mask target by the determination by the similarity determination unit (234) has moved in the center direction of the image frame;
The position determination unit (250) may activate the mask replenishment unit (260) when the region determined as the mask target is moving in the center direction of the image frame.

Furthermore, the present invention provides
An image processing method for searching image areas to be masked from video data and adding a mask to them,
A mask candidate extraction step (ST103, ST104) for extracting, as a mask candidate area, an area that can be a mask target from the image frame of the moving image data based on a pre-registered extraction reference list;
A similarity calculation step (ST106) for comparing the latest mask candidate area with the mask target reference list and calculating the similarity between the two based on a mask target reference list registered in advance;
A similarity determination step (ST107) in which the calculated similarity is compared with a predetermined threshold, and when the similarity is equal to or higher than the predetermined threshold, the mask candidate region is determined as a mask target;
The movement trajectory obtained by temporally tracking the movement of the area determined as the mask target by the determination in the similarity determination step (ST107) is temporally extrapolated forward (ST123), and obtained by this extrapolation. A mask replenishment step (ST124) for creating a replenishment mask over the region;
A mask applying step (ST125) for adding a mask to the region of the replenishment mask created in the mask replenishment step (ST124) in addition to the region determined as a mask object by the determination in the similarity determination step (ST107); An image processing method is provided.

Furthermore, the present invention provides a computer incorporated in an image processing apparatus that searches image areas to be masked from moving image data and adds a mask to them.
A mask candidate cutout unit (222) that cuts out an area that can be a mask target from the image frames of the moving image data as a mask candidate area based on a pre-registered cutout reference list;
A similarity calculation unit (233) that compares the latest mask candidate region with the mask target reference list and calculates the similarity between the two based on a mask target reference list registered in advance;
A similarity determination unit (234) that compares the calculated similarity with a predetermined threshold and determines the mask candidate region as a mask target when the similarity is equal to or greater than the predetermined threshold;
The movement trajectory obtained by temporally tracking the movement of the region determined as the mask target by the determination by the similarity determination unit (234) is temporally extrapolated forward and applied to the region obtained by this extrapolation. A mask replenishment unit (260) for creating a replenishment mask;
A mask applying unit (240) for adding a mask to the region of the replenishment mask created by the mask replenishment unit 260 in addition to the region determined as a mask target by the determination by the similarity determination unit (234); An image processing program characterized by functioning is provided.

The figure which shows an example of the moving image data recording / reproducing system which this invention assumes. The figure which shows a mode that the moving image is image | photographed. The figure which shows the example of the image | photographed moving image. The functional block diagram of a mask process part in 1st Embodiment. The figure which shows typically the state which detected the mask object area | region which has a similarity more than a threshold value in 1st Embodiment. The figure which shows the example of the coordinate information hold | maintained in the data holding part for extrapolation in 1st Embodiment. The figure which shows the movement locus | trajectory of the detected mask object area | region in 1st Embodiment. The figure which shows the example of a graph which extrapolated the movement locus | trajectory of mask object to the front in time in 1st Embodiment. The figure which shows the example of a movement mask in 1st Embodiment. 6 is a flowchart showing an operation procedure from moving image shooting (ST10) to output of masked data (ST60) in the first embodiment. 6 is a flowchart illustrating a process of creating a mask processing metafile in the first embodiment. The figure which shows a mode that a mask candidate is cut out in 1st Embodiment. The flowchart which shows the process of producing a movement mask in 1st Embodiment. The figure which shows the example of the moving image which added the mask in 1st Embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 shows an example of a moving image data recording / reproducing system 900 assumed by the present invention.
Each individual takes an image of his / her favorite subject with the video camera 100. For example, you may go out and take pictures of the city, or take a picture of a family playing in a park. The moving image data shot in this way is stored in the moving image memory 120 built in the video camera 100.

  When the photographer returns home, he connects the video camera 100 to the personal computer 910 and sends the captured moving image data to the server 930 on the Internet 920 via the personal computer 910. Then, the moving image data is stored in the server 930, and a third party can view the moving image using the Internet 920.

However, the moving images shot in this way may include images that are inconvenient for the public. For example, there is a possibility that the face of a family member, the face of a passerby unintentionally reflected, and information for identifying an individual, such as a name, a telephone number, or a car license plate, may be reflected.
If these images related to personal information are published on the Internet as they are, there is a risk that problems may arise from the viewpoint of privacy protection. Therefore, it is necessary to perform mask processing on the image portion related to privacy.

  Here, as an example of a moving image used in the following description, it is assumed that a specific person CA is photographed as shown in FIG. At this time, the photographer P moves the video camera 100 so as to slowly change the direction from right to left, and shoots so that the person CA slowly enters the frame in the frame. Then, for example, as shown in FIG. 3, a moving image in which the person CA gradually enters the frame from the left of the frame will be shot. Here, according to the above-described prior art, if about 80% of the face is reflected in the frame as in the frame F40, it can be determined that this is a mask target. Therefore, the face of the person CA can be masked on the frames F50 and F60 after the frame F40. However, until the frame F40, although the face of the person CA is shown, only a part of the face is shown, so it cannot be determined that it is a mask target. It will be released. Therefore, in this embodiment, the mask is also applied to the frame before it is determined that it is a mask target.

(Configuration of the first embodiment)
The configuration of the first embodiment will be described.
FIG. 4 is a functional block diagram of the mask processing unit 200 and also shows the main elements of the video camera 100. In the present embodiment, it is assumed that the mask processing unit 200 is built in the video camera 100, but a mask processing unit may be provided as one function of the personal computer 910. The moving image data captured by the imaging unit 110 of the video camera 100 is temporarily stored in the moving image memory 120. Here, the imaging unit 110 of the video camera 100 includes a lens unit, a CCD (photoelectric conversion element) circuit, and a predetermined logic circuit, and generates moving image data (frame) from a moving image (video) signal obtained by photographing a subject. The moving image data includes luminance data and color data. As the moving image memory 120, for example, a flash memory can be used.

  The mask processing unit 200 includes a data input unit 210, a mask candidate acquisition unit 220, a mask target determination unit 230, a mask flag assignment unit 240, a position determination unit 250, a mask supplementation unit 260, and a metafile creation unit 270. A mask adding unit 280, and a data output unit 290.

  The data input unit 210 reads the moving image data stored in the moving image memory 120 and outputs it to the subsequent stage. The data input unit 210 includes a decoding unit 211 and outputs an image frame obtained by decoding moving image data.

  Here, not all image frames are required when the purpose is not only to view a moving image but only to mask processing. Therefore, among the I picture, P picture, B picture, etc., only the frame corresponding to the purpose of processing may be provided from the data input unit 210 to the subsequent circuit. For example, only an I picture and a P picture may be used for mask processing. Of course, mask processing may be performed using all frames.

  The mask candidate acquisition unit 220 cuts out only an image area that may require a privacy mask. For example, when it is necessary to put a mask on the face of Mr. A who is a specific person, an area having a skin color of a predetermined area or more may be a human face, and such an area is cut out as a mask candidate.

  The mask candidate acquisition unit 220 includes a cutout reference list storage unit 221 and a mask candidate cutout unit 222.

  The cut-out reference list storage unit 221 stores a cut-out reference list for detecting a region to be cut out as a mask candidate. The extraction reference list may be preset, may be automatically generated from the features to be masked, or may be set and input by the user. Examples of the cut-out reference list include a skin color area having a predetermined area or more, and list data of characters and numbers.

The meaning of having the list data of letters and numbers as the cut-out reference list will be supplemented.
As for privacy, in addition to the face, there are name, organization name (company name or school name), address display, telephone number, car number, etc. Therefore, it is necessary to put a privacy mask on a phone number such as 090-XXXX-XXXX. Therefore, if a number such as “0” or some characters appear in the image frame as described later, it is necessary to cut them out as candidates for masking. Finally, if the numbers are arranged like “090-”, the number string is masked. If the pattern looks just “0” or “0”, Does not require masking.

As another example, if characters are finally arranged like "Yokohama", "Yokohama City XXX Ward", or "Horizontal X Gakuen XX School", they will be masked. If there is only one character, there is no need for masking because it has nothing to do with privacy.
As characters, alphabets, Hangul, and Chinese abbreviations should be added to the extracted reference list so that foreign languages can be handled.

The mask candidate cutout unit 222 searches the image frames sequentially input from the data input unit 210 and searches for whether there is a match with the cutout reference list stored in the cutout reference list storage unit 221. If there is an image frame that matches the cut-out reference list, the region portion is cut out as a mask candidate region, and is output to the subsequent circuit.
At this time, it is assumed that the extracted mask candidate area holds information associated with the original frame. The information associated with the frame is, for example, the frame number of the cut-out source and the coordinates within the frame.

Next, the mask target determination unit 230 verifies whether or not the mask candidate area acquired by the mask candidate acquisition unit 220 is truly a mask target.
The mask target determination unit 230 includes a feature amount calculation unit 231, a mask target reference list storage unit 232, a similarity calculation unit 233, and a similarity determination unit 234.

The feature amount calculation unit 231 calculates a feature amount for the latest mask candidate extracted by the mask candidate extraction unit 222. The feature amount is a combination of various index values representing the characteristics of the target image. The index value representing the feature point of the target image, the index value representing the distance relationship between the feature points, and the index value representing the size of the feature portion. , An index value representing the contour of the target image, an index value representing luminance, an index value representing color, and the like.
The feature amount calculation unit 231 sends the calculated feature amount to the similarity calculation unit 233.

  The mask target reference list storage unit 232 stores reference information for detecting an area to be masked. Reference information stored in the mask target reference list storage unit 232 includes a face of a specific person who needs a privacy mask (A's face, B's face ...), a telephone number, and a vehicle number. Examples include a numeric string for identifying, a name / organization name (company name or school name), a character string for identifying address display, and the like. These pieces of reference information are stored as a set with each feature amount.

The similarity calculation unit 233 compares the feature amount calculated by the feature amount calculation unit 231 with the feature amount stored in the mask target reference list storage unit 232, and calculates the similarity. When calculating the similarity, the index values of the mask candidates and the index values of the mask target reference list are matched to each other, and the similarity is obtained by comprehensively evaluating the degree of similarity between the two. If there is a possibility that the mask candidate is a skin color area and a human face, it is meaningless to calculate the similarity by comparing this with the feature quantity of the telephone number. The reference information that is matched in the case of the skin color region is the face of the specific person.
There are various ways to select appropriate reference information for calculating similarity, but for example, it may be judged by color or brightness, or the highest similarity will be obtained after matching with all reference information You may choose.

  The similarity determination unit 234 compares the similarity calculated by the similarity calculation unit 233 with a predetermined threshold. When the similarity exceeds the threshold, it is determined that the mask candidate area is a mask target area. When the similarity exceeds the threshold value, the fact is notified to the mask flag assignment unit 240 and the position determination unit 250.

The mask flag assigning unit 240 sets a mask flag in the mask target area. In setting the mask flag in the mask target area, it goes without saying that the mask flag is given to the mask target area that is determined to be a mask target by the determination by the similarity determination unit 234.
As shown in FIG. 5, since the face is reflected in the frame F40, the frame F50, the frame F60, and the like so that the degree of similarity is equal to or greater than the threshold value, the mask flag adding unit 240 masks the area where these faces are reflected. A flag is given. Here, it is necessary to also mask the video (frame F10 to frame 30) from when the mask target (subject) starts to frame in until it is determined to be the mask target. This is a position determination unit described later. 250 and the mask replenishment unit 260.

  The position determination unit 250 checks the positional relationship between the determined position of the mask target area and the frame of the image frame. Specifically, it is determined whether or not the distance between the mask target region and the frame is equal to or less than a predetermined threshold value. When the distance between the mask target area and the frame frame is equal to or smaller than the predetermined threshold, the position determination unit 250 supplements the frame (frame F10 to frame 30) before the frame (frame F40) in which the mask target was detected. In order to create a typical mask, the mask replenishment unit 260 is activated. Then, after the mask replenishment unit 260 is activated, the coordinate information of the mask target area acquired via the similarity determination unit 234 is output to the mask replenishment unit 260.

Here, when the distance between the mask target area and the frame frame is equal to or smaller than a predetermined threshold value, it is considered that the mask target is gradually framed in from the end of the frame. For example, referring to FIG. 5, in the frame F40, the mask target area is very close to the frame frame. In this case, in the previous frame (frame F10 to frame F30) of the frame F40, there is a possibility that the mask target (subject) is not sufficiently contained in the frame and is not recognized as the mask target.
In the case of the position of the mask target area, the center coordinates Cc of the mask target area may be used, or any of the four corners of the rectangle may be used if the mask target area is a square. What is necessary is just to adjust the value of a predetermined threshold value and the determination method appropriately according to how to take the position of the mask target area.

Next, the mask supplement unit 260 will be described.
The mask supplementation unit 260 creates a mask to be applied to the frame before the mask target is detected based on the movement trajectory of the mask target region.
The mask supplementation unit 260 includes a coordinate information acquisition unit 261, an extrapolation data holding unit 262, a motion compensation external 263, and a movement mask creation unit 264.

  The coordinate information acquisition unit 261 extracts the coordinate information of the mask target area acquired via the similarity determination unit 234 and the position determination unit 250 and stores the extracted coordinate information in the extrapolation data holding unit 262. Here, the coordinate information is the coordinate value of the mask target area, the size (size) of the mask target area, and the time stamp of the frame to which the mask target area belongs. To hold. The extrapolation data holding unit 262 is a temporary memory, and holds the coordinate value, size, time, and frame number of the mask target area together as shown in FIG.

The motion extrapolation 263 estimates the motion before the mask target is detected based on the coordinate information held in the extrapolation data holding unit 262 by the extrapolation method.
For example, when the data held in the extrapolation data holding unit 262 is plotted on a graph as shown in FIG. 6, the movement trajectory of the mask target area after the frame F40 is obtained as shown in FIG. Here, for the sake of simplicity, only the change in the x coordinate is shown, but the y coordinate and the region size can be similarly plotted as a function of time. Then, in order to estimate the movement trajectory of the mask target in the frame before the frame F40 by the extrapolation method, the graph of FIG. 7 is extended before the time. Then, as shown in FIG. 8, the estimated trajectory of the mask target in the frame temporally prior to the frame F40 is obtained.

  The movement mask creating unit 264 creates a mask that moves so as to be applied to a mask target that moves along this locus with respect to the estimated locus obtained as shown in FIG. For example, as shown in FIG. 9, a mask that moves so as to gradually enter the frame from the left with respect to the frames F10 to F39 is created. The movement mask creation unit 264 outputs information on the movement mask created in this way to the mask flag assignment unit 240. The movement mask information is information on the coordinate value of the movement mask, the size (size) of the movement mask, and the frame number to which the movement mask is added.

  The metafile creation unit 270 acquires information about the area to be masked from the mask flag assignment unit 240. The area to be masked is a mask target area, and the information about the area to be masked is information such as the coordinates, size, and frame number of the mask target area. Again, the mask target area includes not only the area determined as the mask target by the threshold determination by the similarity determination unit 234 but also the area to which the movement mask created by the movement mask creation unit 264 is added. Of course. The metafile creation unit 270 creates a metafile that includes a command for applying a mask to the mask target area. The metafile creation unit 270 outputs the created mask metafile to the mask addition unit 280.

  The mask addition unit 280 masks the moving image data in accordance with the mask processing command of the metafile. At this time, the mask adding unit 280 may start the masking process after waiting for the creation of metafiles related to all the moving picture data stored in the moving picture memory 120. That is, after the decoding process for creating the metafile, the moving image data is extracted from the moving image memory 120 and decoded again. At this time, it is not necessary to activate the mask candidate acquisition unit 220 and the mask target determination unit 230, and the decoded image data is sent only to the mask addition unit 280, where mask addition according to the metafile is performed.

  Alternatively, the mask processing in the mask adding unit 280 may be advanced with a predetermined time delay from the creation of the metafile. For example, if the mask candidate cutout unit 222 has a delay of several minutes with respect to the time stamp of the frame being processed, the necessity of the mask processing for the frame is determined.

  The moving image data masked by the mask adding unit 280 is output from the data output unit 290. Then, for example, the data is once stored in a hard disk of a personal computer and then sent to a server on the Internet.

(Operation of the first embodiment)
Subsequently, the operation of the first embodiment will be described.
As shown in FIG. 10, the flow from the moving image shooting (ST10) by the user to the output of the masked data (ST60) is as follows: shooting (ST10), storage of moving image data (ST20), and advance preparation of mask processing (ST30) ), Creation of a mask processing metafile (ST40), mask addition (ST50), and data output (ST60).

As described above, a moving image is shot as shown in FIG. 2 (ST10), and the moving image data is stored in the moving image memory 120 (ST20). Next, preliminary preparation for mask processing is performed (ST30). As advance preparation for mask processing (ST30), the user stores information on the mask target in the mask target reference list storage unit 232. In this example, the facial feature amount of the child CA is stored in the mask target reference list storage unit 232.
In storing the feature amount of the face of the child CA in the mask target reference list storage unit 232, for example, data of the face of the child CA photographed separately may be set in the video camera 100. Alternatively, an area showing the face of the child CA may be designated from the moving picture taken this time, and this may be set in the video camera 100 as a mask target. The video camera 100 automatically calculates the feature amount from the area set as the mask target and stores it in the mask target reference list storage unit 232.

  In accordance with this, it is necessary to register the extracted reference list in the extracted reference list storage unit 221. In this regard, the video camera 100 may automatically generate a cut-out reference list from information in the mask target reference list storage unit 232. Alternatively, if a person's face is set as a mask target, a skin color region having a predetermined area or more may be automatically added to the extraction reference list.

  When such advance preparation (ST30) is completed, the mask processing unit 200 is activated to create a mask processing metafile (ST40). The mask processing metafile creation step (ST40) will be described with reference to the flowchart of FIG. The captured moving image data is stored in the moving image memory 120 and then input from the data input unit 210 to the mask processing unit 200 (ST101). The moving image data is decoded by the decoding unit 211 of the data input unit 210 (ST102), and the image frames are sequentially sent to the mask candidate cutout unit 222. The mask candidate cutout unit 222 searches for a mask candidate region that matches the cutout reference list in the image frame. At the stage of frame F00, since there is no skin color area, there is no area that is a mask candidate (NO in ST130). If there is no mask candidate area, the search target is moved to the next image frame.

As the frame F00 shifts to the frame F10, the child CA gradually enters the frame. If the face of the child CA exceeds the predetermined area and enters the frame, this matches the cut-out reference list (YES in ST103).
For example, assume that the skin color area exceeds a predetermined value in the frame F10. Then, the mask candidate cutout unit 222 cuts out a skin color region R10 having a predetermined area or more as a mask candidate (ST104) (see FIG. 12). At this time, as described above, the frame number of the cut-out frame F10 and the coordinates in the frame F10 are added to the data of the cut-out area (R10).

When the mask candidate region (R10) is cut out, the feature amount calculation unit 231 calculates the feature amount for this region (R10) (ST105). The calculated feature amount is sent to the similarity calculation unit 233, where the similarity based on the comparison with the mask target reference list is obtained (ST106).
The calculated similarity is compared with a threshold value in the similarity determination unit 234. Since there is no point similar to the facial feature amount of the child CA in the region R10, the similarity does not exceed the threshold Th (NO in ST107). In this case, the region R10 is not determined as a mask target (NO in ST107), and the process proceeds to the next image frame.

  As the area from the frame F10 to the frame F30 is gradually increased, the area in which the child is reflected gradually increases, so all skin color areas are subject to extraction as mask candidates (YES in ST103), but the child's face is still fully framed. In the similarity determination (ST107), the similarity does not exceed the threshold value Th (NO in ST107).

  Consider the point in time when the frame F40 is further processed from the frame F30 and becomes the frame to be processed. In the frame F40, about 80% of the face in the region R40 is in the frame. Since the region R40 is a skin color region having a predetermined area or more, it is cut out as a mask candidate by the mask candidate cutout unit 222 (ST104), and furthermore, about 80% of the face of the child CA is shown, so that it is included in the mask target reference list. A certain index (feature value of a child's face) has a high correlation at a plurality of points, and a large value is calculated as the similarity. Therefore, the similarity S40 obtained for R40 is equal to or greater than the threshold Th (YES in ST107).

  As described above, when the similarity S40 is equal to or greater than the threshold Th, the similarity determination unit 234 notifies the mask flag giving unit 240 that an area having the similarity S equal to or greater than the threshold Th has appeared. Then, a mask flag is assigned by the mask flag assigning unit 240 to the region R40 where the similarity is equal to or greater than the threshold (ST108).

  Next, when the mask target appears in this way, position determination (ST109) by the position determination unit 250 is executed. That is, the distance between the area R40 to be masked and the frame is compared with a predetermined threshold (ST110).

  If the distance between the mask target (R40) and the frame frame exceeds a predetermined threshold (NO in ST110), it is determined that it is not necessary to replenish the moving mask. Accordingly, the metafile creation unit 270 creates a metafile for masking the mask target region R40 according to the mask flag attached to the region R40 determined as the mask target (ST111).

  On the other hand, in ST110, when the distance between the mask target region (R40) and the frame is equal to or smaller than the predetermined threshold (YES in ST110), there is a possibility that the necessary mask is not put on the subject that has entered the frame. Therefore, in this case, the mask replenishment unit 260 is activated to start moving mask creation (ST120).

The process of creating the movement mask will be described with reference to the flowchart of FIG.
In order to create a movement mask, the coordinate information acquisition unit 261 acquires the coordinate information of the mask target area (ST121).
This is because the coordinate information acquisition unit 261 performs the similarity determination unit 234 and the frame F40 and subsequent frames with respect to the frame after the frame F40 when the ST101 to ST107 are sequentially performed on the frames after the mask target is determined (frames after the frame F40). The coordinate information of the area to be masked is acquired via the position determination unit 250. Accordingly, coordinate information such as coordinates, size, and time stamp of the frame is stored in the extrapolation data holding unit 262 for the mask target area after the frame F40 (see FIG. 6).
In ST122, it is determined whether or not a necessary amount of data has been collected. As shown in FIG. 8, the motion extrapolation 263 determines whether or not there is a sufficient amount of data for estimating the motion of the mask target in the temporally previous frame by the extrapolation method. .

The criteria for determining whether the number of data is sufficient may be set as appropriate.
For example, a fitting curve is drawn with respect to the coordinate information after the frame F40 using a regression line, a function curve of an appropriate degree, an appropriate spline curve, or the like (see FIG. 7). If the residual with respect to several tens of frames (from the frame F40 to around the frame F60) counted from the frame (frame F40) first determined as a mask target is equal to or less than a predetermined value, the data can be applied to the extent that extrapolation can be applied. You may judge that they have gathered.
It should be noted that the weight of the contribution increases as the distance from the frame first determined as the mask target (frame F40) increases, and the motions in the frames immediately after the frame first determined as the mask target (frame F40) are more strongly reflected. You may make it do.

  When sufficient data is collected to apply the extrapolation method (YES in ST122), the movement trajectory of the mask target region is extrapolated (ST123), and the motion before the mask target is detected is estimated (see FIG. 8). ). In this way, the movement mask creation unit 264 creates a movement mask so as to affect the movement of the mask target estimated by extrapolation. As a result, as shown in FIG. 9, a mask for a frame (frame F10 to frame F30...) Before the mask target is detected is created.

  Information on the created moving mask is output to the mask flag assigning section 240, and the mask flag assigning section 240 assigns a mask flag to the area of the moving mask (ST125).

  The mask target data provided with the mask flag is sent to the metafile creation unit 270, and a metafile for instructing mask processing is created (ST111). All frames of the moving image data stored in the moving image memory 120 are verified (YES in ST112), and the mask processing metafile is completed.

  The moving image data is masked according to the metafile thus created (ST50). That is, mask adding section 280 masks the area specified in the metafile for the decoded moving picture frame data (ST50). Then, as shown in FIG. 14, mask processing is applied from the point where the child starts to appear in the frame, and the child's privacy can be protected.

According to 1st Embodiment provided with such a structure, there can exist the following effects.
(1) In this embodiment, the mask supplementing unit 260 is provided, and the motion of the mask target is estimated by the extrapolation method on the frame before the mask target is detected, and the moving mask is applied.
Thereby, it is possible to mask an image from when the mask target (subject) starts to be framed in until the mask target (subject) is determined to be the mask target, and privacy can be reliably protected.

(2) In this embodiment, coordinate information is held in the extrapolation data holding unit 262 when replenishing the moving mask. However, since these are not numerical information but merely numerical values, the extrapolation data holding unit 262 has a small capacity. Also good. For example, there is a method of temporarily storing a predetermined amount of moving image data in a buffer so that a related part can be traced back in time after detecting a mask target as in the prior art, but this is very large. A large amount of memory is required. The image data overflowing from the buffer cannot be retrieved retroactively, and the portion may be disclosed without a privacy mask.
In this respect, in the present embodiment, only the coordinate information is necessary for creating the movement mask, and it is not necessary to store the image information itself, so the memory capacity may be extremely small. Also, no matter how many frames must be tracked to extrapolate the movement trajectory of the mask object, the memory capacity does not become a problem because it is only necessary to temporarily hold the coordinate information.
Therefore, a movement mask with high accuracy can be created after collecting a sufficient amount of coordinate information if necessary.

(3) A position determination unit 250 is provided, and the position determination unit 250 determines whether the newly detected mask target area is close to the frame frame, thereby determining whether or not a moving mask is necessary. Simply, every time a new mask target is detected, if a moving mask is replenished by extrapolation, unnecessary masks may be confused. In this respect, in this embodiment, the appearance position of the mask target region Based on the above, the necessity of the moving mask can be appropriately determined. Thereby, a movement mask can be appropriately added only when necessary.

(Modification 1)
In the first embodiment, the position determination unit 250 determines whether or not a moving mask is necessary based only on the distance between a newly appearing mask target area and the frame frame. In determining whether or not a moving mask is necessary, another determination requirement may be used.
For example, it may be determined whether or not the mask target area is moving in the center direction of the frame. Alternatively, both of the above two may be used, and the mask replenishment unit may be activated when both are satisfied. Furthermore, the mask replenishment unit may be activated if any one of the two requirements is satisfied.

The present invention is not limited to the above-described embodiments and modifications, and can be appropriately changed without departing from the spirit of the present invention.
In the above embodiment, the case where the privacy mask is put on the face of the specific person has been described, but conversely, the mask is not put on the specific person, and the face of the general person who happens to be reflected is put on the privacy mask. Of course, you may do it. In this case, it is only necessary to determine that the subject is “a person's face but not a specific person's face” by threshold determination, and the mask target reference list and similarity determination unit are modified for such determination. It will be apparent to those skilled in the art.

  Since “skin color” varies depending on race, it should be interpreted in consideration of the race to be masked. For example, if the country where the video camera is used is Asia, the skin color means, for example, the skin color of Mongoloid, that is, light yellow. Naturally, if the subject is a Caucasian or Niggloid, the skin color may be white or tan.

  In the above-described embodiment, an example in which the mask processing unit is activated and mask processing is performed after photographing is completed has been described. However, mask processing may be performed in parallel while photographing.

  The CPU and memory are arranged so as to function as a computer, and a predetermined control program is installed in this memory via a communication means such as the Internet or a recording medium such as a CD-ROM or a memory card. The function as each functional unit described in the above embodiment may be realized by operating a CPU or the like with a program.

DESCRIPTION OF SYMBOLS 100 ... Video camera, 110 ... Video camera imaging part, 120 ... Movie memory, 200 ... Mask processing part, 210 ... Data input part, 211 ... Decoding part, 220 ... Mask candidate acquisition part, 221 ... Extraction reference list storage , 222 ... Mask candidate cutout part, 222 ... Sequential mask candidate cutout part, 222 ... Mask candidate cutout part, 230 ... Mask target determination part, 231 ... Feature quantity calculation part, 232 ... Mask target reference list storage part, 233 ... Similarity calculation unit, 234 ... Similarity determination unit, 240 ... Mask flag assignment unit, 250 ... Position determination unit, 260 ... Mask supplementation unit, 261 ... Coordinate information acquisition unit, 262 ... Extrapolation data holding unit, 263 ... Movement Supplementary external, 264 ... movement mask creation unit, 270 ... metafile creation unit, 280 ... mask addition unit, 290 ... data output unit, 900 ... moving image data recording / playback Stems, 910 ... PC, 920 ... the Internet, 930 ... server.

Claims (6)

An image processing apparatus for searching image areas to be masked from moving image data and adding a mask to them,
Based on a pre-registered cut-out reference list, a mask candidate cut-out unit that cuts out an area that can be a mask target from among the image frames of the moving image data as a mask candidate area;
Based on a pre-registered mask target reference list, a similarity calculation unit that calculates the similarity of both by comparing the latest mask candidate region with the mask target reference list;
A similarity determination unit that compares the calculated similarity with a predetermined threshold and determines the mask candidate region as a mask target when the similarity is equal to or higher than the predetermined threshold;
Estimating the temporal movement trajectory of the area determined as the mask target using a movement trajectory obtained by tracking the movement of the area determined as the mask target by time determination by the similarity determination unit; A mask replenishment unit that creates a replenishment mask over the area obtained by the estimated movement trajectory in time;
A mask applying unit that adds a mask to the region of the replenishment mask created by the mask replenishment unit in addition to the region determined as a mask target by the determination by the similarity determination unit. apparatus. The mask replenishment unit
A coordinate information acquisition unit that sequentially acquires coordinate information of regions determined as mask targets by the determination by the similarity determination unit;
An extrapolation data holding unit for holding the coordinate information acquired by the coordinate information acquisition unit;
Extrapolation of the data held in the extrapolation data holding unit in time and extrapolating the movement before the mask target is detected by extrapolating in advance in time,
The image processing apparatus according to claim 1, further comprising: a moving mask creating unit that creates a mask that moves so as to be subjected to a motion obtained by extrapolation by the motion compensation external. A position determination unit that determines whether or not the distance between the region determined as the mask target by the determination by the similarity determination unit and the image frame is equal to or less than a predetermined threshold;
The image processing apparatus according to claim 1, wherein the position determination unit activates the mask replenishment unit when the distance is equal to or less than a predetermined threshold. A position determination unit that determines whether or not the region determined as a mask target by the determination by the similarity determination unit is moving in the center direction of the image frame;
The position determination unit activates the mask replenishment unit when the region determined as the mask target is moving in the center direction of the image frame. An image processing apparatus according to 1. An image processing method for searching image areas to be masked from video data and adding a mask to them,
Based on a pre-registered cut-out reference list, a mask candidate cut-out step of cutting out an area that can be a mask target from among the image frames of the moving image data as a mask candidate area;
Based on a pre-registered mask target reference list, a similarity calculation step of comparing the latest mask candidate region with the mask target reference list and calculating the similarity between the two,
A similarity determination step of comparing the calculated similarity with a predetermined threshold and determining the mask candidate region as a mask target when the similarity is equal to or higher than the predetermined threshold;
Estimating the temporal movement trajectory of the area determined as the mask target using a movement trajectory obtained by tracking the movement of the area determined as the mask target in time by the determination by the similarity determination step; A mask replenishment step for creating a replenishment mask over the region obtained by the estimated movement trajectory in time;
A mask applying step of adding a mask to the region of the replenishment mask created in the mask replenishment step in addition to the region determined as the mask target by the determination in the similarity determination step. Method. An image processing program for searching image areas to be masked from video data and adding a mask to them,
This computer,
Based on a pre-registered cut-out reference list, a mask candidate cut-out unit that cuts out an area that can be a mask target from among the image frames of the moving image data as a mask candidate area;
Based on a pre-registered mask target reference list, a similarity calculation unit that calculates the similarity of both by comparing the latest mask candidate region with the mask target reference list;
A similarity determination unit that compares the calculated similarity with a predetermined threshold and determines the mask candidate region as a mask target when the similarity is equal to or higher than the predetermined threshold;
Estimating the temporal movement trajectory of the area determined as the mask target using a movement trajectory obtained by tracking the movement of the area determined as the mask target by time determination by the similarity determination unit; A mask replenishment unit that creates a replenishment mask that is applied to the region by the estimated movement trajectory in time,
An image processing program that functions as a mask applying unit that adds a mask to a region of the replenishment mask created by the mask replenishment unit in addition to a region determined as a mask target by determination by the similarity determination unit.

Images