JP2010009305A - Image-processing device, image-processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image-processing device, an image-processing method and an image-processing program capable of leveling load as processing total with respect to the processing for extracting and magnifying a face image from image data. <P>SOLUTION: An image-processing device (10) is equipped with: a face detection part (11) in which the processing load for the detection varies depending on the size of the face when the face is detected from the inputted image, and a magnifying part (12) in which the processing load for magnification varies depending on the size of the face image when information regarding the position and the size of the face image detected by the face detection part is received and, based on this information, the face image is extracted and the face image is magnified to the image of the designated size. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for detecting and enlarging a face image from an image.

  As a technique for extracting a specific image included in image data, for example, a face image detection technique is known. Furthermore, there are many needs not only to extract face images, but also to enlarge and display the extracted face images.

In the technique described in Patent Document 1, a target subject such as a human face is detected from an image displayed on the screen, and the target subject is enlarged and displayed according to the position and size of the detected target subject. A display area and an enlargement ratio are determined. Then, an enlarged display is performed on the screen according to the determined display area and enlargement ratio.
JP 2005-31888 A

  By the way, when extracting and enlarging a face image as a series of processes, it is required to equalize the load required for the process as a problem to be solved.

  That is, it is considered that the process of extracting and enlarging a face image is often used by being incorporated in a system such as a video display device rather than functioning as a single device. Therefore, even if each of the load of the process for extracting the face image and the load of the process for enlarging the face image is controlled individually, the total load of the series of processes may vary. It will fluctuate system load and reduce its function.

  However, Patent Document 1 has no description regarding or suggesting that the load as such processing total is equalized.

  The present invention has been made in view of such circumstances, and relates to a process of extracting and enlarging a face image from image data, and an image processing apparatus, an image processing method, and an image processing apparatus capable of leveling a load as a total processing The purpose is to provide a program.

  In the present invention for solving the problem, when detecting a face from an input image, the detection processing load varies depending on the size of the detected face, and the face detected by the face detection unit When information on the position and size in the image is received, and a face image is extracted based on this information and the face image is enlarged to an image of a predetermined size, the processing load of enlargement varies depending on the size of the face image. An image processing apparatus comprising: an enlargement unit.

  Further, according to the present invention, when detecting a face from an input image, a face detection step for detecting a face so that the detection processing load decreases as the size of the detected face increases. An information acquisition step for receiving information on the position and size of the face in the image, an extraction step for extracting the face image based on this information, and the face to be enlarged when the face image is enlarged to a predetermined size image And an enlargement step for enlarging the face so that the enlargement processing load increases as the image size increases.

  The present invention is also an image processing program, wherein when detecting a face from an input image, a face detection step for detecting a face so that the detection processing load is reduced as the size of the detected face increases; An information acquisition step for receiving information on the position and size of the face detected in the face detection step in the image, an extraction step for extracting the face image based on this information, and expanding the face image to an image of a predetermined size In this case, the image processing program causes the computer to execute an enlargement step of enlarging the face so that the enlargement processing load increases as the size of the face image to be enlarged increases.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and a program capable of leveling a load as a total processing regarding a process of extracting and enlarging a face image from image data.

  FIG. 1 is a diagram in which an image processing apparatus according to this embodiment is incorporated into a digital television. This digital TV has a function of detecting a face from a received video signal and enlarging and displaying the face part in real time.

The digital television 1 includes a tuner control unit 2, a separation unit 3, an audio demodulation unit 4, a video demodulation unit 5, a synthesis unit 6, an AV output unit 7, a display unit 8, a control unit 9, and an image processing device 10.
The image processing apparatus 10 includes a face detection unit 11, an enlargement unit 12, and a DB (database) 13.

  The tuner control unit 2 is a broadcast receiving unit that receives broadcast data corresponding to BS, CS, and terrestrial waves and performs channel selection processing for channel selection, and is connected to a corresponding antenna (not shown). The separation unit 3 separates the received broadcast data into audio data and video data, and outputs them to the audio demodulation unit 4 and the video demodulation unit 5, respectively. The audio demodulator 4 demodulates the audio data. The video demodulator 5 demodulates video data.

  The face detection unit 11 of the image processing apparatus 10 detects a face image included in the demodulated video data. The enlargement unit 12 enlarges the portion of the image data corresponding to the detected face image to a face image of a predetermined size. The DB 13 stores a dictionary (details will be described later) used when the face detection unit 11 detects a face.

The synthesizer 6 synthesizes the audio data and the enlarged face image. The AV output unit 7 outputs the synthesized video to the display unit 8, and the display unit 8 displays the synthesized video on a display such as liquid crystal or plasma.
The control unit 9 is a main CPU (Central Processing Unit) provided for controlling the operation of each unit of the digital television 1.

FIG. 2 is a diagram for explaining a processing example of the image processing apparatus 10.
When a plurality of faces are displayed in the image data demodulated by the video demodulation unit 5, the face detection unit 11 preferentially executes detection of a large size face, and the largest face portion in the image data Is detected. Then, information for specifying the detected face portion is calculated. For example, a rectangular area including a face portion is defined, and information on the position and size for specifying the rectangle is extracted. The information may be the coordinates of one vertex or center of the rectangle and the vertical and horizontal sizes of the rectangle, or the coordinates of the four vertices of the rectangle.

The enlargement unit 12 extracts a face portion from the image data based on the information from the face detection unit 11, calculates an enlargement ratio for enlarging the image to a predetermined size, and enlarges the face image according to the enlargement ratio. To do. For example, it is conceivable to enlarge the display so that it can be displayed in a full high-definition size.
In addition, when enlarging, the image processing is performed in addition to enlarging the size. When a face is detected, the location of the facial features, such as skin and hair, is known, so the image processing parameters are changed to the optimal values for the facial features and the facial part is magnified particularly beautifully. Is possible. For example, parameters relating to sharpness in image processing can be changed between a face portion and a portion other than the face, and the face portion can be processed to have a soft color tone.

FIG. 3 is a diagram for explaining the flow of data between modules in the image processing apparatus 10.
Procedure (1) The image data from the video demodulator 5 is input to the face detector 11.
Procedure (2) Image data from the video demodulation unit 5 is input to the enlargement unit 12 from the face detection unit 11.
Procedure (3) From the face detection unit 11, information such as the position and size of the face detected based on the image data from the video demodulation unit 5 is input to the enlargement unit 11.
Procedure (4) The enlargement unit 11 outputs image data obtained by enlarging a part of the image data from the video demodulation unit 5 based on information such as the detected position and size of the face.

  Here, each module of the image processing apparatus 10 is configured to execute operations sequentially according to the above-described data flow. This is because the face image is extracted and enlarged in real time.

It is possible to input the image data from the video demodulating unit 5 in parallel to the face detecting unit 11 and the enlarging unit 12 and omit the data exchange shown in the procedure (2). However, in this case, since the enlargement unit 12 uses the result of the face detection unit 11, a buffer for temporarily storing the image data from the video demodulation unit 5 is provided in front of the enlargement unit 12 in order to realize a synchronized operation. There is a need.
FIG. 4 is a diagram for explaining a method of detecting a face by scanning an image input by the face detection unit 11.
Assume that the size of the image input from the video demodulation unit 5 is horizontal X pixels and vertical Y pixels. On the other hand, an area 20 for detecting a face is provided. The size of this area 20 is assumed to be horizontal a pixel and vertical b pixel.

  Then, while scanning the region 20 in the right direction every predetermined pixel (for example, one pixel), it is determined whether or not a face image having a predetermined pattern exists in the region 20. After scanning to the right end, the region 20 moves downward by a predetermined pixel (for example, 1 pixel) and scans similarly from the left end to the right end of the image. By repeating this operation, the face in the image is detected by scanning the region 20 from the upper left to the lower right of the image.

This operation continues until a face is detected in the area 20, and when a face is detected in the area 20, the scanning is terminated at that time. On the other hand, if the face pattern having a predetermined pattern does not exist even when the image is scanned by the area 20 of one size, the operation of changing the size of the area 20 and scanning again from the upper left to the lower right of the image is repeated.
The scanning range refers to a region to be scanned out of the entire input image region.

FIG. 5 is a flowchart showing the processing operation of the face detection unit 11. In response to a periodic activation instruction from the control unit 9, the face detection unit 11 is activated, for example, every one frame period, and executes the process according to the following procedure.
In step S01, the face detection unit 11 sets information such as a face size and a scanning range for detecting a face for the image data input from the video demodulation unit 5.
In the present embodiment, the initial size of the face is set to a large size, and the size is reduced each time scanning is repeated. That is, a face with a large size is detected preferentially. This is because it is considered that the main person (object) in the image usually has a large area. Therefore, a desired face image can be efficiently detected by preferentially detecting a large size face.

In step S02, it is checked whether or not the image has been scanned with all the sizes in the planned range.
In the case of No in step S02, that is, when there is a face size that has not been scanned yet, in step S02, the image is scanned with that size.

In the case of No in step S03, that is, when scanning is not yet completed, face detection processing is executed every time the region 20 is moved in step S04.
The face detection unit 11 reads a dictionary stored in the DB 13. This dictionary stores a plurality of types of standard face patterns (for example, face images in which front faces, sideways faces, male faces, female faces, etc. are combined). The face detection unit 11 compares and collates the image in the region 20 with the standard pattern of each face using, for example, a pattern matching technique. At this time, for example, even if a face image exists in the region 20, if the proportion of the face region is small compared to the standard pattern, it is recognized that there is no face.

In the case of No in step S05, that is, if the comparison / collation has not been completed for all the patterns in the dictionary, the face detection process in step S04 is repeatedly executed.
If Yes in step S05, that is, if the comparison and collation for all patterns in the dictionary have been completed, it is checked in step S06 whether a face has been detected.

If Yes in step S06, that is, if a face is detected, the further detection operation is stopped, and face detection determination processing is executed in step S07.
In the face detection determination process, the position where the face is detected (for example, the coordinates of the upper left corner of the area 20 at that time) and the size of the area 20 are extracted.
Then, the face detection process ends.

On the other hand, if No in step S06, that is, if no face is detected, the process returns to step S03, moves in the region 20, and repeats the face detection processing in steps S04 to S05.
If Yes in step S03, that is, if the scanning of the scanning range is completed, the process returns to step S01, the face size (area 20 size) is set to a small size, and the face detection processing in steps S02 to S06. repeat.

  In the case of Yes in step S02, that is, when a face is not detected as a result of scanning for all face sizes, face detection determination processing is executed in step S07. In the face detection determination process, information indicating that no face has been detected is created.

FIG. 6 is a flowchart showing the processing operation of the enlargement unit 12. The enlarging unit 12 executes processing in the following procedure based on the processing result of the face detecting unit 11.
In step S <b> 11, the enlargement unit 12 receives the face detection position, the size of the region 20, and the input image data from the face detection unit 11.

In step S12, an enlargement position and an enlargement rate are determined.
Here, the enlargement position is determined based on the face detection position and the size of the region 20. In other words, in the detected face image, if the image is enlarged as it is, the portion occupied by the face may be too large. In addition, it is desirable that the aspect ratio matches the enlarged aspect ratio because the original image is not deformed. Therefore, an area of a new size obtained by correcting the size of the area 20 is set as an enlarged position. Then, the enlargement ratio is determined based on this enlargement position. For example, when enlarging to 1920 × 1080 which is a full high-definition size, if the new size is a × b, the enlargement rate is obtained as 1920 / a (= 1080 / b).
In the case of a high-definition video, since the resolution is good, even if the image is enlarged, the image is not easily deteriorated. Therefore, the enlargement ratio may be limited to be a predetermined value or less.

In step S13, the image at the enlargement position is cut out from the input image, and in step S14, the image is enlarged according to the enlargement ratio.
At this time, the processing content is changed according to the feature or enlargement ratio of the original image. For example, softening the tone of the face, bringing the face color closer to the skin color, increasing the brightness, changing the resolution (gamma characteristic), etc., depending on the characteristics and magnification of the original image Adjust to execute a correct display. For this reason, what kind of image processing should be performed is determined by analyzing the original image. Specifically, the image processing content is determined by calculating a plurality of indexes based on color information, luminance information, distribution information, and the like for each pixel.

  In this way, by changing and executing the image processing parameters so as to be adapted during the enlargement process, the enlarged face image can be made a higher-definition desired image. Normally, when the face image is simply enlarged, the enlarged face often becomes unnatural, but this arrangement can greatly improve the enjoyment of viewing the enlarged image. it can.

In addition, when the original image is small, that is, when the enlargement ratio is large, an unnatural face may be obtained by performing the above-described processing. This is because the portion to be processed is small, so that the effect of the image processing cannot be sufficiently exhibited, but rather the reverse effect.
Therefore, in the present embodiment, when the original image is small, that is, when the enlargement ratio is large, control may be performed so as to suppress image processing.

  Next, the reason why load leveling can be achieved in the present embodiment will be described.

FIG. 7 is a diagram illustrating a load in the face detection process.
As described above, in the face detection process, face detection is performed by scanning the region 20 from the upper left to the lower right of the image. The concept of “processing cost” is introduced as a value representing the load at this time.

The face detection process includes an operation of scanning the area 20 and identifying whether or not there is a face in the area 20.
Here, a method is adopted in which the time required for the operation of comparing and collating with the face pattern stored in the dictionary is substantially constant regardless of the size of the area 20, that is, the face detection image size. Therefore, the processing cost is considered to be proportional to the number of times that the image is identified.
On the other hand, the number of times of identifying an image differs depending on the scanning method and the face detection image size.

  A processing cost when processing a full HD image (1920 × 1080) will be described.

FIG. 7A is a diagram illustrating a case where the size of the region 20 is 1000 × 1000, and the region 20 is scanned pixel by pixel from the upper left to the lower right.
The number of times of identification at this time is represented by Expression (1).
73,600 = (1920−1000) × (1080−1000) (1)
FIG. 7B is a diagram illustrating a case where the size of the region 20 is 25 × 25 and the region 20 is scanned pixel by pixel from the upper left to the lower right.
The number of times of identification at this time is expressed by Expression (2).
1,999,225 = (1920-25) × (1080-25) (2)
In the present embodiment, the size of the region 20 is gradually reduced from a large size to perform face detection scanning, but it can be seen that the identification count decreases as the face detection size increases. Therefore, the processing cost of face detection is smaller as the face detection size is larger, and is larger as the face detection size is smaller.

FIG. 8 is a diagram illustrating a load in the enlargement process.
As described above, the enlargement process according to the present embodiment captures the characteristics of the original image and adaptively changes the process according to the characteristics to obtain a high-quality enlarged image. Therefore, the processing cost of the enlargement process mainly depends on the load for accurately identifying the features of the original image.
Therefore, when the detected face size is large as shown in FIG. 8 (1), the identification portion is wide, and the identification cost increases. In addition, the image processing cost increases because the image processing portion covers a wide range.
Conversely, when the detected face size is small as shown in FIG. 8 (2), the identification part is narrow, so the identification cost is small, and the image processing part is narrow, so the image processing cost is low.
Therefore, the processing cost for face enlargement increases as the face detection size increases and decreases as the face detection size decreases.

FIG. 9 is a diagram illustrating the load state of the face detection unit 11 and the enlargement unit 12 according to the face detection size.
When detecting a large face, the load is small because the processing cost of the face detection unit 11 is low, but the processing cost increases and the load increases because the enlargement unit 12 has a large number of pixels for identification processing.
When detecting a small face, the load is large because the face detection unit 11 has a large processing cost, but the enlargement unit 12 has a small processing cost and a small load because the number of pixels to be identified is small.
If a face is not detected, as in the case of detecting a small face, the face detection unit 11 has a large processing cost because the processing cost is high, but the enlargement unit 12 does not need to perform identification processing, so the processing cost is It becomes smaller and the load becomes smaller.
As a result, the load required for the process of detecting and enlarging the face is leveled without greatly changing depending on the face detection size.

  In the present embodiment, the technique for detecting a face has been described. However, the present invention can be applied not only to face detection but also to face recognition such as recognition of a specific person. .

  Further, although an example in which one person's face is detected has been described in the present embodiment, the present invention can also be applied to a case where a plurality of faces exist in one video. In the above description, the face detection process is finished when the face is first found, but the process is finished with the top 10 people in the order in which the faces are detected, and the process is finished when the face is reduced to a certain size. It is possible to perform processing for detecting a plurality of faces. That is, the detection method and the number of detections given in the present embodiment are examples, and a method that can be easily estimated from this example is naturally included in the technical idea of the present invention.

  Further, in the present embodiment, the discussion has been made based on the face detection technique for detecting the entire screen in units of pixels, but the face detection method is not limited to this. That is, any face detection method can be applied as long as it is a face detection algorithm that preferentially detects a large face.

  In this embodiment, face detection has been described as an example. However, the present invention is not limited to face detection. For example, the present invention can be applied to a system that detects and enlarges an object. At this time, for example, in the image processing, unlike the face, variations in image processing such as increasing sharpness can be taken.

  Furthermore, the face detection method of the present embodiment can be speeded up. As the method, for example, (1) the process is performed centering on the position based on the previous face detection result, (2) the corner of the image is not processed, and (3) the face detection is performed for each frame. Alternatively, the speed may be increased by performing face detection processing only once every few frames.

  The typical components of the present invention can be illustrated as follows.

<Example 1>
When detecting a face from an input image, a face detection unit (11) whose detection processing load varies depending on the size of the detected face;
When receiving information on the position and size of the face detected in the image from the face detection unit, extracting the face image based on this information and enlarging the face image to an image of a predetermined size, An image processing apparatus (10) comprising: an enlargement unit (12) in which an enlargement processing load varies depending on an image size.

<Example 2>
The face detection unit (11) is configured such that the detection processing load decreases as the size of the detected face increases.

<Example 3>
The enlargement unit (12) increases the processing load for enlargement as the face image size increases.

<Example 4>
The face detection unit (11)
The image is scanned and searched so as to preferentially detect a face image of a large size in the image. When a face image is detected, the subsequent scanning is stopped.

<Example 5>
The face detection unit (11)
A part of the image is cut out while scanning the image with the size of the face to be detected, and it is determined whether or not a face corresponding to the size of the face is included in the cut out image.

<Example 6>
The face detection unit (11)
Whether or not a face is included in the cut-out image is determined by a pattern matching method using a standard face pattern.

<Example 7>
An image processing unit (12) for performing image processing on the face image before enlarging processing;
The image processing unit obtains at least one index representing the feature of the face image, and determines image processing contents to be applied to the face image according to the index.

<Example 8>
When detecting a face from an input image, a face detection step for detecting a face so that the processing load decreases as the size of the detected face increases,
An information acquisition step for receiving information on the position and size of the face detected in the face detection step in the image;
An extraction step of extracting a face image based on this information;
An image processing method comprising: an enlargement step of enlarging a face so that the processing load increases as the size of the face image to be enlarged increases when the face image is enlarged to an image of a predetermined size.

<Example 9>
When detecting a face from an input image, a face detection step for detecting a face so that the processing load decreases as the size of the detected face increases,
An information acquisition step for receiving information on the position and size of the face detected in the face detection step in the image;
An extraction step of extracting a face image based on this information;
An image processing program for causing a computer to execute an enlargement step of enlarging a face so that the processing load increases as the size of the face image to be enlarged increases when the face image is enlarged to an image of a predetermined size.

  Each function described in the above embodiment may be configured using hardware, or may be realized by reading a program describing each function into a computer using software. Each function may be configured by appropriately selecting either software or hardware.

  Furthermore, each function can be realized by causing a computer to read a program stored in a recording medium (not shown). Here, as long as the recording medium in the present embodiment can record a program and can be read by a computer, the recording format may be any form.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine suitably the component covering different embodiment.

1 is a diagram in which an image processing apparatus according to an embodiment is incorporated into a digital television. The figure explaining the process example of an image processing apparatus. The figure explaining the flow of data between the modules in an image processing device. FIG. 6 is a diagram for explaining a method of detecting a face by scanning an image input by a face detection unit. The flowchart which shows the processing operation of a face detection part. The flowchart which shows the processing operation of an expansion part. The figure explaining the load in face detection processing. The figure explaining the load in an expansion process. The figure explaining the load state of the face detection part by the face detection size, and an expansion part.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Digital television, 2 ... Tuner control part, 3 ... Separation part, 4 ... Audio demodulation part, 5 ... Image demodulation part, 6 ... Composition part, 7 ... AV output part, 8 ... Display part, 9 ... Control part, 10 Image processing apparatus, 11 Face detection unit, 12 Enlargement unit, 13 DB, 20 Area.

Claims (9)

When detecting a face from an input image, a face detection unit whose detection processing load varies depending on the size of the detected face;
When receiving information on the position and size of the face detected in the image from the face detection unit, extracting the face image based on this information and enlarging the face image to an image of a predetermined size, An image processing apparatus comprising: an enlargement unit in which an enlargement processing load varies depending on an image size.   The image processing apparatus according to claim 1, wherein the face detection unit is configured such that the detection processing load decreases as the size of the detected face increases.   The image processing apparatus according to claim 2, wherein the enlargement unit is configured such that the enlargement processing load increases as the size of the face image increases. The face detection unit
4. The scanning is performed by scanning the image so as to preferentially detect a large-sized face image in the image, and the subsequent scanning is stopped when the face image is detected. Image processing apparatus. The face detection unit
5. A part of the face is cut out while scanning the image with the size of the face to be detected, and it is determined whether or not the face corresponding to the size of the face is included in the cut out image. An image processing apparatus according to 1. The face detection unit
6. The image processing apparatus according to claim 5, wherein whether or not a face is included in the cut-out image is determined by a pattern matching method using a standard face pattern. An image processing unit that performs image processing on the face image before enlarging processing;
The image processing apparatus according to claim 3, wherein the image processing unit obtains at least one index representing the feature of the face image, and determines image processing content to be applied to the face image according to the index. . When detecting a face from an input image, a face detection step for detecting the face so that the detection processing load decreases as the size of the detected face increases.
An information acquisition step for receiving information on the position and size of the face detected in the face detection step in the image;
An extraction step of extracting a face image based on this information;
An enlargement step of enlarging the face so that the enlargement processing load increases as the size of the enlargement of the face image increases when the face image is enlarged to a predetermined size image. An image processing program,
When detecting a face from an input image, a face detection step for detecting the face so that the detection processing load decreases as the size of the detected face increases.
An information acquisition step for receiving information on the position and size of the face detected in the face detection step in the image;
An extraction step of extracting a face image based on this information;
When enlarging the face image to an image of a predetermined size, an image is caused to cause the computer to execute an enlargement step of enlarging the face so that the enlargement processing load increases as the size of the face image to be enlarged increases. Processing program.

Images