JP2006318151A - Digital image display device and digital image display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically correct a size of a lower end part or an upper end part of a digital image to create a natural image. <P>SOLUTION: This digital image display device is provided with: a face area decision means 3 distinguishing each face area from the digital image wherein a plurality of persons aligned are back-and-forth disposed; a line decision means 30 detecting the face area to be included in the same line from a position of the face area, and finding an average value of the face area belonging to each the line; an arithmetic means 31 setting magnification for the face area of each the line having a different size to be magnified/reduced on the basis of an average size of the face area of each the line found by the line decision means 30; and a CPU 1 magnifying/reducing an area of a photographic subject element of each the line on the basis of the magnification. The face area of each the line is magnified/reduced in not only a longitudinal direction but also lateral direction. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a digital image display device and a digital image display method capable of specifying a human face area as a subject element and enlarging / reducing it, and more particularly to a digital camera.

  Conventionally, a display device having a function of identifying a human face from a photographed digital image has been proposed (see, for example, Patent Document 1). As shown in FIG. 18, this is an image input unit (8) for inputting a face image for one frame, an image storage unit (80) for converting the face image into digital data, and storing it. A face area extraction unit (81) for detecting a face image, a pupil detection unit (82), a nostril detection unit (83), and an image development memory (56) are provided. In the image storage unit (80), the face image is binarized, noise is removed by a well-known labeling process or contraction / diffusion, and the face image shown in FIG. 19 is obtained. In the binarized face image of FIG. 19, the outline of a face in which black pixels are connected in a ring shape is shown, and the pupil and nostril are expressed in black (see, for example, Patent Document 2). In the face area extraction unit (81), left and right threshold lines B1 and B2 and upper and lower threshold lines C1 and C2 of the contour line are determined. The pupil detection unit (82) detects the pupil by raster scanning from the top to the bottom of the image in order to confirm that the binarized images within the threshold lines B1, B2, C1, and C2 are faces.

In the pupil detection unit (82), as shown in FIG. 20, there are a pupil dictionary obtained by photographing the shapes of the pupils of a large number of subjects in advance and a non-pupil dictionary obtained by photographing a part in the face that is completely different from the pupil. Stored. The pupil detection unit (82) determines whether or not the black image obtained by scanning is a pupil. Specifically, a high point is set as the similarity if the image approximates the image in the pupil dictionary, and a low point is set if the image is dissimilar. Similarly, if the image approximates the image in the non-pupil dictionary, a high point is set as the dissimilarity, and a low point is set if the images are not similar.
Next, the pupil detection unit (82) obtains an evaluation value by subtracting the similarity point with the non-pupil dictionary from the similarity point with the pupil dictionary. For example, since the image obtained by scanning the location A1 in FIG. 19 is eyebrows, the similarity point with the pupil dictionary is low, and the dissimilarity point is high. Therefore, the evaluation value is low and it can be seen that the scanned image is not a pupil.
The image obtained by scanning the location A2 is a pupil, the point of similarity with the pupil dictionary is high, and the point of dissimilarity is low. Therefore, the evaluation value is high and it can be seen that the scanned image is a pupil. In the pupil dictionary, various pupil states such as horizontal eyes and half eyes are prepared.

Next, the nostril detection unit (83) detects the nostril. The nostril detection unit (83) stores a nostril dictionary and a nostril dictionary obtained from the nostrils of a large number of subjects, and obtains an evaluation value in the same manner as the pupil detection unit (82). Since the nostril is below the pupil, the nostril is detected at a location A3 below the location A2. Thereby, it is confirmed that the images in the threshold lines B1, B2, C1, and C2 are faces, and the image in which the area is specified is stored in the image development memory (56). The area of the face area and the upper left coordinate K of the area are also known from the image in the image development memory (56). Even if there are a plurality of faces in the image, the face area is specified.
As shown in FIG. 21, when a plurality of persons (7), (7), and (7) arranged in front and rear are photographed with a camera, as shown in FIG. 22, the face of the front person (7) is large, A so-called tilt phenomenon occurs in which the face of the rear person (7) appears small. In view of this, a digital camera having a correction function for correcting this tilt phenomenon, specifically, for increasing a small digital image has been proposed (for example, see Patent Document 3). In the case of a digital image, the image is enlarged by creating interpolated pixels by software between the pixels.

JP 2000-259833 A JP 2003-338952 A JP 2004-72181 A

In the case of a group photo in which a plurality of humans are photographed side by side and front and back, the human face in the rear row appears smaller and the human face in the front row appears larger due to the tilt phenomenon. In this case, it is troublesome to manually specify the face to be enlarged / reduced and correct its size each time. In addition, discontinuity occurs at the boundary between the enlarged / reduced image and the adjacent image that is not enlarged / reduced, which is unnatural.
An object of the present invention is to automatically correct the size of the upper end or lower end of a digital image to create a natural image.

The digital image display device includes: a first determination unit configured to determine a region of each subject element from a plurality of digital images of the subject arranged in front and back in a row of subject elements arranged side by side;
Second determination means for detecting a region of the subject element that should be included in the same column from the position of the region of the subject element;
Means for obtaining an average value or an intermediate size of the area of the subject element belonging to each column;
Calculation means for setting a magnification for enlarging / reducing the subject element area of each column having a different size based on an average value or an intermediate size of the subject element areas of each column obtained by the calculation means;
Means is provided for enlarging / reducing the area of the subject element in each row based on the magnification.

  The area of the subject element that should be included in the same column, that is, the face region, is detected by the second determining means, and the size is determined based on the intermediate size of the face region in each column, specifically the average value. The face area of each different column is automatically enlarged / reduced. As a result, a natural image can be created even though a part of the image is enlarged / reduced.

(First embodiment)
Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this example, the digital image to be processed is a group photo in which a row of people (7) arranged side by side as shown in FIG. 3 is a group photo provided before and after the front row. The face of the person (7) is large and the face of the person (7) in the back row is small. That is, in this example, the subject element is a human face, but is not limited to this.
FIG. 1 is an internal block diagram of a digital camera according to the present invention. The light from the objective lens (5) is focused on the CCD (50). The CCD (50) converts the received optical signal into an image signal, which is an electrical signal, and outputs the image signal. After the noise is reduced by the CDS (correlated double sampling) circuit (51), the AGC (automatic gain) The level is adjusted by the control (52) circuit. The image signal is converted into a digital signal by the A / D converter (53), and after the black level and white balance are adjusted by the digital signal processing unit (54), the image signal is input to the CPU (1) which is a control means. . The CPU (1) is connected to the memory card (9), and the image signal is compressed by the compression / decompression circuit (55) and stored in the memory card (9). The image in the memory card (9) is decompressed by the compression / decompression circuit (55) and displayed on the display (2) which is a liquid crystal panel.

  The CPU (1) is connected to a ROM (10) in which image processing software is stored and a face area determination means (3). The face area judging means (3) has a configuration from the image storage section (80) to the nostril detection section (83) shown in FIG. Identify. The face area judging means (3) is connected to the column judging means (30), and the row judging means (30) judges which column the face area belongs to before and after, and calculates the average area of the face areas belonging to the same column. Ask. The column determining means (30) is connected to the calculating means (31), and the calculating means (31) obtains how many times the image in the last column is enlarged and feeds it back to the CPU (1). Of course, the image in the foremost row may be reduced, but the case where the image in the last row is enlarged will be described below for convenience of explanation.

FIG. 2 is a flowchart showing an outline of image processing of the digital camera of this example. The photographed image is digitized and temporarily stored in the image development memory (56) (S1-S3). The face area judging means (3) specifies the area of each face area and the coordinates of the upper left corner (S4). Next, a column to which each face region belongs is obtained, an average value of the sizes of the face regions belonging to the column is obtained, and a correction coefficient that is a magnification for enlarging the face region in each column is obtained from the average value for each column. Obtain (S5). Based on the correction coefficient, the face area of each column is enlarged (S6), the image is compressed (S7), and stored in the memory card (9) (S8).
In order to reproduce and process an image that has already been shot and stored in the memory card (9), the procedure from steps S3 to S8 is performed after the image is reproduced.

FIG. 3 is a digital image of a group photo to be processed, and the dotted line shown around each face is the area of each face obtained by the face area determination means (3). The digital image has an origin (0, 0) at the left corner, and the X axis extends rightward from the origin and the Y axis extends downward. The coordinates of the right corner of the image are (X, Y). FIG. 4 is a flowchart showing in detail the operations from step S4 to S6 in the flowchart of FIG.
First, in STEP 1, "face recognition" is started, that is, the face area determination means (3) specifies the area of each face area and the position of the face area including the coordinates of the upper left corner. Next, in STEP 2, the face area determination means (3) obtains the area of each face area, specifically, the area of the area indicated by the dotted rectangle in FIG.
Next, the obtained area and position of the face area are sent to the column determining means (30), and in STEP 3, it is determined which column each face area belongs to. For this, first, it is necessary to specify the face area that should belong to the most back row. Further, as the first step, the face area that becomes the reference among the plurality of face regions that should belong to the most back row is specified. This procedure is shown in the flowchart of FIG.

First, the image shown in FIG. 3 has 15 face regions. Among these, the face area located at the uppermost left is assumed to be the face area 1, followed by the face areas 2... 5 toward the right, and the lowermost face area is taken as the face area 15. It is checked whether this face area 1 belongs to the rearmost row. First, the upper left Y coordinate of the face area 1 is set to Y1, and the number of the face area to be checked is set to N (initially 1) (S10).
It is determined which of Y1 and the upper left Y coordinate of the face area 2 is larger (S12). If the upper left Y coordinate of the face area 1 is smaller than the upper left Y coordinate of the face area 2 as shown in FIG. 6A, then 1 is added to N (S14), and the face area 3 is checked.
As shown in FIG. 6B, if the upper left Y coordinate of the face area 1 is larger than the upper left Y coordinate of the face area 2, the upper left Y coordinate of the face area 2 is set to Y1 (S13), and the subsequent processing proceeds. If all the face areas are checked (S11), the process is terminated, and a reference face area is specified from among a plurality of face areas that should belong to the rearmost row. In the following description, it is assumed that the face region 1 has the smallest Y coordinate, that is, the face region serving as the reference of the last row.

Next, in STEP 3 of FIG. 4, a face area that should be included in the same column as the face area 1 is detected. As described above, the Y coordinate of the upper left corner of each face area is known. As shown in FIG. 7, if the Y coordinate of the upper left corner K2 of the face area 2 is within half H / 2 of the vertical width H of the face area 1 from the Y coordinate of the upper left corner K1 of the face area 1, The face area 2 is a face area belonging to the same column as the face area 1, otherwise the face area 2 is a face area belonging to the front row than the face area 1. Note that the value of H / 2 can be changed as appropriate, and may be, for example, H / 3.
When there is no face area belonging to the same column as face area 1, a face area having the smallest Y coordinate is obtained from the face areas excluding these face areas, and a face area that should be included in the same column is detected in the same manner. .

Finally, as shown in FIG. 8, five face regions belonging to the last column, five face regions belonging to the center column, and five face regions belonging to the front row are obtained. As shown in STEP 4 of FIG. 4, the column determining means (30) obtains the average value Ave (Size R1)... Ave (Size Rn) of the face area areas belonging to each column. Instead of the average value, an intermediate value of the face area area belonging to each column may be used.
Here, the column n is a value indicating the number of columns counted from the last column. In FIG. The column determination means (30) also includes SumR1... SumRn which is the total number of face regions belonging to each column, and Ave (Dist R1)... Ave (Dist Rn) which is the average value of the distances from the X axis of the face regions belonging to each column. ) Is also obtained and sent to the computing means (31).

As shown in STEP 5 of FIG. 4, the calculation means (31) obtains a correction coefficient from Ave (Dist R1) ... Ave (Dist Rn) and Ave (Size R1) ... Ave (Size Rn). The correction coefficient is obtained by the equation shown in FIG.
By multiplying the correction coefficient by the distance from the X axis of the face area, the magnification for enlarging each face area can be obtained.
Specifically, in FIG. 8, when the size of the face region in the last row is matched with the size of the face region in the front row, n = 3, so Ave (Size R3) is Ave (Size). 9 is larger than R1), and Ave (Dist R3) is larger than Ave (Dist R1). Therefore, the correction coefficient in FIG. By multiplying this correction coefficient by the distance Ave (Dist R1), the magnification for enlarging the face area in the last row is obtained.
The calculation means (31) transmits this magnification to the CPU (1), and the CPU (1) automatically enlarges the image in the image development memory (56) based on this magnification. The digital image enlargement method is a method of forming interpolated pixels between pixels as in the conventional method. In the above description, the face area in the last row has been described as being enlarged, but the face area in the front row may be reduced.
By the method of this example, the size of the upper end or lower end of the digital image is automatically corrected. As a result, a natural image can be created even though a part of the image is enlarged / reduced.
The size of the face region in the last row and the size of the face region in the front row may be obtained, and the size of the face region in the intermediate row existing between the last row and the front row may be corrected from the result. That is, in FIG. 8, the size of the face area in the second row may be automatically corrected from the size of the face area in the first row and / or the size of the face area in the third row. .

(Second embodiment)
As shown in FIG. 10, when a group photo is taken at the vertical position of the camera, the face of the person (7) appears sideways in the actual image as shown in FIG. In FIG. 11, when the face area size in the last row and the face area size in the front row are compared, the difference in the left-right direction is larger than in the up-down direction.
That is, in the image of FIG. 11, the face size is smaller due to the tilt phenomenon when proceeding to the right from the origin of the image. Therefore, it is reasonable to correct in the horizontal direction rather than correcting the size in the vertical direction. It is.
In this example, the face area determining means (3) obtains the vertical and horizontal widths of the face area from the face area area belonging to each column. The calculation means (31) obtains a ratio obtained by dividing the width of the face area in the last row by the width of the face area in the front row not only in the vertical direction but also in the horizontal direction, and compares the ratio. The face area in the last row is enlarged along the smaller direction.

Specifically, in the image shown in FIG. 11, the face area determination means (3) first obtains the widths H1 and H2 of the face areas belonging to each column along the vertical direction, and the calculation means (31) Then, a ratio obtained by dividing the width H2 of the face area in the last row by the width H1 of the face area in the front row is obtained. In the image shown in FIG. 11, the width H2 of the face area in the last row is substantially equal to the width H1 of the face area in the front row, and the ratio is a value close to 1.
Next, the face area determination means (3) obtains the horizontal widths HY1 and HY2 of the face areas belonging to each column along the left-right direction, and the calculation means (31) calculates the horizontal width HY2 of the face area in the last row in the front row. The ratio divided by the width HY1 of the face area is obtained. In the image shown in FIG. 11, the horizontal width HY2 of the face area in the last row is clearly smaller than the horizontal width HY1 of the face area in the front row, so the ratio is a value smaller than 1. That is, the ratio between the face area in the last row and the face area in the front row is smaller in the left-right direction, in other words, the face area has a larger difference in the left-right direction. From this result, the CPU (1) enlarges the area of the face area of the last row in the image development memory (56) in the horizontal direction, that is, in the horizontal direction. Thereby, a more natural correction image can be created.

(Third embodiment)
As described above, the face area is recognized from the binarized image. However, the recognition result of the face area is easily influenced by the hairstyle and the face angle. Therefore, there is a possibility that a person who is determined to have a large face area is in the last row or a person who is determined to have a small face area is in the front row. In this case, as described above, when the face area area is corrected for each column, the face area determined to be large may be further increased, or conversely, the face area determined to be small may be further decreased. In order to avoid this fear, the following method can be considered.
First, as shown in FIG. 12, the screen (6) is divided into a plurality of small screens (60) arranged vertically and horizontally. In FIG. 12, four horizontal small screens (60) and (60) are arranged in three upper and lower stages, and each small screen (60) is numbered from 1 to 12 from the upper left to the lower right. It is not limited to. In the face area determination means (3), the number of each small screen (60), the X and Y coordinates of the upper left corner, the vertical and horizontal widths are stored in advance. The operation of this example will be described below with reference to the flowchart of FIG.

  The column determining means (30) first detects in which small screen (60) the face area detected by the face area determining means (3) is included (S20). Next, the face area information existing in the small screen (60) is totaled (S21). This information includes the number of face areas existing in the small screen (60), the vertical and horizontal widths of the face area, and the inclination angle of the face area as will be described later. Next, an average value of the vertical and horizontal widths and angles of the face area is obtained for each small screen (60) (S22), and the vertical width and angle values are calculated for each row (ie, vertical direction) of the small screen (60). Aggregate (S23). The calculation means (31) obtains a coefficient to be corrected in the vertical direction from the vertical width information (S24) and sends it to the CPU (1). The face area judging means (3) also sums up the width value for each column (ie, horizontal direction) of the small screen (60) (S25), and the calculating means (31) corrects the horizontal width information from the horizontal width information. A coefficient to be obtained is obtained (S26) and sent to the CPU (1). Thus, the problem that the digital image is enlarged / reduced for each small screen (60) and the face area determined to be larger, for example, is further enlarged is solved.

(Fourth embodiment)
As shown in FIG. 14, when a group photo in which all persons (7) are standing straight is taken by tilting the camera, the image actually stored in the memory card (9) is as shown in FIG. The image is inclined to In this tilted image, as shown in STEP 3 in FIG. 4, when it is determined which column each face region belongs to, the face region that should belong to the same column is not regarded as being in the same column. As a result, there may be a problem that the enlargement magnification of the face area at the left end of the screen and the face area at the right end of the screen that should belong to the same column is different.
Therefore, since the face area determination means (3) recognizes the coordinates of the left corner of each face area, the applicant has corrected the inclination of the image from the coordinate data if the captured image is inclined. The idea was to determine which column each face region belongs to.

FIG. 16 is an enlarged view showing a state in which each face area is inclined. Since the coordinates of the left corners K1, K2, and K3 of each of the face areas 1, 2, and 3 are known, the face area determining means (3) obtains an inclination angle from the coordinates of K1 and K2, and subsequently, K2 and K3. Find the tilt angle from the coordinates. If these two inclination angles are Φ shown in FIG. 16 and are substantially the same, it is assumed that the face regions 1, 2, and 3 are inclined in the same direction by an angle Φ. A signal indicating that the inclination should be corrected is issued. The CPU (1) corrects the inclination of the image in the image development memory (56) by a known matrix transformation. The inclination angle may be obtained from a regression line linearly approximated from the coordinates of K1, K2, and K3 by the least square method. The calculation of the angle Φ may be performed by the column determining means (30) instead of the face area determining means (3).
However, as shown in FIG. 17, when the face areas 1, 2, and 3 are inclined in different directions from the coordinates K1, K2, and K3 of the face areas 1, 2, and 3, the face area determining means (3) If it is determined, the tilt correction of the image is not performed. This is because if the inclination is corrected in such a case, the image becomes more uncomfortable.

  In the above example, a group photo of a person is exemplified as the subject image, but the group photo is not limited to a human face as long as it is a group photo that causes a tilt phenomenon. Also, a device that displays a digital image is not limited to a digital camera.

  The above description of the embodiments is for explaining the present invention, and should not be construed as limiting the invention described in the claims or reducing the scope thereof. In addition, the configuration of each part of the present invention is not limited to the above embodiment, and various modifications can be made within the technical scope described in the claims.

It is an internal block diagram of a digital camera. It is a flowchart which shows the outline of the image processing of a digital camera. It is a front view of a group photo. 3 is a flowchart showing in detail an operation from steps S4 to S6 in the flowchart of FIG. It is a flowchart which shows the procedure which specifies the face area used as a reference | standard among the several face area which should belong to the back row most. (a), (b) is a figure which shows the arrangement | sequence relationship of a face area | region. It is a figure which shows the arrangement | sequence relationship of a face area | region. It is a front view which shows the last row | line and the front row | line. It is a formula for obtaining a correction coefficient. It is a front view of another group photo. It is a figure which shows an actual image. It is a front view which shows arrangement | positioning of a small screen. It is a flowchart which shows the procedure which expands / reduces a digital image for every small screen. It is a front view of another group photo. It is a figure which shows the image imaged by tilting the camera. It is a figure which shows the arrangement | sequence relationship of a face area | region. It is a figure which shows the arrangement | sequence relationship of a face area | region. It is a figure which shows the structure which obtains the conventional binarized image. It is a figure which shows a binarized image. It is a figure which shows a pupil dictionary and a non-pupil dictionary. It is a figure of the state which a human image | photographs the image located in order. It is a figure which shows the image which produced the tilt phenomenon.

Explanation of symbols

(1) CPU
(3) Face area determination means
(30) Column judgment means
(31) Calculation means
(60) Small screen

Claims (7)

A first determination means for determining a region of each subject element from a plurality of digital images of the subject arranged before and after the row of subject elements arranged side by side;
Second determination means for detecting subject element areas to be included in the same column from the positions of the subject element areas;
Means for obtaining an average value or an intermediate size of the subject element regions belonging to each column;
Calculation means for setting a magnification for enlarging / reducing the subject element area of each column having a different size based on an average value or an intermediate size of the subject element areas of each column obtained by the calculation means;
A digital image display device provided with means for enlarging / reducing the subject element area of each column based on the magnification. The digital image display device according to claim 1, wherein the subject element is a human face, and the digital image of the subject is a group photo of the human. The digital image display device according to claim 1, wherein a magnification for enlarging / reducing the subject element area is obtained for each of the vertical and horizontal directions. The first or second determination means calculates the angle at which the subject element region is tilted from the coordinates of the subject element region, and issues a signal indicating that the image should be tilt corrected by the angle. The digital image display device described. A first determination means for determining a region of each subject element from a plurality of digital images of the subject arranged before and after the row of subject elements arranged side by side;
A second determination unit that divides a screen on which a digital image is displayed into a plurality of vertical and horizontal small screens, and detects a region of a subject element to be included in each small screen from a position of the subject element region;
Means for obtaining an average value or an intermediate size of subject element areas belonging to each small screen;
Calculating means for setting a magnification for enlarging / reducing the subject element area of each small screen having a different size based on an average value or an intermediate size of the subject element area for each small screen obtained by the calculating means; ,
A digital image display device provided with means for enlarging / reducing the subject element area of each small screen based on the magnification. The digital image display device according to claim 5, wherein the magnification for enlarging / reducing the subject element area is obtained for each of the vertical and horizontal directions. A step of determining a region of each subject element from a digital image of a subject arranged in front and back, a row of subject elements arranged side by side;
Detecting a subject element to be included in the same column from the position of the region of the subject element;
Obtaining an average value or an intermediate size of the area of the subject element belonging to each column;
A step of setting a magnification for enlarging / reducing the subject element area of each column having a different size based on an average value or an intermediate size of the subject element areas of each column obtained by the calculation means;
A digital image display method comprising a step of enlarging / reducing an area of a subject element in each column based on the magnification.

Images