JP2006039666A - Image processing method, image processor and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the accuracy of head-and-tail determination of a photographed image regardless of the photographing environment. <P>SOLUTION: The image processor 1 determines a light distribution condition (normal light, back light, or strobing) from photographed image data (step S2), and determines an extraction condition of image area based on the determined light distribution condition (step S3). According to the determined extraction condition, a blank area or a facial area (if a person is contained in the image data) is extracted from the photographed image data (step S4), and the head and tail of the photographed image data is determined based on the extracted image area (step S5). <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing method, an image processing apparatus, and an image processing program for performing image processing on captured image data.

  In recent years, digital still cameras (including toys, cameras incorporated in devices such as mobile phones and laptop computers, general-purpose applications for general users, and high-functional professional cameras. Hereinafter, abbreviated as DSC). Widely used, as with color photographic film, it is output as a hard copy image or displayed on an output medium such as a CRT (Cathode Ray Tube).

  In addition, a technique is widely used in which an image formed on a color photographic film is photoelectrically read by a CCD (Charge Coupled Device) sensor or the like and converted into an image signal. Such an image signal is subjected to various kinds of image processing represented by negative / positive inversion, brightness adjustment, color balance adjustment, grain removal, and sharpness enhancement, and then CD-R (CD-Recordable), CD-RW ( CD-ReWritable), floppy (registered trademark) disk, memory card, or other recording media, distributed via the Internet, and output as a hard copy on silver halide photographic paper, inkjet printers, thermal printers, etc. It can be viewed on an output medium such as a liquid crystal display or plasma display.

  When viewing images or processing images, the top and bottom of the shooting scene are not displayed correctly (the top and bottom are reversed), because the uplifting feeling during viewing is impaired, and the processing efficiency of image processing decreases (rotation processing) This is not preferable). Many techniques for automatically determining the top and bottom of a captured image in response to such a background have been disclosed. For example, Patent Document 1 refers to top and bottom information recorded in tag information such as magnetic information of an APS (Advanced Photo System) film and Exif (Exchangeable Image File Format) of digital image data at the time of image display or image processing. However, a technique is disclosed in which rotation processing is applied if the top and bottom information of the tag information and the top and bottom actually displayed are reversed.

Patent Document 2 discloses a technique for automatically determining the top and bottom by analyzing a shooting scene of a shot image. This patent document 2 uses a characteristic that has a gradation from a dark color to a white color from the zenith to the horizon, and first, the sky region is extracted from the photographed image, and the extracted region The top and bottom direction is determined based on the color gradient.
Japanese Patent Laid-Open No. 10-171035 Japanese Patent Laid-Open No. 2001-195591

  However, the technique disclosed in Patent Document 1 is not necessarily the one in which the top-and-bottom information is recorded in the tag information, but is incomplete, and the recorded information is also the top and bottom of the actual shooting scene. The accuracy of the top and bottom judgment could not be said to be sufficient because it was not compatible. Also, in actual shooting scenes, the sky color varies, and in images shot with a digital camera with a low dynamic range, the gradient of the sky area or monochrome background area may be low, depending on the shooting conditions. Since there are many cases (for example, the background is white in the backlight state), even if the technique of Patent Document 2 is applied, it cannot be said that the accuracy of the top-and-bottom determination is sufficient.

  The subject of this invention is improving the precision of the top-and-bottom determination of a picked-up image irrespective of a photography environment.

  In order to solve the above-described problem, the invention according to claim 1 is a determination step of determining a light distribution condition at the time of photographing from photographed image data, and a predetermined image region extraction condition from the determined light distribution condition. A determination step of determining at least one image area from the captured image data according to the determined extraction condition, and determining a top and bottom of the captured image data based on the extracted image area And a determination step.

  The invention according to claim 2 is a determination step of determining a light distribution condition at the time of shooting from captured image data, a determination step of determining an extraction condition of a predetermined image region from the determined light distribution condition, In accordance with the determined extraction condition, an extraction step of extracting at least one image region from the captured image data, a calculation step of calculating a predetermined feature amount from the extracted image region, the calculated feature amount, and And a determination step of determining the top of the photographed image data using a neural network using a feature amount calculated based on the determined light distribution condition as an input signal.

  Here, the “predetermined feature amount” has two types, that is, the position of the extracted image region in the captured image data and the accuracy of the extracted image region (for example, the accuracy of being empty). Examples of the position include the position of the center of gravity and the direction of the gradient of the color value, and examples of the accuracy include statistical values and gradient values regarding the color, shape, contrast, texture uniformity, and the like. In the present invention, it is preferable that a feature quantity representing a position is included as the predetermined feature quantity. Further, it is more preferable that a feature amount representing accuracy is included with the position.

  According to a third aspect of the present invention, in the image processing method according to the first or second aspect, the light distribution condition at the time of photographing includes a backlight state and / or a flash light utilization state.

  According to a fourth aspect of the present invention, in the image processing method according to any one of the first to third aspects, the image region extracted in the extraction step includes at least a background region of a main subject. It is a feature.

  The “background region” in the present invention is preferably an image region in which a subject that is invariant with certainty in the positional relationship between the top and bottom is photographed. Specifically, it is preferably a subject (for example, sky or cloud) that has a constant hue and has no texture or strong edges and has a flat or gentle gradient. It is also preferable to add a subject (for example, a portion where lawn, ground, and leaves are dense) composed of a certain periodic texture.

  According to a fifth aspect of the present invention, in the image processing method according to the fourth aspect, the image region extracted in the extraction step includes at least a human face region.

  The invention according to claim 6 is a discriminating means for discriminating a light distribution condition at the time of photographing from photographed image data; a determining means for determining an extraction condition for a predetermined image region from the discriminated light distribution condition; An extraction unit that extracts at least one image area from the captured image data in accordance with the determined extraction condition; and a determination unit that determines the top of the captured image data based on the extracted image area. It is characterized by.

  The invention according to claim 7 is a discriminating means for discriminating a light distribution condition at the time of photographing from photographed image data; a determining means for determining an extraction condition for a predetermined image region from the discriminated light distribution condition; In accordance with the determined extraction condition, an extraction unit that extracts at least one image region from the captured image data, a calculation unit that calculates a predetermined feature amount from the extracted image region, the calculated feature amount, and And determining means for determining the top and bottom of the photographed image data using a neural network with the feature amount calculated based on the determined light distribution condition as an input signal.

  According to an eighth aspect of the present invention, in the image processing device according to the sixth or seventh aspect, the light distribution condition at the time of photographing includes a backlight state and / or a flash light utilization state.

  The invention according to claim 9 is the image processing apparatus according to any one of claims 6 to 8, wherein the extraction unit extracts at least a background area of a main subject from the photographed image data. It is said.

  According to a tenth aspect of the present invention, in the image processing apparatus according to the ninth aspect, the extracting means extracts at least a human face area from the photographed image data.

  According to an eleventh aspect of the present invention, a computer executing image processing has a discrimination function for discriminating a light distribution condition at the time of photographing from photographed image data, and extraction of a predetermined image area from the determined light distribution condition. A determination function for determining a condition; an extraction function for extracting at least one image area from the captured image data according to the determined extraction condition; and determining the top and bottom of the captured image data based on the extracted image area And a determination function to perform.

  According to a twelfth aspect of the present invention, a computer executing image processing has a discrimination function for discriminating a light distribution condition at the time of photographing from photographed image data, and extraction of a predetermined image area from the discriminated light distribution condition. A determination function for determining a condition, an extraction function for extracting at least one image area from the captured image data in accordance with the determined extraction condition, and a calculation function for calculating a predetermined feature amount from the extracted image area; And a determination function for determining the top of the photographed image data using a neural network using the calculated feature value and the feature value calculated based on the determined light distribution condition as an input signal.

  According to a thirteenth aspect of the present invention, in the image processing program according to the eleventh or twelfth aspect, the light distribution condition at the time of photographing includes a backlight state and / or a flash light utilization state.

  According to a fourteenth aspect of the present invention, in the image processing program according to any one of the eleventh to thirteenth aspects, at least the background area of the main subject is extracted from the captured image data when the extraction function is realized. It is characterized by letting.

  According to a fifteenth aspect of the present invention, in the image processing program according to the fourteenth aspect, when realizing the extraction function, at least a human face region is extracted from the photographed image data.

  According to the present invention, the image area determined according to the light distribution condition is extracted from the captured image data, and the top / bottom determination is performed based on the extracted image area. In addition, it is possible to improve the determination accuracy of the top / bottom determination.

  In addition, the image area determined according to the light distribution condition is extracted from the photographed image data, and the feature amount of the extracted image area and the neural network using the feature quantity of the light distribution condition as an input signal are used to Therefore, it is possible to improve the determination accuracy of the top / bottom determination in any shooting environment, and to perform the top / bottom determination reflecting the user's shooting situation.

  Furthermore, as the light distribution conditions at the time of shooting, the frequency of backlighting and flashing (strobe) is high, and the influence on the color of the subject is great. By extracting the image area, the accuracy of the top and bottom determination can be further improved.

  In particular, if the top / bottom determination is performed based on the background region, the determination accuracy can be further improved. When the photographed image data includes a person, the determination accuracy can be further improved by performing the top / bottom determination based on the person's face area as well as the background area.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
First, an external configuration and an internal configuration of an image processing apparatus 1 to which the present invention is applied will be described as a configuration common to Embodiments 1 and 2 of the present invention with reference to FIGS.

[Embodiment 1]
<Appearance Configuration of Image Processing Apparatus 1>
FIG. 1 is a perspective view showing an external configuration of an image processing apparatus 1 according to Embodiments 1 and 2 of the present invention. As shown in FIG. 1, the image processing apparatus 1 is provided with a magazine loading unit 3 for loading a photosensitive material on one side surface of a housing 2. An exposure processing unit 4 that exposes the photosensitive material and a print creation unit 5 that develops the exposed photosensitive material, dries it, and creates a print are provided inside the housing 2. The created print is discharged to a tray 6 provided on the other side of the housing 2.

  In addition, a CRT (Cathode Ray Tube) 8 serving as a display device, a film scanner unit 9 serving as a device for reading a transparent document, a reflective document input device 10, and an operation unit 11 are provided on the upper portion of the housing 2. Further, the housing 2 includes an image reading unit 14 that can read image information recorded on various digital recording media, and an image writing unit 15 that can write (output) image signals to various digital recording media. Yes. In addition, a control unit 7 that centrally controls these units is provided inside the housing 2.

  The image reading unit 14 includes a PC card adapter 14a and a floppy (registered trademark) disk adapter 14b, and a PC card 13a and a floppy (registered trademark) disk 13b can be inserted therein. The PC card 13a has a memory in which a plurality of image data captured by a digital camera is recorded. For example, a plurality of image data captured by a digital camera are recorded on the floppy (registered trademark) disk 13b. In addition, examples of the recording medium having image data include a multimedia card, a memory stick, MD (Mini Disc) data, and a CD-ROM (Compact Disc Read Only Memory).

  The image writing unit 15 includes a floppy (registered trademark) disk adapter 15a, an MO adapter 15b, and an optical disk adapter 15c, and a floppy (registered trademark) disk 16a, MO 16b, and an optical disk 16c can be inserted into the image writing unit 15, respectively. Image information can be written to an image recording medium. Examples of the optical disk 16c include a CD-R (Compact Disc-Recordable), a DVD-R (Digital Versatile Disk-Recordable), and the like.

  In FIG. 1, the operation unit 11, the CRT 8, the film scanner unit 9, the reflection original input device 10, and the image reading unit 14 are integrally provided in the housing 2. One or more may be provided separately.

  In the image processing apparatus 1 shown in FIG. 1, an example is illustrated in which a photosensitive material is exposed and developed to create a print, but the print creation method is not limited to this, and for example, an inkjet method, an electronic A method such as a photographic method, a thermal method, or a sublimation method may be used.

<Internal Configuration of Image Processing Apparatus 1>
FIG. 2 shows a main part configuration inside the image processing apparatus 1. As shown in FIG. 2, the image processing apparatus 1 includes a control unit 7, an exposure processing unit 4, a print creation unit 5, a CRT 8, a film scanner unit 9, a reflective original input device 10, an operation unit 11, an image reading unit 14, and an image. The writing unit 15 includes a communication unit (input) 32, a communication unit (output) 33, and a data storage unit 71.

  The control unit 7 includes a microcomputer, and various control programs such as an image processing program stored in a storage unit (not shown) such as a ROM (Read Only Memory), and a CPU (Central Processing Unit) (not shown). The operation of each part constituting the image processing apparatus 1 is comprehensively controlled in cooperation with the above.

  The control unit 7 includes an image processing unit 70, and reads image data acquired by the film scanner unit 9 or the reflection original input device 10 from the image reading unit 14 based on an input signal (command information) from the operation unit 11. The processed image data and image data input from an external device via the communication means (input) 32 are subjected to image processing to generate output image data, which are output to the exposure processing unit 4. Further, the image processing unit 70 performs a conversion process corresponding to the output form on the image processed image data, and outputs the converted image data. Output destinations of the image processing unit 70 include the CRT 8, the image writing unit 15, the communication means (output) 33, and the like.

  The exposure processing unit 4 exposes an image to the photosensitive material and outputs the photosensitive material to the print creating unit 5. The print creating unit 5 develops and exposes the exposed photosensitive material to create prints P1, P2, and P3. The print P1 is a service size, high-definition size, panoramic size print, the print P2 is an A4 size print, and the print P3 is a business card size print.

  The film scanner unit 9 reads an image recorded on a transparent original such as a developed negative film N or a reversal film taken by an analog camera.

  The reflection original input device 10 reads an image formed on a print P (photo print, document, various printed materials) by a flat bed scanner (not shown).

  The operation unit 11 includes information input means 12. The information input unit 12 is configured by a touch panel, for example, and outputs a pressing signal of the information input unit 12 to the control unit 7 as an input signal. Note that the operation unit 11 may be configured to include a keyboard, a mouse, and the like. The CRT 8 displays image data and the like according to the display control signal input from the control unit 7.

  The image reading unit 14 includes, as the image transfer means 30, a PC card adapter 14a, a floppy (registered trademark) disk adapter 14b, and the like, and the PC card 13a inserted into the PC card adapter 14a or a floppy (registered trademark). ) Image data recorded on the floppy (registered trademark) disk 13b inserted into the disk adapter 14b is read out and transferred to the control unit 7. For example, a PC card reader or a PC card slot is used as the PC card adapter 14a.

  The image writing unit 15 includes, as the image conveying unit 31, a floppy (registered trademark) disk adapter 15a, an MO adapter 15b, and an optical disk adapter 15c. In accordance with a write signal input from the control unit 7, the image writing unit 15 includes a floppy (registered trademark) disk 16a inserted into the floppy (registered trademark) disk adapter 15a, an MO 16b inserted into the MO adapter 15b, The generated image data is written to the optical disk 16c inserted into the optical disk adapter 15c.

  The communication means (input) 32 receives image data representing a captured image and a print command signal from another computer in the facility where the image processing apparatus 1 is installed, or a remote computer via the Internet or the like.

  The communication means (output) 33 sends image data representing the captured image after image processing and order information to another computer in the facility where the image processing apparatus 1 is installed, or a remote computer via the Internet or the like. Send to.

  The data storage means 71 stores and sequentially stores image data and corresponding order information (information about how many prints are to be created from images of which frames, information on print sizes, etc.).

<Configuration of Image Processing Unit 70>
FIG. 3 shows a main part configuration inside the image processing unit 70. As shown in FIG. 3, the image processing unit 70 includes a film scan data processing unit 701, a reflection original scan data processing unit 702, an image data format decoding processing unit 703, an image adjustment processing unit 704, a CRT specific processing unit 705, a printer specific A processing unit (1) 706, a printer specific processing unit (2) 707, and an image data format creation processing unit 708 are configured.

  A film scan data processing unit 701 performs proofing operations specific to the film scanner unit 9 on the image data input from the film scanner unit 9, negative reversal in the case of a negative document, dust flaw removal, gray balance adjustment, contrast adjustment, granular noise Removal, sharpening, and the like are performed, and the result is output to the image adjustment processing unit 704. The film scan data processing unit 701 also includes information on film size, negative / positive type, ISO (International Organization for Standardization) sensitivity recorded on the film optically or magnetically, manufacturer name, information on main subject, and shooting conditions ( For example, the description information content of APS (Advanced Photo System) is also output to the image adjustment processing unit 704.

  The reflection document scan data processing unit 702 performs a calibration operation unique to the reflection document input device 10, negative / positive reversal in the case of a negative document, dust flaw removal, gray balance adjustment, and contrast adjustment for the image data input from the reflection document input device 10. , Noise removal, sharpening enhancement, and the like, and output to the image adjustment processing unit 704.

  The image data format decoding processing unit 703 performs compression code restoration, color data expression method conversion, and the like as necessary according to the data format of the image data input from the image transfer means 30 or the communication means (input) 32. Then, the data is converted into a data format suitable for calculation in the image processing unit 70 and output to the image adjustment processing unit 704.

  The image adjustment processing unit 704 performs various processes on the images received from the film scanner unit 9, the reflective original input device 10, the image transfer unit 30, and the communication unit (input) 32 based on the command from the operation unit 11 or the control unit 7. Performs image processing, and outputs processed image data to the CRT specific processing unit 705, printer specific processing unit (1) 706, printer specific processing unit (2) 707, image data format creation processing unit 708, and data storage means 71. . Image processing executed by the image adjustment processing unit 704 according to the first embodiment will be described in detail later with reference to FIGS.

  The CRT specific processing unit 705 performs processing such as changing the number of pixels and color matching on the image data input from the image adjustment processing unit 704 as necessary, and combines the information with information that needs to be displayed, such as control information. Image data is output to the CRT 8.

  A printer-specific processing unit (1) 706 performs printer-specific calibration processing, color matching, pixel number change, and the like on the image data input from the image adjustment processing unit 704 as necessary. Output.

  When an external printer 34 such as a large-format ink jet printer is connected to the image processing apparatus 1, a printer specific processing unit (2) 707 is provided for each connected printer. The printer-specific processing unit (2) 707 performs appropriate printer-specific calibration processing, color matching, pixel number change, and the like on the image data input from the image adjustment processing unit 704.

  The image data format creation processing unit 708 applies JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and Exif (Exchangeable Image File Format) to the image data input from the image adjustment processing unit 704 as necessary. ) And the like, and the image is converted to various general-purpose image formats and output to the image transport unit 31 and the communication means (output) 33.

  A film scan data processing unit 701, a reflection original scan data processing unit 702, an image data format decoding processing unit 703, an image adjustment processing unit 704, a CRT specific processing unit 705, a printer specific processing unit (1) 706, a printer specific processing unit. (2) The category 707 and the image data format creation processing unit 708 are provided to help understanding of the function of the image processing unit 70, and need not be realized as a physically independent device. Alternatively, it may be realized as a type of software processing in a single CPU. The image processing apparatus 1 according to the first and second embodiments is not limited to the above-described contents, and may be applied to various forms such as a digital photo printer, a printer driver, and various image processing software plug-ins. Can do.

Next, the operation in the first embodiment will be described.
First, the overall flow of image processing executed by the image adjustment processing unit 704 will be described with reference to the flowchart of FIG.

  First, photographic image data to be image-processed is acquired via the film scanner unit 9, the reflective original input device 10, the image transfer unit 30, the communication unit (input) 32, and the like (step S1), and the acquired photographic image is acquired. A light distribution condition determining process for determining the light distribution condition (light source state) at the time of photographing from the data is performed (step S2). The light distribution conditions at the time of photographing include forward light, backlight, and strobe (flash light use state). The light distribution condition determination process in step S2 will be described in detail later with reference to FIG.

  Next, an extraction condition for a predetermined image area is determined based on the light distribution condition determined in step S2 (step S3), and an image area is extracted from the captured image data according to the determined extraction condition (step S3). S4). Here, the predetermined image area includes an empty area and a face area (when a person is photographed). The process of extracting a sky area from captured image data and the process of extracting a face area from captured image data will be described in detail later with reference to FIG.

  Next, a top / bottom determination process for determining the top / bottom of the captured image data is performed based on the image area extracted in step S4 (step S5), and the main image processing is completed. In step S5, the sky direction (upward direction) of the photographed image data is determined from the position of the sky region with respect to the entire photographed image data. In addition, when a face area is further extracted from the captured image data, the top and bottom can be reliably determined from the positional relationship between the sky area and the face area.

  Next, the light distribution condition determination process shown in step S2 of FIG. 4 will be described with reference to the flowchart of FIG.

  First, the captured image data is divided into predetermined image areas, and an occupancy ratio calculation process is performed to calculate an occupancy ratio indicating the ratio of each divided area to the entire captured image data (step S6). The occupation rate calculation process will be described in detail later with reference to FIGS.

  Next, based on the occupancy calculated in step S6 and a coefficient set in advance according to the imaging condition, an index (index 1 to 5) that specifies the light distribution condition (quantitatively represents the light source state) is calculated. (Step S7). The index calculation process in step S7 will be described in detail later.

  Next, the light distribution condition of the photographed image data is determined based on the index calculated in step S7 (step S8), and the main light distribution condition determination process ends. A method for determining the light distribution condition will be described in detail later.

  Next, the occupation rate calculation process shown in step S6 of FIG. 5 will be described in detail with reference to the flowchart of FIG.

  First, the RGB values of the photographed image data are converted into the HSV color system (step S10). FIG. 7 shows an example of a conversion program (HSV conversion program) that obtains a hue value, a saturation value, and a lightness value by converting from RGB to the HSV color system in a program code (c language). In the HSV conversion program shown in FIG. 7, the values of digital image data as input image data are defined as InR, InG, and InB, the calculated hue value is defined as OutH, the scale is defined as 0 to 360, and the saturation The value is OutS, the brightness value is OutV, and the unit is defined as 0 to 255.

  Next, the photographed image data is divided into regions composed of combinations of predetermined brightness and hue, and a two-dimensional histogram is created by calculating the cumulative number of pixels for each divided region (step S11). Hereinafter, the area division of the captured image data will be described in detail.

The lightness value (V) is 0-25 (v1), 26-50 (v2), 51-84 (v3), 85-169 (v4), 170-199 (v5), 200-224 (v6) , 225 to 255 (v7). Hue (H) is a skin hue hue area (H1 and H2) with a hue value of 0 to 39, 330 to 359, a green hue area (H3) with a hue value of 40 to 160, and a blue hue area with a hue value of 161 to 250 It is divided into four areas (H4) and a red hue area (H5). Note that the red hue region (H5) is not used in the following calculation because it is known that the contribution to the determination of the light distribution condition is small. The flesh-color hue area is further divided into a flesh-color area (H1) and other areas (H2). Hereinafter, among the flesh color hue regions (H = 0 to 39, 330 to 359), a hue color '(H) that satisfies the following equation (1) is defined as a flesh color region (H1), and a region that does not satisfy the equation (1) is ( H2).
10 <Saturation (S) <175,
Hue '(H) = Hue (H) + 60 (when 0 ≤ Hue (H) <300),
Hue '(H) = Hue (H)-300 (when 300 ≤ Hue (H) <360),
Luminance Y = InR × 0.30 + InG × 0.59 + InB × 0.11
As
Hue '(H) / Luminance (Y) <3.0 × (Saturation (S) / 255) +0.7 (1)
Therefore, the number of divided areas of the captured image data is 4 × 7 = 28. In addition, it is also possible to use the brightness (V) in the formula (1).

When the two-dimensional histogram is created, a first occupancy ratio indicating the ratio of the cumulative number of pixels calculated for each divided region to the total number of pixels (the entire captured image) is calculated (step S12), and the main occupancy ratio calculation is performed. The process ends. Assuming that the first occupancy ratio calculated in the divided area composed of the combination of the lightness area vi and the hue area Hj is Rij, the first occupancy ratio in each divided area is expressed as shown in Table 1.

Next, a method for calculating the index 1 and the index 2 will be described.
Table 2 shows the first coefficient necessary for calculating the index 1 that quantitatively indicates the accuracy of strobe shooting obtained by discriminant analysis, that is, the lightness state of the face area at the time of strobe shooting. Show. The coefficient of each divided area shown in Table 2 is a weighting coefficient by which the first occupation ratio Rij of each divided area shown in Table 1 is multiplied.

  FIG. 8 shows a lightness (v) -hue (H) plane. According to Table 2, a positive (+) coefficient is used for the first occupancy calculated from the region (r1) distributed in the skin color hue region of high brightness in FIG. 8, and the other hue is blue. A negative (−) coefficient is used for the first occupancy calculated from the hue region (r2). FIG. 10 shows a curve (coefficient curve) in which the first coefficient in the skin color area (H1) and the first coefficient in the other areas (green hue area (H3)) continuously change over the entire brightness. It is shown. According to Table 2 and FIG. 10, in the region of high brightness (V = 170 to 224), the sign of the first coefficient in the skin color region (H1) is positive (+), and other regions (for example, the green hue region) The sign of the first coefficient in (H3)) is negative (-), and it can be seen that the signs of the two are different.

従って、H1〜H4領域の和は、下記の式(2-1)〜式(2-4)のように表される。
H1領域の和＝R11×(-44.0)＋R21×(-16.0)＋（中略）...＋R71×(-11.3) (2-1)
H2領域の和＝R12×0.0＋R22×8.6＋（中略）... ＋R72×(-11.1) (2-2)
H3領域の和＝R13×0.0＋R23×(-6.3)＋（中略）...＋R73×(-10.0) (2-3)
H4領域の和＝R14×0.0＋R24×(-1.8)＋（中略）...＋R74×(-14.6) (2-4) When the first coefficient in the lightness region vi and the hue region Hj is Cij, the sum of the Hk regions for calculating the index 1 is defined as in Expression (2).

Accordingly, the sum of the H1 to H4 regions is expressed by the following formulas (2-1) to (2-4).
H1 area sum = R11 x (-44.0) + R21 x (-16.0) + (omitted) ... + R71 x (-11.3) (2-1)
Sum of H2 area = R12 x 0.0 + R22 x 8.6 + (omitted) ... + R72 x (-11.1) (2-2)
Sum of H3 area = R13 x 0.0 + R23 x (-6.3) + (omitted) ... + R73 x (-10.0) (2-3)
Sum of H4 area = R14 x 0.0 + R24 x (-1.8) + (omitted) ... + R74 x (-14.6) (2-4)

The index 1 is defined as Expression (3) using the sum of the H1 to H4 regions shown in Expressions (2-1) to (2-4).
Index 1 = sum of H1 area + sum of H2 area + sum of H3 area + sum of H4 area + 4.424 (3)

Table 3 shows, for each divided region, the second coefficient necessary for calculating the index 2 that quantitatively indicates the accuracy of backlight photographing obtained by discriminant analysis, that is, the brightness state of the face region at the time of backlight photographing. Show. The coefficient of each divided area shown in Table 3 is a weighting coefficient that is multiplied by the first occupation ratio Rij of each divided area shown in Table 1.

  FIG. 9 shows a lightness (v) -hue (H) plane. According to Table 3, a negative (-) coefficient is used for the occupancy calculated from the region (r4) distributed in the intermediate lightness of the flesh color hue region in FIG. 9, and the low lightness (shadow) region ( A positive (+) coefficient is used for the occupation ratio calculated from r3). FIG. 11 shows the second coefficient in the skin color region (H1) as a curve (coefficient curve) that continuously changes over the entire brightness. According to Table 3 and FIG. 11, the sign of the second coefficient of the intermediate lightness region of the flesh color hue region having the lightness value of 85 to 169 (v4) is negative (−), and the lightness value is 26 to 84 (v2, It can be seen that the sign of the second coefficient in the low brightness (shadow) region of v3) is positive (+), and the sign of the coefficient in both regions is different.

従って、H1〜H4領域の和は、下記の式(4-1)〜式(4-4)のように表される。
H1領域の和＝R11×(-27.0)＋R21×4.5＋（中略）...＋R71×(-24.0) (4-1)
H2領域の和＝R12×0.0＋R22×4.7＋（中略）... ＋R72×(-8.5) (4-2)
H3領域の和＝R13×0.0＋R23×0.0＋（中略）...＋R73×0.0 (4-3)
H4領域の和＝R14×0.0＋R24×(-5.1)＋（中略）...＋R74×7.2 (4-4) When the second coefficient in the lightness area vi and the hue area Hj is Dij, the sum of the Hk areas for calculating the index 2 is defined as in Expression (4).

Therefore, the sum of the H1 to H4 regions is expressed by the following formulas (4-1) to (4-4).
H1 area sum = R11 x (-27.0) + R21 x 4.5 + (omitted) ... + R71 x (-24.0) (4-1)
Sum of H2 area = R12 x 0.0 + R22 x 4.7 + (omitted) ... + R72 x (-8.5) (4-2)
Sum of H3 area = R13 x 0.0 + R23 x 0.0 + (omitted) ... + R73 x 0.0 (4-3)
H4 area sum = R14 x 0.0 + R24 x (-5.1) + (omitted) ... + R74 x 7.2 (4-4)

The index 2 is defined as in Expression (5) using the sum of the H1 to H4 regions shown in Expressions (4-1) to (4-4).
Index 2 = sum of H1 area + sum of H2 area + sum of H3 area + sum of H4 area + 1.554 (5)
Since the index 1 and the index 2 are calculated based on the brightness of the captured image data and the distribution amount of the hue, it is effective for determining the light distribution condition when the captured image data is a color image.

  Next, the occupation rate calculation process (step S6 in FIG. 5) for calculating the index 3 will be described in detail with reference to the flowchart in FIG.

  First, the RGB values of the photographed image data are converted into the HSV color system (step S20). Next, the photographed image data is divided into regions each composed of a combination of the distance from the outer edge of the photographed image screen and the brightness, and a two-dimensional histogram is created by calculating the cumulative number of pixels for each divided region (step S21). Hereinafter, the area division of the captured image data will be described in detail.

  FIGS. 13A to 13D show four regions n1 to n4 divided according to the distance from the outer edge of the screen of the captured image data. A region n1 shown in FIG. 13A is an outer frame, a region n2 shown in FIG. 13B is a region inside the outer frame, and a region n3 shown in FIG. A region n4 shown in FIG. 13D is an inner region and is a central region of the captured image screen. Further, the lightness is divided into seven regions v1 to v7 as described above. Therefore, when the captured image data is divided into regions composed of combinations of the distance from the outer edge of the captured image screen and the brightness, the number of divided regions is 4 × 7 = 28.

When the two-dimensional histogram is created, a second occupancy ratio indicating the ratio of the cumulative number of pixels calculated for each divided region to the total number of pixels (the entire captured image) is calculated (step S22), and the main occupancy ratio calculation is performed. The process ends. If the second occupancy calculated in the divided area composed of the combination of the brightness area vi and the screen area nj is Qij, the second occupancy in each divided area is expressed as shown in Table 4.

図１４は、画面領域ｎ１〜ｎ４における第３の係数を、明度全体に渡って連続的に変化する曲線（係数曲線）として示したものである。 Next, a method for calculating the index 3 will be described.
Table 5 shows the third coefficient necessary for calculating the index 3 for each divided region. The coefficient of each divided area shown in Table 5 is a weighting coefficient by which the second occupancy Qij of each divided area shown in Table 4 is multiplied, and is obtained by discriminant analysis.

FIG. 14 shows the third coefficient in the screen areas n1 to n4 as a curve (coefficient curve) that continuously changes over the entire brightness.

従って、n1〜n4領域の和は、下記の式(6-1)〜式(6-4)のように表される。
n1領域の和＝Q11×40.1＋Q21×37.0＋（中略）...＋Q71×22.0 (6-1)
n2領域の和＝Q12×(-14.8)＋Q22×(-10.5)＋（中略）...＋Q72×0.0 (6-2)
n3領域の和＝Q13×24.6＋Q23×12.1＋（中略）...＋Q73×10.1 (6-3)
n4領域の和＝Q14×1.5＋Q24×(-32.9)＋（中略）...＋Q74×(-52.2) (6-4) If the third coefficient in the brightness area vi and the screen area nj is Eij, the sum of the nk area (screen area nk) for calculating the index 3 is defined as in Expression (6).

Accordingly, the sum of the n1 to n4 regions is expressed as the following formulas (6-1) to (6-4).
n1 area sum = Q11 x 40.1 + Q21 x 37.0 + (omitted) ... + Q71 x 22.0 (6-1)
Sum of n2 area = Q12 x (-14.8) + Q22 x (-10.5) + (omitted) ... + Q72 x 0.0 (6-2)
n3 area sum = Q13 x 24.6 + Q23 x 12.1 + (omitted) ... + Q73 x 10.1 (6-3)
n4 area sum = Q14 x 1.5 + Q24 x (-32.9) + (omitted) ... + Q74 x (-52.2) (6-4)

The index 3 is defined as in Expression (7) using the sum of the N1 to H4 regions shown in Expressions (6-1) to (6-4).
Index 3 = sum of n1 regions + sum of n2 regions + sum of n3 regions + sum of n4 regions−12.6201 (7)
The index 3 is calculated on the basis of compositional characteristics (distance from the outer edge of the screen of the photographed image data) based on the lightness distribution position of the photographed image data. It is also effective to do.

The index 4 is defined as Expression (8) using the indices 1 and 3, and the index 5 is defined as Expression (9) using the indices 1 to 3.
Index 4 = 0.565 x Index 1 + 0.565 x Index 3 + 0.457 (8)
Indicator 5 = (-0.121) x Indicator 1 + 0.91 x Indicator 2 + 0.113 x Indicator 3-0.072 (9)
Here, the weighting coefficient by which each index is multiplied in Expression (8) and Expression (9) is set in advance according to the shooting conditions.

このように指標４、５の値により配光条件を定量的に判別することができる。 <Determination method of light distribution conditions>
Next, a method for determining light distribution conditions will be described.
In FIG. 15, 60 images were taken under each light distribution condition of forward light, backlight, and strobe, and indexes 4 and 5 were calculated for a total of 180 digital image data, and the indexes 4 and 5 under each light distribution condition were calculated. The values are plotted. According to FIG. 15, when the value of the index 4 is larger than 0.5, there are many strobe scenes, and when the value of the index 4 is 0.5 or less and the value of the index 5 is larger than −0.5, the backlight scene is I understand that there are many. Table 6 shows the determination contents of the light distribution conditions based on the values of the indexes 4 and 5.

Thus, the light distribution condition can be determined quantitatively based on the values of the indices 4 and 5.

  Next, the image region extraction process shown in steps S3 and S4 of FIG. 4 will be described with reference to the flowchart of FIG.

  First, the process of extracting the region 1 (empty) region as an image region will be described with reference to the flowchart in FIG.

  First, the light distribution state is determined from the light distribution condition determined in the light distribution condition determination process (see FIG. 5) (step S30). When the light distribution state is backlight (step S30; backlight), the extraction condition of the region 1 (sky) region corresponding to the backlight is determined (step S31). Next, according to the extraction condition determined in step S31, a white background region is extracted from the captured image data as region 1 (sky) (step S32), and the region 1 (sky) extraction process ends. In the case of backlight, the sky color tends to be whitish and lighter than the actual sky color in the captured image data, so that an area having a flat or gentle gradient of approximately white to light aqua color is extracted. By using the extraction condition, the corresponding region can be extracted with high accuracy.

  When the light distribution state is the follow light (step S30; follow light), the extraction condition of the region 1 (sky) region (extraction of the sky blue (light blue) region) corresponding to the follow light is determined (step S33). Next, in accordance with the extraction condition determined in step S33, a sky blue (light blue) background region is extracted from the captured image data as region 1 (sky) (step S34), and the region 1 (sky) extraction process ends.

  When the light distribution state is strobe (step S30; strobe), the extraction condition for the region 1 (sky) corresponding to the strobe is determined (step S35). Next, according to the extraction condition determined in step S35, a black background region is extracted from the captured image data as region 1 (sky) (step S36), and the region 1 (sky) extraction process ends. In the case of a strobe, since the image is often photographed when the surroundings are dark, an extraction condition may be set such that a hue region corresponding to a low-lightness (dark) sky blue is extracted. However, when extracting an extremely low brightness area assuming nighttime, it is preferable to exclude the designation of the hue from the extraction conditions because the hue value becomes unstable in calculation.

  Next, processing for extracting the region 2 (face) region as the image region will be described with reference to the flowchart of FIG.

  First, the light distribution state is determined from the light distribution condition determined in the light distribution condition determination process (see FIG. 5) (step S40). When the light distribution state is backlight (step S40; backlight), a face candidate region extraction condition (extraction of low brightness skin color region) corresponding to the backlight is determined (step S41), and according to the determined extraction condition, A low brightness skin color region is extracted from the photographed image data (step S42).

  When the light distribution state is the follow light (step S40; follow light), a face candidate region extraction condition (extraction of a predetermined skin color region) corresponding to the follow light is determined (step S43), and the determined extraction condition is determined. Accordingly, a predetermined skin color region is extracted from the photographed image data (step S44).

  When the light distribution state is strobe (step S40; strobe), the extraction condition of the face candidate region corresponding to the strobe (extraction of the skin color region of high brightness from substantially white) is determined (step S45), and the determined extraction In accordance with the conditions, a skin color region having high brightness from substantially white is extracted from the photographed image data (step S46).

  As the face candidate region extraction processing applied in steps S42, S44, and S46, the simple region expansion method can be used. The simple region expansion method is an image processing method for extracting an image region by expanding adjacent regions with pixels adjacent to each other whose data difference between pixels is equal to or less than a threshold value belonging to the same image region. That is, starting from an initial pixel that matches a specified specific condition, if the data difference between adjacent pixels (either 4-connected or 8-connected) is equal to or less than a threshold value, It is assumed that the initial pixel belongs to the same image area, and the same determination is performed for pixels adjacent to the pixel that belongs to the same pixel, and the same area is gradually expanded starting from the initial pixel. This is an image processing method for extracting an image region. In the present embodiment, it is preferable to further use a process for extracting a face component region by extracting an edge component in an image and ending simple region expansion when the image edge is reached.

  As the initial pixel specifying condition, a condition for specifying a pixel representing a person's skin may be employed. A condition for specifying a pixel representing a person's skin may be any condition as long as it is set based on a predetermined RGB value, and the person's skin color has a hue value within a certain range under substantially the same light source. The flesh-color initial pixel is preferably selected based on a condition that defines the hue. For example, it is preferable to select pixels in the range of 0 to 30 and 330 to 359 with the hue value (Hue) of the HSV color system.

  It is also preferable to define the saturation (Satulation) together. The conditions defining the hue and saturation are preferably changed according to the light source type at the time of photographing, and the light source type at the time of photographing is preferably automatically determined by a known method.

Furthermore, a condition corresponding to the light distribution condition related to brightness may be added to the initial pixel specifying condition. When Th1, Th2, Th3, Th4, Th5, Th6 are thresholds for limiting the brightness of the extraction target area, and the parameter representing the brightness in the area (for example, the average value of the luminance component) is Vave, It may be set in advance so as to satisfy the following relational expression based on the RGB values of a plurality of face images in individual light distribution states.
Backlight: Th1 <Vave <Th2
Forward light: Th3 <Vave <Th4
Strobe: Th5 <Vave <Th6
Here, Th1 to 6 satisfy Th1 <Th2 ≦ Th3 <Th4 ≦ Th5 <Th6, for example,
Backlight: 20 <Vave <65
Forward light: 65 <Vave <200
Strobe: 200 <Vave <255
It is sufficient to set the conditions. In addition, if there is a margin in processing capability, it is possible to improve the extraction accuracy of the face candidate region by providing a condition such as the following formula (when the ranges overlap).
Th1 <Th3 <Th5
Th2 <Th4 <Th6
Th1 <Th2 <Th3
Th5 <Th4 <Th6

  When a face candidate area is extracted in any one of steps S42, S44, and S46, a face determination process is performed on the face candidate area in order to extract a face area from the extracted face candidate area. (Step S47). Then, a face area is extracted from the face candidate area based on the determination result of the face determination process (step S48), and the main area 2 (face) extraction process ends.

  For the face determination process in step S47, a known determination method using pattern matching or a neural network can be applied. For example, a feature part such as an eye pixel and a mouth pixel is extracted from the extracted face candidate area, and an area ratio of the feature part with respect to the face candidate area, an average hue value of the face candidate area, an average saturation value, and the like are input signals, It is possible to determine whether or not the face candidate region is a face by using a neural network whose output signal is a parameter representing the likelihood of a face.

  As described above, according to the image processing apparatus 1 of the first embodiment, the image area determined according to the light distribution condition is extracted from the captured image data, and the top / bottom determination is performed based on the extracted image area. By doing so, it is possible to improve the determination accuracy of the top-and-bottom determination in any shooting environment.

  In particular, if the top / bottom determination is performed based on a background region with a small texture such as the sky, the determination accuracy can be further improved. In addition, when the photographed image data includes a person, if the top / bottom determination is performed based on the person's face area as well as the background area, the top / bottom determination may be performed in consideration of the positional relationship between the background area and the face area. Therefore, the determination accuracy can be further increased.

[Embodiment 2]
Next, Embodiment 2 of the present invention will be described with reference to FIGS.
Since the external configuration and internal configuration of the image processing apparatus according to the second embodiment are the same as those illustrated in FIGS. 1 to 3 in the first embodiment, the same reference numerals as those in the first embodiment are used, and the description of the configuration is omitted.

Next, the operation of the image processing apparatus 1 according to the second embodiment will be described.
First, the flow of the entire image processing executed in the image adjustment processing unit 704 of the second embodiment will be described with reference to the flowchart of FIG.

  First, photographic image data to be image processed is acquired via the film scanner unit 9, the reflective original input device 10, the image transfer means 30, the communication means (input) 32, and the like (step S50), and the acquired photographic image is acquired. A light distribution condition determination process for determining the light distribution condition (light source state) at the time of photographing from the data is performed (step S51). The light distribution condition determination processing in step S51 is the same as the processing shown in FIG.

  Next, an extraction condition for a predetermined image area is determined based on the light distribution condition determined in step S51 (step S52), and an image area is extracted from the captured image data according to the determined extraction condition (step S52). S53). The processing in steps S52 and S53 is the same as the processing shown in FIG. 16 in the first embodiment.

  Next, a feature amount calculation process for calculating a predetermined feature amount from the image region extracted in step S53 is performed (step S54). Here, the predetermined feature amount is data such as the center of gravity, the area ratio, and the gradient of the color value of the extracted image region. The feature amount calculation process in step S54 will be described in detail later with reference to FIG.

  Next, using the feature value calculated in step S54 and the feature value calculated based on the light distribution condition determined in step S51 as an input signal, the top and bottom determination of the captured image data is performed using a neural network (step S51). (S55) The main image processing ends. Here, the feature quantities calculated based on the light distribution conditions are forward light intensity (degree of forward light), reverse light intensity (degree of backlight), and strobe degree (degree of strobe). Is calculated based on The top / bottom determination process using the neural network in step S55 will be described in detail later with reference to FIGS.

  Next, the feature quantity calculation process shown in step S54 of FIG. 17 will be described in detail with reference to the flowchart of FIG.

  When region 1 (sky) is extracted by region 1 (sky) extraction processing in FIG. 16A, the center of gravity of region 1 is calculated as feature amount 1 of region 1 (step S60), and feature amount 2 is The area ratio of the region 1 is calculated (step S61), and the gradient (gradation direction) of the color value of the region 1 is calculated as the feature amount 3 (step S62). The area ratio of the region 1 is a parameter indicating how much the region 1 occupies with respect to the number of pixels constituting the entire captured image data, and is expressed as (number of pixels in the region 1) / (total captured image data). Number of pixels).

  Further, as other feature quantities n of the region 1, the lightness / hue / saturation statistical values, the lightness / hue / saturation gradients, the ratio of the region 1 to the four sides of the image, and the like are also calculated (step S63). .

  When region 2 (face) is extracted by region 2 (face) extraction processing in FIG. 16B, the center of gravity of region 2 is calculated as feature amount 1 of region 2 (step S70). The area ratio of the region 2 is calculated (step S71), and the number of regions 2 (that is, the number of faces) is calculated as the feature amount 3 (step S72). Further, the other feature amount n of the region 2 is also calculated (step S73).

<neural network>
Next, the neural network used in the top / bottom determination process (step S55) in FIG. 17 will be briefly described. As shown in FIG. 19, this neural network is a hierarchical neural network having an input layer, an intermediate layer, and an output layer (hereinafter simply referred to as a neural network). The neural network preferably uses a digital neural network chip, but can also be realized by using a general-purpose DSP (Digital Signal Processor) and a dedicated emulation program, or using a normal CPU and an emulation program. It doesn't matter.

ここで、ｆ（ｘ）は一般に式（１１）に示すような非線形のシグモイド関数が用いられるが、必ずしもこれに限定されるわけではない。

Each layer of the neural network is composed of structural units called neurons. Except for the input layer, an arbitrary neuron receives a large number of inputs called synapses as shown in FIG. 20, and multiplies each input value x _i by a predetermined weight w _i called bond strength to obtain a sum, It functions as a functional element that gives the output y as a result of evaluating it with a predetermined output function f. The output function f is expressed as in Expression (10).

Here, f (x) is generally a non-linear sigmoid function as shown in Equation (11), but is not necessarily limited thereto.

Here, the input weight w _i of each neuron is important information for determining the relationship between the input and output of the neural network, in other words, the operation of the neural network, and is determined by performing a learning operation on the neural network. In FIG. 19, only one intermediate layer is shown, but this may have a configuration of one or more arbitrary layers. Hierarchical networks are known to be able to achieve any function that can be synthesized if there is only one intermediate layer, but increasing the number of intermediate layers is advantageous in terms of learning efficiency and the number of neurons. It has been known.

The neural network has the following characteristics.
(1) A multi-input / multi-output nonlinear system can be realized with a relatively simple configuration. (2) Each neuron in each layer can be operated independently, and high-speed operation can be expected by parallel processing.
(3) Arbitrary input / output relations can be realized by providing appropriate teacher data for learning.
(4) The system has generalization ability. That is, there is an ability to give a generally correct output even to an unlearned input pattern that is not necessarily given as teacher data.

  Here, the teacher data is a pair of an input pattern and a desired output pattern for the input pattern, and a plurality of teacher data are usually prepared. The operation of determining a desired output pattern for a specific input pattern is generally determined by relying on the subjective judgment of an expert. Major applications of neural networks include (a) nonlinear function approximation and (b) clustering (identification, recognition, classification).

  In general, nonlinear function approximation is easy to implement and it is easy to obtain a system with high generalization ability. On the other hand, application to clustering is highly dependent on teacher data, and there is a high possibility that there is a contradiction between a plurality of teacher data for complex input. For this reason, when applied to clustering, unless the input is very simple, the result is not only a low generalization ability, but also an unsatisfactory system in terms of suitability for specific teacher data. There are many things that are not.

ここで、式（１２）の関数ｆ（ｘ）は、式（１１）に示した非線形のシグモイド関数、Ｙ_iLはニューロンの出力、Ｕ_iLはニューロンの内部状態、Ｙ_i＜Ｌ−１＞はL−１層のニューロンの出力＝Ｌ層のニューロンの入力、ｗ_ij＜Ｌ＞は結合強度である。 Next, a learning method using teacher data for the neural network will be described. The state of the i-th neuron in the L-th layer with respect to the p-th input pattern is expressed as in Expression (12) and Expression (13).

Here, the function f (x) in Expression (12) is the nonlinear sigmoid function shown in Expression (11), Y _iL is the output of the neuron, U _iL is the internal state of the neuron, and Y _i <L−1> is L-1 layer neuron output = L layer neuron input, w _ij <L> is the connection strength.

ただしｋは出力層の番号であり、Ｔ_i（ｐ）はｐ番目の入力パターンに対する望ましい出力パターンである。 In such a definition, the widely used error back propagation learning method (back propagation learning method, hereinafter referred to as BP method) uses the mean square error as an error evaluation function, and the error Define E.

Where k is the output layer number and T _i (p) is the desired output pattern for the pth input pattern.

ここで、∂は偏微分記号（ラウンド）、ηは学習率を表す係数である。Δｗ_ij＜Ｌ＞を各々のｗ_ij＜Ｌ＞に加算することによって学習が進む。なお、この形式のＢＰ法では学習が進んだ結果、教師データとの誤差の絶対値が小さくなってくると、Δｗの絶対値も小さくなり、学習が進まなくなるという現象が指摘されており、その問題点を回避するために、種々の改良が提案されている。 In this case, the correction amount Δw of each bond strength based on the BP method is determined by the following equation.

Here, ∂ is a partial differential symbol (round), and η is a coefficient representing a learning rate. Learning proceeds by adding Δw _ij <L> to each w _ij <L>. In addition, as a result of learning progressed in the BP method of this form, if the absolute value of the error from the teacher data becomes small, the absolute value of Δw also becomes small, and the phenomenon that learning does not progress has been pointed out. Various improvements have been proposed in order to avoid problems.

式（１６）の第二項は慣性項（モーメンタム項）と呼ばれ、現在の修正量だけでなく、過去の修正量も考慮することで収束時の振動を抑え、学習を高速化する効果がある。安定化定数αは1．0以下の正の実数で、過去の修正量の考慮の度合いを規定する係数である。 For example, there is a proposal to change the definition of Δw as follows.

The second term in equation (16) is called an inertial term (momentum term), and has the effect of suppressing the vibration at the time of convergence by considering not only the current correction amount but also the past correction amount, and speeding up learning. is there. The stabilization constant α is a positive real number of 1.0 or less, and is a coefficient that defines the degree of consideration of past correction amounts.

Alternatively, there is also a proposal of dynamically changing the learning rate η so as to satisfy the following expression according to the number of learnings n.

  Furthermore, there is also a proposal to make the error evaluation in a form that cancels out the derivative of the sigmoid function instead of the mean square error. In any case, an output approximate to the teacher data can be obtained by performing the learning operation a sufficient number of times using the BP method.

  FIG. 21 shows an example of a neural network applied in the top / bottom determination process of the second embodiment. The neural network used here is calculated based on the feature amount 1, feature amount 2 and feature amount n of region 1 (sky), feature amount 1, feature amount 2 and feature amount n of region 2 (face), and light distribution conditions. The output value Outx representing the sky direction in the x direction (horizontal direction) of the image, and the output representing the sky direction in the y direction (vertical direction) of the image, using the obtained feature amount (backlight level, strobe level, etc.) as input signals It is a neural network that uses the value Outy as an output signal.

  FIG. 22 shows the relationship between the output values Outx and Outy and the top direction of the image. For example, when Outx takes a value in the range of 0.0 to 1.0 from the left to the right of the image, if Outx is 1.0, the right direction of the image is the celestial direction, and if Outx is 0.0, the left side of the image The direction is the celestial direction. When Outy takes a value in the range of 0.0 to 1.0 from the bottom to the top of the image, if Outy is 1.0, the top direction of the image is the top direction, and if Outy is 0.0, the bottom of the image The direction is the celestial direction. Accordingly, the sky direction can be determined according to the magnitude of the two output values Outx and Outy. For example, when Outx is 0.5 and Outy is 1.0, the upward direction is regarded as the top direction.

  In FIG. 21, the number of elements in the output layer is two, but may be four corresponding to the four directions of up, down, left, and right. The configuration of the neural network may be two or more intermediate layers, and the number of elements in each layer may be determined so that the recognition rate is the highest.

  As described above, according to the image processing apparatus 1 of the second embodiment, the image area determined according to the light distribution condition is extracted from the captured image data, and the feature amount and the distribution of the extracted image area are extracted. Improve the accuracy of the top / bottom determination in any shooting environment by using a neural network that uses the feature value of the light condition as the input signal and the output value that represents the top direction of the image as the output signal. And the top / bottom determination reflecting the user's shooting situation.

  In particular, when the top / bottom determination is performed based on the feature amount calculated from the background region with a small texture such as the sky, the determination accuracy can be further improved. In addition, when a person is included in the captured image data, not only the sky area but also the person's face area is extracted, and when the top and bottom determination is performed based on the feature amount of the sky area and the feature amount of the face area, the user's shooting is performed. It is possible to perform the top / bottom determination according to the intention, and the determination accuracy of the top / bottom determination can be further enhanced.

  In addition, the description content in each of the above embodiments can be changed as appropriate without departing from the spirit of the present invention.

FIG. 3 is a perspective view showing an external configuration of the image processing apparatus according to Embodiments 1 and 2 of the present invention. FIG. 3 is a block diagram illustrating a main configuration of the image processing apparatus according to the first and second embodiments. The block diagram which shows the principal part structure of the image processing part of FIG. 5 is a flowchart showing the overall flow of image processing executed by the image adjustment processing unit of the first embodiment. The flowchart which shows a light distribution condition discrimination | determination process. The flowchart which shows the occupation rate calculation process which calculates the 1st occupation rate for every area | region of brightness and hue. The figure which shows an example of the program which converts from RGB to HSV color system. The figure which shows the brightness | luminance (V) -hue (H) plane, and the area | region r1 and the area | region r2 on a VH plane. The figure which shows the brightness | luminance (V) -hue (H) plane, and the area | region r3 and the area | region r4 on a VH plane. The figure which shows the curve showing the 1st coefficient by which the 1st occupation rate for calculating the parameter | index 1 is multiplied. The figure which shows the curve showing the 2nd coefficient by which the 1st occupation rate for calculating the parameter | index 2 is multiplied. The flowchart which shows the occupation rate calculation process which calculates a 2nd occupation rate based on the composition of picked-up image data. The figure which shows the area | regions n1-n4 determined according to the distance from the outer edge of the screen of picked-up image data. The figure which shows the curve showing the 3rd coefficient for multiplying the 2nd occupation rate for calculating the parameter | index 3 according to area | region (n1-n4). The plot figure of the parameter | index 4 and the parameter | index 5 calculated according to light distribution conditions (forward light, strobe, backlight). The flowchart (a) which shows an area | region 1 (sky) extraction process, and the flowchart (b) which shows an area | region 2 (face) extraction process. 9 is a flowchart showing the overall flow of image processing executed by an image adjustment processing unit according to the second embodiment. The flowchart which shows a feature-value calculation process. The figure which shows the structure of a hierarchical neural network typically. The figure which shows the structure of the neuron which comprises a neural network. FIG. 5 is a diagram showing a neural network applied in the second embodiment. The figure which shows the relationship between the output value of the neural network of FIG. 21, and a picked-up image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 4 Exposure processing part 5 Print preparation part 7 Control part 8 CRT
DESCRIPTION OF SYMBOLS 9 Film scanner part 10 Reflective original input device 11 Operation part 14 Image reading part 15 Image writing part 30 Image transfer means 31 Image conveyance part 32 Communication means (input)
33 Communication means (output)
34 External Printer 70 Image Processing Unit 701 Film Scan Data Processing Unit 702 Reflected Original Scan Data Processing Unit 703 Image Data Format Decoding Processing Unit 704 Image Adjustment Processing Unit 705 CRT Specific Processing Unit 706 Printer Specific Processing Unit (1)
707 Printer-specific processing unit (2)
708 Image data format creation processing unit 71 Data storage means

Claims (15)

A discrimination process for discriminating light distribution conditions at the time of shooting from the shot image data,
A determining step of determining an extraction condition of a predetermined image region from the determined light distribution condition;
An extraction step of extracting at least one image region from the captured image data according to the determined extraction condition;
A determination step of determining the top and bottom of the captured image data based on the extracted image region;
An image processing method comprising: A discrimination process for discriminating light distribution conditions at the time of shooting from the shot image data,
A determining step of determining an extraction condition of a predetermined image region from the determined light distribution condition;
An extraction step of extracting at least one image region from the captured image data according to the determined extraction condition;
A calculation step of calculating a predetermined feature amount from the extracted image region;
A determination step of determining the top and bottom of the captured image data using a neural network, using the calculated feature amount and the feature amount calculated based on the determined light distribution condition as an input signal;
An image processing method comprising:   The image processing method according to claim 1, wherein the light distribution condition at the time of photographing includes a backlight state and / or a flash light utilization state.   The image processing method according to claim 1, wherein the image area extracted in the extraction step includes at least a background area of a main subject.   The image processing method according to claim 4, wherein the image area extracted in the extraction step includes at least a human face area. Discriminating means for discriminating light distribution conditions at the time of shooting from the shot image data;
Determining means for determining an extraction condition of a predetermined image region from the determined light distribution condition;
Extracting means for extracting at least one image region from the captured image data according to the determined extraction condition;
Determining means for determining the top and bottom of the captured image data based on the extracted image region;
An image processing apparatus comprising: Discriminating means for discriminating light distribution conditions at the time of shooting from the shot image data;
Determining means for determining an extraction condition of a predetermined image region from the determined light distribution condition;
Extracting means for extracting at least one image region from the captured image data according to the determined extraction condition;
Calculating means for calculating a predetermined feature amount from the extracted image region;
Determination means for determining the top of the photographed image data using a neural network, using the calculated feature value and the feature value calculated based on the determined light distribution condition as an input signal;
An image processing apparatus comprising:   The image processing apparatus according to claim 6, wherein the light distribution condition at the time of photographing includes a backlight state and / or a flash light use state.   The image processing apparatus according to claim 6, wherein the extraction unit extracts at least a background area of a main subject from the captured image data.   The image processing apparatus according to claim 9, wherein the extraction unit extracts at least a human face area from the captured image data. On the computer that performs image processing,
A discriminating function for discriminating light distribution conditions at the time of shooting from shot image data,
A determination function for determining a predetermined image region extraction condition from the determined light distribution condition;
An extraction function for extracting at least one image region from the captured image data according to the determined extraction condition;
A determination function for determining the top and bottom of the captured image data based on the extracted image region;
An image processing program for realizing On the computer that performs image processing,
A discriminating function for discriminating light distribution conditions at the time of shooting from shot image data,
A determination function for determining a predetermined image region extraction condition from the determined light distribution condition;
An extraction function for extracting at least one image region from the captured image data according to the determined extraction condition;
A calculation function for calculating a predetermined feature amount from the extracted image region;
A determination function for determining the top and bottom of the photographed image data using a neural network, using the calculated feature amount and the feature amount calculated based on the determined light distribution condition as an input signal,
An image processing program for realizing   The image processing program according to claim 11 or 12, wherein the light distribution condition at the time of photographing includes a backlight state and / or a flash light utilization state.   The image processing program according to any one of claims 11 to 13, wherein when the extraction function is realized, at least a background area of a main subject is extracted from the captured image data.   15. The image processing program according to claim 14, wherein when the extraction function is realized, at least a human face area is extracted from the photographed image data.

Images