JP2015216604A - Image processing system, method of the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system capable of effectively excluding only image data not intended by a photographer.SOLUTION: A digital camera 100 as the image processing system comprises a region splitter 116 and an image composition unit 117. The region splitter 116 splits image data into plural regions and the image composition unit 117 composes a region other than a night sky and a region of the night sky split from the image data by the region splitter 116 by comparatively bright composition or comparatively dark composition.

Description

  The present invention relates to an image processing apparatus and method, and a program.

  There is known an imaging apparatus that performs comparatively bright combination of a plurality of image data photographed at a constant time interval in order to photograph a light trail of a star moving in the night sky (see, for example, Patent Document 1).

  By the way, the image data used for comparatively bright combination sometimes includes high-intensity image data not intended by the photographer, for example, image data of a building illuminated by the light of a headlight of a car. When performing comparatively bright combination using image data including such high-brightness image data, the image data after comparatively bright combination includes high-intensity image data not intended by the photographer. As a result, the brightness of a part of the image data increases, so that the clearness of the image data of the light trail of the star having relatively low brightness may be lost in the entire image data after the comparative bright combination.

  Correspondingly, in the imaging apparatus of Patent Document 1, each of two pieces of image data to be subjected to comparatively bright combination is divided into a plurality of regions, a luminance average value for each divided region is calculated, and two pieces of image data are calculated. The difference between the average brightness values of the corresponding areas is calculated in step S1, and the area where the calculated difference between the average brightness values exceeds a threshold preset by the user is not used when the two image data are relatively brightly combined. The image data of the building illuminated by the light of the headlight of the car is only temporarily bright, and the building is the first image data and the second image data taken at different timings. Therefore, the image pickup apparatus of Patent Document 1 can exclude the high-luminance building image data from the combined image data.

JP 2013-172372 A

  However, in the imaging apparatus of Patent Document 1, a region in which the difference between the calculated average luminance values exceeds a threshold value set in advance by the user is not used when two image data are relatively brightly synthesized. When the threshold value is set to a relatively small value, not only high brightness building image data not intended by the photographer at the time of comparative bright combination, but also high brightness star image data is combined. There is a possibility that only image data not intended by the photographer can be effectively excluded.

  An object of the present invention is to provide an image processing apparatus and method, and a program that can effectively exclude only image data not intended by a photographer.

  In order to achieve the above object, an image processing apparatus according to the present invention includes a combining unit configured to combine the first image data and the second image data captured at different timings by comparative bright combination or comparative dark combination, One image data is divided into a first area and a second area, and the second image data is divided into a third area and a second area so as to correspond to the first area and the second area, respectively. Dividing means for dividing the area into four areas, wherein the combining means combines the first area and the third area, and the second area and the fourth area with the comparatively bright combination or the comparison, respectively. It is characterized by being synthesized by dark synthesis.

  According to the present invention, it is possible to effectively exclude only image data not intended by the photographer.

It is a block diagram which shows the internal structure of the digital camera as an image processing apparatus which concerns on embodiment of this invention. It is a figure used in order to demonstrate the image data image | photographed when the exposure start time was t0 with the digital camera of FIG. It is a figure used in order to demonstrate the image data image | photographed when the exposure start time was t1 with the digital camera of FIG. It is a figure used in order to explain the image data which carried out comparative dark composition of each image data of Drawing 2A and Drawing 2B. It is a figure used for demonstrating the edge detection image data which detected the edge by the area division part from the image data of FIG. 2C. It is a figure used in order to demonstrate the image data image | photographed when the exposure start time was t0 with the digital camera of FIG. It is a figure used in order to demonstrate the image data image | photographed when the exposure start time was t1 with the digital camera of FIG. It is a figure used in order to explain the image data which carried out comparative bright composition or comparative dark composition of each image data of Drawing 3A and Drawing 3B. It is a figure used in order to demonstrate the image data image | photographed when the exposure start time is t2 with the digital camera of FIG. It is a figure used in order to explain the image data which carried out comparative bright combination or comparative dark combination of each image data of Drawing 3C and Drawing 3D. It is a flowchart which shows the procedure of the interval imaging | photography process performed by CPU in FIG. It is a flowchart which shows the procedure of the division | segmentation process performed by FIG.4 S406. FIG. 5 is a flowchart showing a procedure of region-by-region combining processing executed in step S408 of FIG. 4. FIG. It is a flowchart which shows the procedure of the area | region determination process performed by step S603 of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing an internal configuration of a digital camera 100 as an image processing apparatus according to an embodiment of the present invention.

  The digital camera 100 includes a photographing lens 101, apertures 102 and 103, an imaging unit 104, an A / D conversion unit 105, a timing generator (hereinafter referred to as “TG”) 106, a memory 107, an image processing unit 108, a recording I / F 109, A display I / F 110, a monitor 111, an operation unit 112, a CPU 113, a system bus 114, a recording medium 115, an area division unit 116, and an image composition unit 117 are provided.

  The A / D conversion unit 105, the memory 107, the image processing unit 108, the recording I / F 109, the display I / F 110, the CPU 113, the area dividing unit 116, and the image composition unit 117 are connected to each other via the system bus 114 and recorded. A recording medium 115 and a monitor 111 are connected to the I / F 109 and the display I / F 110, respectively. In addition, the diaphragms 102 and 103, the TG 106, and the operation unit 112 are connected to the CPU 113, the imaging unit 104 and the A / D conversion unit 105 are connected to the TG 106, and the imaging unit 104 is connected to the A / D conversion unit 105. Has been.

  The photographing lens 101 is composed of a plurality of lenses including, for example, a zoom lens and a focus lens, and is moved in the optical axis direction by a lens driving unit (not shown). The diaphragms 102 and 103 adjust the amount of light incident through the photographing lens 101. The imaging unit 104 includes a CCD or a CMOS element that converts an optical image into an electrical signal. The A / D converter 105 converts an analog signal into a digital signal. The TG 106 supplies a timing pulse to the imaging unit 104 and the A / D conversion unit 105 and controls the driving timing of the imaging unit 104 and the A / D conversion unit 105. The memory 107 stores captured image data and image data obtained by combining a plurality of image data, and records a digital signal converted by the A / D conversion unit 105.

  The image processing unit 108 performs processing on the image data, for example, changes the gradation brightness of the image data. The monitor 111 is, for example, a liquid crystal monitor, and displays captured image data and a screen for adjusting brightness and contrast. The CPU 113 controls the digital camera 100 by executing a sequence program stored in a ROM (not shown). The recording medium 115 is a memory card with a built-in hard disk or semiconductor memory and stores, for example, image data. The operation unit 112 has a release button and an operation button.

  The area dividing unit 116 divides the image data stored in the memory 107. The area dividing unit 116 detects an edge that is a boundary between light and dark in the image data. The image synthesis unit 117 synthesizes, for example, two image data stored in the memory 107. The image synthesis unit 117 may synthesize image data for each region divided by the region division unit 116, and the two image data are synthesized by, for example, comparative bright synthesis or comparative dark synthesis.

  Note that comparatively bright combination means that when two image data shot at different timings are combined, each of the pixels constituting one image data of the same address and each of the pixels constituting the other image data are combined. This is a method of synthesizing image data by adopting the brightness of pixels with higher brightness. Further, comparative dark combining is the process of combining two image data shot at different timings, each of the pixels constituting one image data of the same address and each of the pixels constituting the other image data. This is a method of synthesizing image data by using the luminance of a pixel having a lower luminance.

  2A to 2D are diagrams used to explain a method of dividing image data into a plurality of regions by the region dividing unit 116 of FIG.

  2A and 2B are diagrams used to explain image data captured by the digital camera 100 of FIG. The image data of FIG. 2A is taken when the time when the digital camera 100 starts exposure (hereinafter referred to as “exposure start time”) at the time of shooting is t0, and the image data of FIG. 2B is constant from the exposure start time t0. The image is taken at the exposure start time t1 when the time has passed. The image data in FIG. 2A includes stars 201 and 202 and a building 203, and the image data in FIG. 2B includes stars 204 and 205 and a building 203 which are the same as the stars 201 and 202, respectively. The stars 201 and 202 move to the positions of the stars 204 and 205 with the passage of time due to the rotation of the earth.

  FIG. 2C is a diagram used for explaining image data obtained by comparatively dark-combining the image data in FIGS. 2A and 2B. Since the stars 201 and 202 have moved to the positions of the stars 204 and 205 between the exposure start time t0 and the exposure start time t1, the stars 201 and 202 are located at positions corresponding to the positions of the stars 201 and 202 in FIG. 2B. Does not exist, and the brightness at this position is extremely low. Similarly, in FIG. 2A, the stars 204 and 205 do not exist at positions corresponding to the positions of the stars 204 and 205, and the luminance at the positions is extremely low. Therefore, when the image data of FIGS. 2A and 2B are comparatively dark-synthesized, the positions of the synthesized image data corresponding to the stars 201 and 202 and the positions corresponding to the stars 204 and 205 are The stars 204 and 205 do not exist, and only the building 203 exists in the synthesized image data (FIG. 2C).

  FIG. 2D is a diagram used for explaining edge detection image data in which an edge is detected by the region dividing unit 116 from the image data of FIG. 2C. Only the building 203 exists in the image data of FIG. 2C, and when an edge is detected, only the outline of the building 203 is detected as a light / dark boundary portion of the image data (FIG. 2D).

  As described above, since the stars 201 and 202 and the stars 204 and 205 do not exist in the image data synthesized by the comparative dark synthesis of the image data of FIGS. 2A and 2B, the image data of FIGS. When the image is divided into an area where stars exist (hereinafter referred to as “night sky area”) and an area where no stars exist (hereinafter referred to as “non-night sky area”), the area where the edge is detected is an area other than the night sky. Can be specified. In other words, the image data can be divided into an area other than the night sky and an area of the night sky with the edge as a boundary.

  According to the method of dividing the image data of FIGS. 2A to 2D into a plurality of regions, the image data of FIG. 2A and the image data of FIG. 2B are synthesized by comparative dark synthesis, and edges are synthesized from the synthesized image data (FIG. 2C). (FIG. 2D), and the image data can be divided into a region other than the night sky and a night sky region with the edge as a boundary. Therefore, when combining image data in FIGS. 3A to 3E described later, comparative dark combining is applied. Areas other than the power night sky (high brightness areas) and night sky areas (low brightness areas) to which the comparatively bright composition should be applied can be accurately discriminated, so that a predetermined law independent of image data, for example, Compared to the case where image data is divided into a plurality of areas according to a method of dividing image data into predetermined blocks, and comparative dark synthesis or comparatively bright synthesis is applied to each divided area. It is possible to improve the accuracy of the application of the dark synthesis and lighten compositing.

  Note that the image data may be divided based on the distance from the digital camera 100 to the subject. For example, in the image data of FIG. 2A, stars 201 and 202 and a building 203 are detected as subjects, and the distances from the digital camera 100 to the stars 201, 202 and the building 203 are measured by a triangulation, a phase difference method, or the like. The Since the distance from the digital camera 100 to the stars 201 and 202 is extremely longer than the distance from the digital camera 100 to the building 203, the image data is divided into an area including the stars 201 and 202 and an area including the building 203. At this time, a region including the stars 201 and 202 corresponds to a night sky region, and a region including the building 203 corresponds to a region other than the night sky.

  3A to 3E are diagrams used to describe a method of combining each of the regions divided from the image data by the region dividing unit 116 when the image combining unit 117 combines a plurality of image data. .

  3A, 3B, and 3D are diagrams used for explaining image data captured by the digital camera 100 of FIG. The image data of FIG. 3A is taken when the exposure start time is t0, and the image data of FIG. 2B is taken when the exposure start time t1 when a certain time has elapsed from the exposure start time t0. The image data is taken at the exposure start time t2 when a certain time has elapsed from the exposure start time t1.

  3A includes stars 301 and 302 and a building 203, the image data illustrated in FIG. 3B includes stars 303 and 304 and a building 203, and the image data illustrated in FIG. 3D includes stars 305 and 306 and a building 203. And high-intensity light 307 irradiated by a car (not shown) that passes by chance.

  In the method of combining a plurality of image data of FIGS. 3A to 3E, the image data of FIG. 2A to FIG. 2D is divided into a plurality of regions as a premise, so that the image data is divided into regions other than the night sky and night sky regions. The image data of FIG. 3A, FIG. 3B, and FIG. 3D is synthesized for each of the divided areas other than the night sky and the night sky area. Specifically, the area of the night sky in FIG. 3A and the area of the night sky in FIG. 3B are relatively brightly synthesized, and the area other than the night sky in FIG. 3A and the area other than the night sky in FIG. Thereby, image data in which the stars 301 to 304 and the building 203 exist is obtained (FIG. 3C).

  Next, the image data of FIGS. 3C and 3D is synthesized for each of the area other than the night sky and the night sky area. Specifically, the night sky region of FIG. 3C and the night sky region of FIG. 3D are comparatively brightly combined, and the region other than the night sky of FIG. 3C and the region other than the night sky of FIG. At this time, even when the area other than the night sky in FIG. 3C or FIG. 3D temporarily becomes a high-brightness area by the high-intensity light 307, the area other than the night sky in FIG. 3C and the area other than the night sky in FIG. Are synthesized by comparative dark synthesis, the high-intensity light 307 is excluded from the synthesized image data, and image data including only the stars 301 to 306 and the building 203 is obtained (FIG. 3E).

  According to the method for synthesizing a plurality of image data in FIGS. 3A to 3E, the areas other than the night sky in FIGS. 3A, 3B, and 3D are synthesized by comparative bright synthesis, and the night sky in FIGS. 3A, 3B, and 3D is synthesized. Regions are synthesized by comparative dark synthesis. Thus, since the comparative bright combination or the comparative dark combination can be used properly for each divided region, when the high luminance light 307 not intended by the photographer is generated, the divided region including the high luminance light 307 is generated. By using comparative dark combining for combining (regions other than the night sky), only the high-intensity light 307 can be effectively excluded from the combined image data.

  FIG. 4 is a flowchart showing the procedure of interval shooting processing executed by the CPU 113 in FIG. The interval shooting is, for example, shooting in which an exposure operation is started at a preset time interval when shooting a light trail of a star moving in the night sky.

  In FIG. 4, in order to execute interval shooting, the time interval at which the exposure operation is started is set by the user (step S401), and it is determined whether or not the time for starting the exposure operation has passed (step S402). As a result of the determination in step S402, when the time for starting the exposure operation has not elapsed, the process returns to step S402, and when the time for starting the exposure operation has elapsed, it is determined whether or not a division process described later is being performed. (Step S403).

  If the result of determination in step S403 is that division processing has already been performed, the exposure operation is started (step S404). If division processing has not been performed, it is determined whether or not the division processing can be performed (step S404). Step S405). If the result of determination in step S405 is that division processing can be executed, the division processing is executed to divide the image data into a plurality of regions (step S406), and the process proceeds to step S404. On the other hand, when the division process cannot be executed (NO in step S405), the exposure operation is started (step S407), and the process returns to step S402.

  Next, a region-by-region combining process described later is executed to combine the image data for each region divided from the image data (step S408), and it is determined whether or not the end of interval shooting has been instructed (step S409). When the end of interval shooting is not instructed, the process returns to step S402, and when the end of interval shooting is instructed, the process ends.

  FIG. 5 is a flowchart showing the procedure of the division process executed in step S406 of FIG.

  In FIG. 5, first, according to the method of dividing the image data of FIGS. 2A to 2D into a plurality of regions, two image data used for the division processing are acquired (step S501), and the two acquired image data are compared. The synthesized image data is synthesized by dark synthesis (step S502), the edge of the synthesized image data is detected (step S503), and the image data is divided into a plurality of regions, for example, a region other than the night sky and a night sky region, using the detected edge as a boundary. (Step S504), and this process is terminated.

  FIG. 6 is a flowchart showing the procedure of the region-by-region combining process executed in step S408 of FIG. In the region-by-region combination processing, image data is combined for each region divided from the image data by the division processing in FIG. 5 according to the method of combining a plurality of image data in FIGS. 3A to 3E in principle.

  In FIG. 6, two regions to be combined among the plurality of divided regions are specified (step S601), and a region determination process described later is executed on the specified regions to perform comparative bright combination or comparative darkness on the regions. It is determined which combination is to be applied (step S602), comparative bright combination or comparative dark combination is performed on the specified image data area (step S603), and the combined image data is stored in the memory 107. (Step S604), it is determined whether or not the synthesis of all the regions divided from the image data has been completed (Step S605).

  If the result of determination in step S605 is that the synthesis of all areas divided from the image data has not been completed, the process returns to step S601 to perform the synthesis for another area that has not yet been synthesized, and the image data When the synthesis of all the areas divided from is completed, this processing is terminated.

  FIG. 7 is a flowchart showing the procedure of the area determination process executed in step S603 of FIG.

In FIG. 7, the luminance value of each of the two areas specified in step S601 is calculated, and the luminance difference between the two areas is calculated based on the calculated luminance value (step S701). Is greater than a threshold value (Y _th ) preset by the user (step S702). When the calculated luminance difference is larger than the threshold value, the two regions are synthesized by comparative dark synthesis (step S703), and when the calculated luminance difference is smaller than the threshold value, the two regions are synthesized by comparative bright synthesis (step S703). Step S704).

  Next, it is determined whether or not all divided areas have been combined (step S705). When all the divided areas have not been combined, the process returns to step S701, and when all the divided areas have been combined, this process ends.

  4 to 7, the image data is divided into a plurality of regions (steps S406 and S504), and either the comparative bright combination or the comparative dark combination is applied to each of the regions divided from the image data. Since the two image data are synthesized (steps S408, S601 to 603, S703, and S704), it is possible to selectively use comparatively bright synthesis or comparatively dark synthesis for each divided area. 4 to 7 are executed by the CPU 113 of the digital camera 100, the image data photographed by the digital camera 100 can be used immediately for comparative bright synthesis or comparative dark synthesis.

  Further, according to the processing of FIG. 7, the brightness value of each of the two areas to be combined is calculated, and the brightness difference between the two areas is calculated based on the calculated brightness value (step S701). If the luminance difference is larger than a threshold value set in advance by the user, the image data is synthesized by comparative dark synthesis (step S703). When the brightness difference in a certain area is larger than the set threshold value, it is estimated that image data not intended by the photographer, for example, high brightness light 307 is included in the image data. The region estimated to contain the light 307 can be synthesized by comparative dark synthesis, so that only image data not intended by the photographer can be effectively excluded from the synthesized image data.

  In the process of FIG. 7, a synthesis method (comparative dark synthesis or comparison) applied to each area based on whether or not the luminance difference between the two image data synthesized in each area exceeds a preset threshold value. Bright combination) is determined, but the combination method may be determined based on the area of each divided region. For example, when the image data is divided into a night sky region and a region other than the night sky, the areas of the two regions are compared, and a region with a larger area is synthesized by comparative bright synthesis. When shooting light trails of stars moving in the night sky, the night sky occupies a large area in the image, so it is possible to estimate that the area of the large area is the night sky, and also compare the areas estimated to be the night sky Since the image data is synthesized by the bright synthesis, the possibility that the star image data is excluded from the synthesized image data can be avoided.

  Software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program. In the processing, the program and a computer-readable storage medium storing the program constitute the present invention.

100 Digital camera 113 CPU
116 Region segmentation unit 117 Image composition unit 201-202, 204-205, 301-306 Star 203 Building 307 High brightness light

Claims (9)

Combining means for combining the first image data and the second image data captured at different timings based on brightness;
The first image data is divided into a first area and a second area, and the second image data is divided into a third area so as to correspond to each of the first area and the second area. And a dividing means for dividing into a fourth region,
The image synthesizing unit synthesizes the first area, the third area, the second area, and the fourth area based on brightness.   The image processing apparatus according to claim 1, wherein the synthesizing unit synthesizes the first image data and the second image data by comparative bright synthesis or comparative dark synthesis.   The areas of the first region and the second region are compared, and when the area of the first region is larger than the area of the second region, the first region and the third region are Synthesized by comparative bright synthesis, and when the area of the second region is larger than the area of the first region, the second region and the fourth region are synthesized by the comparative bright synthesis. The image processing apparatus according to claim 2. Calculating means for calculating luminance values of the first region, the second region, the third region, and the fourth region;
The luminance difference between the calculated luminance value of the first region and the luminance value of the third region, or the luminance difference between the calculated luminance value of the second region and the luminance value of the fourth region. Brightness difference calculating means for calculating
The luminance difference between the luminance value of the first region and the luminance value of the third region, or the luminance difference of the luminance value of the second region and the luminance value of the fourth region is greater than a preset threshold value. 4. If larger, the first area and the third area are synthesized by the comparative dark synthesis, or the second area and the fourth area are synthesized. Image processing apparatus.   The dividing unit combines the first image data and the second image data by the comparative dark combination, detects a light / dark boundary portion from the combined image data, and uses the boundary portion as a boundary. The image processing apparatus according to claim 2, wherein the image data is divided into a high luminance area and a low luminance area.   6. The image processing apparatus according to claim 5, wherein the synthesizing unit synthesizes the high luminance area by the comparative dark synthesis and synthesizes the low luminance area by the comparative bright synthesis.   The image processing apparatus according to claim 1, wherein the image processing apparatus is an imaging apparatus. A combining step of combining the first image data and the second image data captured at different timings based on brightness;
The first image data is divided into a first area and a second area, and the second image data is divided into a third area so as to correspond to each of the first area and the second area. And a dividing step of dividing into a fourth region,
In the composition step, the first region and the third region, and the second region and the fourth region are composed based on brightness, respectively. A program for causing a computer to execute an image processing method,
The image processing method includes combining a first image data and a second image data captured at different timings based on brightness;
The first image data is divided into a first area and a second area, and the second image data is divided into a third area so as to correspond to each of the first area and the second area. And a dividing step of dividing into a fourth region,
In the composition step, the first region and the third region, and the second region and the fourth region are respectively composed based on brightness.