KR102415061B1 - Image processing apparatus, image processing method, and photographing apparatus - Google Patents

Abstract

An image processing apparatus according to an embodiment of the present invention includes an image developing unit, a moiré determining unit, and a moiré removing unit. The image developing unit generates at least two original image data by developing at least two image signals output at different timings from a sensor array in which a plurality of light receiving elements are arranged in a two-dimensional Bayer array. The moire determination unit determines an area in which moire occurs from at least two pieces of original image data generated by the image developing unit. The moiré removal unit removes moiré from the video signal according to the determination result of the moiré generation region by the moiré determination unit.

Description

Image processing apparatus, image processing method, and photographing apparatus TECHNICAL FIELD

The present invention relates to an image processing apparatus, an image processing method, and a photographing apparatus.

In imaging devices such as digital still cameras and digital video cameras, the image sensor (Image Sensor) used to capture still images or moving pictures includes a Bayer array CMOS image sensor or CCD. An image sensor is mainly used.

The Bayer arrangement is an arrangement in which four pixels including two green pixels, one red pixel, and one blue pixel are configured as a pair. The Bayer arrangement was developed to reduce the design of a complex optical system and to eliminate pixel shift in relation to the original light spectrum and RGB light reception by a separate sensor. This Bayer arrangement utilizes the characteristic that the human eye is insensitive to red or blue resolution and is sensitive to green or black and white.

When generating an image using the YCbCr color space or RGB color model from the signal output from the image sensor of the Bayer array (hereinafter referred to as “development”), it is due to the irradiation of the image with a period shorter than the existence period of each color Pulsed color (color moiré) may occur. A technique for removing this pulsed color (color moiré) is being developed (see, for example, Japanese Patent Application Laid-Open No. 2005-210218 as Patent Document 1 and Japanese Patent Application Laid-Open No. 2010-4396 as Patent Document 2, etc.) .

Japanese Patent Laid-Open No. 2005-210218 Japanese Patent Laid-Open No. 2010-4396

The technique disclosed in Japanese Patent Laid-Open No. 2005-210218 (hereinafter referred to as Patent Document 1) is to reduce color moiré by changing the Bayer arrangement. In addition, the technique disclosed in Japanese Patent Application Laid-Open No. 2010-4396 (hereinafter referred to as Patent Document 2) uses a temporally different color interpolation image to reduce noise by a three-dimensional noise reduction method to achieve color moiré. is to suppress

However, there is a problem in that neither technique is effective in reducing color moiré when the data of the entire Bayer array is used for development.

The problem of the background art as described above is the content that the inventor possessed for the derivation of the present invention or acquired during the derivation of the present invention, and it cannot be said that the content was known to the general public before the filing of the present invention.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing apparatus, an image processing method, and a photographing apparatus capable of effectively removing color moiré when developed using data of the entire Bayer array.

An image processing apparatus according to a first aspect of the present invention includes an image developing unit, a moiré determining unit, and a moiré removing unit.

The image developing unit develops at least two image signals output at different timings in time from a sensor array in which a plurality of light receiving elements that receive light through an optical system and photoelectrically convert them are arranged in a two-dimensional Bayer array, Generates two original image data.

The moire determination unit determines a region in which moire is generated from the at least two original image data generated by the image developing unit.

The moiré removing unit removes moiré from the video signal according to a determination result of the moiré generation region by the moiré determining unit.

The moiré determination unit may determine an area in which a difference occurs in the repeating patterns of the first color and the second color in the at least two original image data as the moiré generation area.

The first color may be a color having the largest number of pixels in the bayer arrangement, and the second color may be a color generated from a color other than the first color in the bayer arrangement. For example, the first color may be green, and the second color may be magenta.

The moiré removal process performed by the moire removal unit may be an interpolation process performed on the original image data according to the determination result.

In addition, the moire removal processing performed by the moiré removal unit may be a filter processing according to the determination result on the image signals obtained by removing noise from the two image signals. Here, the filter processing performed by the moire removal unit may be a process of adding values of pixels in an oblique direction to values of adjacent pixels in an oblique direction, respectively.

The image processing apparatus may further include a noise removing unit that removes noise from the two image signals used for determination by the moiré determination unit.

An image processing method as a second aspect of the present invention includes an image developing step, a moiré determination step, and a moiré removal step.

In the image developing step, at least two image signals output at different timings from a sensor array in which a plurality of light receiving elements that receive light through an optical system and photoelectrically convert them are arranged in a two-dimensional Bayer array are developed, At least two pieces of original image data are generated.

In the moire determination step, an area in which moire occurs is determined from the at least two original image data generated in the image developing step.

In the moiré removal step, moiré is removed from the video signal according to the determination result of the moiré generation region by the moiré determination step.

A photographing apparatus as a third aspect of the present invention includes a photographing unit, an image developing unit, a moiré determination unit, and a moiré removing unit.

The photographing unit includes a sensor array in which a plurality of light receiving elements that receive light through an optical system and photoelectrically convert the light beam are arranged in a two-dimensional Bayer arrangement.

The image developing unit generates at least two pieces of original image data by developing at least two image signals output from the photographing unit at different timings.

The moire determination unit determines an area in which moire is generated from the at least two original image data generated by the image developing unit.

The moiré removing unit removes moiré from the video signal according to a determination result of the moiré generation region by the moiré determining unit.

According to the image processing apparatus, the image processing method, and the photographing apparatus of the embodiments of the present invention, at least two image signals output at different timings are developed to generate at least two pieces of original image data. and created An area in which moire occurs is determined from the at least two original image data.

Accordingly, color moiré that appears when the data of the entire Bayer array is used for development can be effectively removed.

1 is a view showing a bayer arrangement.
2 is a diagram illustrating an example of an image serving as a photographing source.
FIG. 3 is a view showing an example of an image in which color moiré occurs when the image shown in FIG. 2 is captured.
4 is a view showing a state in which the color moiré of cyan and orange occurs in a Bayer arrangement.
FIG. 5 is a diagram showing a state in which the color moiré of cyan and orange occurs in a Bayer arrangement.
FIG. 6 is a diagram showing the occurrence of green and magenta color moiré in a Bayer arrangement.
7 is a diagram showing an example of the configuration of the photographing apparatus 100 according to an embodiment of the present invention.
8 is a view showing an example of the configuration of the noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention.
9 is a flowchart illustrating an example of an operation of the photographing apparatus 100 according to an embodiment of the present invention.
10 is a diagram illustrating an example in which a part of image data of an (N-1)-th frame and an N-th frame is cut.
11 is a view showing an example of the configuration of the noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, in the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and thus redundant descriptions are omitted.

<1. Background>

Before describing the preferred embodiment of the present invention in detail, the background leading to the embodiment of the present invention will be described.

As described above, in a photographing apparatus such as a digital camera or a digital video camera, a Bayer array CMOS image sensor or a CCD image sensor is mainly used as a sensor used for photographing a still image or a moving image.

1 is a view showing a bayer arrangement. The Bayer arrangement is an arrangement in which 4 pixels of 1 red (R) pixel and 1 blue (B) pixel are configured as a pair for 2 green (G) pixels. A Bayer arrangement as shown in FIG. 1 was developed to reduce the design of a complex optical system and to eliminate pixel shift, which originally received RGB by a separate image sensor by splitting light.

The Bayer arrangement shown in FIG. 1 utilizes the characteristic that the human eye is insensitive to red or blue resolution and is sensitive to green or black and white. In addition, for convenience, the green (G) pixel in the red (R) row is referred to as Gr, and the green (G) pixel in the blue (B) row is referred to as Gb.

Since the Bayer arrangement consists of a pair of 4 pixels of 1 red (R) pixel and 1 blue (B) 1 pixel for 2 green (G) pixels, green is the image sensor in the horizontal and vertical directions. It has the same band as the number of horizontal and vertical pixels, but decreases to half of the horizontal and vertical in the band in the 45 ^O oblique direction. Accordingly, the luminance signal has a small band in the oblique direction. Also, since the Bayer arrangement consists of a pair of 4 pixels of 1 red (R) pixel and 1 blue (B) 1 pixel for 2 green (G) pixels, the blue and red regions are of green color. only half

In order to obtain a normal image with such a Bayer array image sensor, conventionally, for example, a filter that does not resolve lines thinner than the basic 4-pixel thickness of the Bayer array is mounted on the image sensor to blur each color. thing has been done

However, when such a filter is mounted on an image sensor, red and blue thin lines are not output from the image sensor, and green, which is a major component of luminance, is also not output from the sensor if it is thinner than 2-pixel width.

And, of course, mounting such a filter on the image sensor causes cost increase and also causes image resolution to be lowered compared to the actual number of pixels. As a result, the necessity of excluding the filter and satisfying both resolution improvement and cost reduction is argued, and products aiming to satisfy both resolution improvement and cost reduction are also sold.

However, when an image sensor excluding a filter is used, when generating (developing) an image using the YCbCr color space or RGB color model from the signal output from the image sensor of the Bayer array, it is shorter than the existence period of each color. Pulsed color (color moiré) resulting from periodic image irradiation may occur.

When the image sensor excluding the filter is used, for example, a red image of a shorter period than the red existence period of the image sensor is irradiated. Irradiating a red image with a shorter period than the red existence period of the image sensor causes aliasing for red.

This can be said to be the same not only for red, but also for blue in which only one pixel exists among four pixels. In addition, irradiating a vivid green image in an oblique direction also for green causes oblique aliasing with respect to green. This aliasing becomes color moiré that is generated when an image using the YCbCr color space or RGB color model is generated (developed) from the signal output from the image sensor of the Bayer array.

The biggest problem is the moire of green and magenta that occurs when thin lines in the oblique direction are developed. This indicates an oblique disturbance. This green and magenta moiré phenomenon occurs because there are very thin lines in the oblique direction in the captured image.

2 is a diagram illustrating an example of an image serving as a photographing source. And FIG. 3 is a view showing an example of an image in which color moiré occurs when the image shown in FIG. 2 is captured.

When an image in which very thin white and black lines are mixed as shown in FIG. 2 is taken, colors other than white and black do not exist in the original image, but as shown in FIG. 3 , swirl-shaped pulse color (color moiré) may occur. Reference numeral 11 in Fig. 3 denotes a portion where cyan and orange swirl-shaped pulse colors (color moiré) are generated. Reference numeral 12 in Fig. 3 denotes a portion in which green and magenta swirl-shaped pulse colors (color moiré) are generated.

The color moiré of cyan and orange generated in the portion indicated by reference numeral 11 in FIG. 3 is generated to reproduce a vertical or horizontal thin line. 4 and 5 are views showing the occurrence of cyan and orange color moiré in a Bayer arrangement.

As shown in FIGS. 4 and 5 , when the color moiré of cyan and orange occurs, it can be seen that the green part is changed at the boundary between cyan and orange. Therefore, it can be determined whether or not the color moiré of cyan and orange occurs by taking the correlation with green. In addition, when the color moiré of cyan and orange occurs, for example, the color moiré may be removed by connecting a portion where the color moiré occurs with white.

On the other hand, green and magenta color moiré generated in the portion indicated by reference numeral 12 in FIG. 3 is generated to reproduce the thin line in the oblique direction. FIG. 6 is a diagram showing the occurrence of green and magenta color moiré in a Bayer arrangement.

However, in the case of this green and magenta color moiré, the occurrence of color moiré cannot be determined by correlating with green like that of cyan and orange color moiré. This is because, unlike the color moiré of cyan and orange, as shown in FIG. 6 , it cannot be said that green is always attached.

Therefore, the present inventor earnestly studied the technique for removing color moiré, especially the technique for effectively removing the color moiré of green and magenta in which the occurrence of color moiré cannot be determined by taking a correlation with green. As a result, the present inventors, as will be described below, by comparing at least two images that are temporally different, determine whether there is color moirage, and devise a technique for removing color moiré according to the determination result. reached

In the above, the background leading to the embodiment of the present invention has been described. Next, an embodiment of the present invention will be described in detail.

<2. One embodiment of the present invention>

[2.1. configuration example]

First, an example of the configuration of a photographing apparatus according to an embodiment of the present invention will be described in detail. 7 shows an example of the configuration of the photographing apparatus 100 according to an embodiment of the present invention. Hereinafter, an example of the configuration of the photographing apparatus 100 according to an embodiment of the present invention will be described with reference to FIG. 7 .

7 , the photographing apparatus 100 according to an embodiment of the present invention includes a photographing unit 102 , a noise removing unit 104 , an interpolation processing unit 106 , a codec unit 108 , It is configured to include a communication unit 110 , a bus 112 , a control unit 114 , and a DDR   SDRAM (Double   Data   Rate   Synchronous   Dynamic Random Access Memory 116 ).

The photographing unit 102 includes a plurality of lens groups including a zoom lens or a focus lens, an image sensor, and the like. The photographing unit 102 photoelectrically converts a subject image formed on the light-receiving surface of the image sensor, and performs signal processing such as predetermined noise removal processing, sensitivity adjustment processing, and analog/digital conversion (A/D conversion) to digital Generate a video signal.

The image sensor installed in the photographing unit 102 may be a charge coupled device (CCD) image sensor or a complementary metal-oxide semiconductor (CMOS) image sensor. In addition, a primary color filter of a Bayer arrangement is installed on the light receiving surface of the image sensor installed in the photographing unit 102 to capture a color image. The digital image signal generated by the photographing unit 102 is stored in the DDR SDRAM 116 via the bus 112 .

The noise removal unit 104 performs noise removal processing on the digital image signal generated by the photographing unit 102 and stored in the DDR SDRAM 116 . In the present embodiment, the noise removing unit 104 reads out at least two digital image signals generated by the photographing unit 102 at different times from the DDR SDRAM 116 via the bus 112 , and 3D noise Noise reduction is performed on a digital video signal by 3DNR (3-Dimension Noise Reduction). Three-dimensional noise reduction (3DNR) is a noise removal process that detects noise by comparing at least two temporally successive digital image signals and removes the detected noise.

In the present embodiment, the noise removal unit 104 uses at least two digital image signals read out for the noise removal process, and determines whether color moiré, in particular, green and magenta color moiré is generated. The photographing apparatus 100 according to an embodiment of the present invention determines whether or not color moiré is generated by using at least two digital image signals read for noise removal processing, thereby reducing a configuration for detecting color moiré as noise. It can be common to the configuration for the removal process.

After the noise is removed by the noise removal unit 104, the interpolation processing unit 106 performs interpolation processing, for example, CFA (Color Filter Array) interpolation processing, on the digital video signal stored in the DDR SDRAM 116. Develop processing is performed. The video data obtained by performing the development process by the interpolation processing unit 106 is stored in the DDR SDRAM 116 via the bus 112 .

The codec unit 108 performs encoding processing on the image data generated by the interpolation processing unit 106 and decoding processing on the encoded image data. Encoding processing and decoding processing are collectively referred to as codec processing. The video data subjected to the codec process by the codec unit 108 is stored in the DDR SDRAM 116 via the bus 112 .

The communication unit 110 executes a process for transmitting information to an external device and a process for receiving information from an external device. For example, the communication unit 110 transmits image data encoded by the codec unit 108 to an external device.

The control unit 114 is constituted by, for example, a CPU (Central Processing Unit) and the like, and controls the operation of each unit of the photographing apparatus 100 .

Although not shown in FIG. 7 , the photographing apparatus 100 includes a shutter button for instructing the user to take an image with respect to the photographing unit 102 , an operation button for accepting a user's operation on the photographing apparatus 100 , and interpolation A recording medium or the like for storing the image data generated by the processing unit 106 may be provided. The recording medium 116 for preserving the image data generated by the interpolation processing unit 106 may be detachable from the photographing apparatus 100 .

In the above, an example of the configuration of the photographing apparatus 100 according to an embodiment of the present invention has been described with reference to FIG. 7 . Next, an example of a detailed configuration of the noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention will be described.

8 is a diagram illustrating an example of the configuration of the noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention. Hereinafter, an example of the configuration of the noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention will be described with reference to FIG. 8 .

As shown in FIG. 8 , the noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention includes RDMA (Read Direct Memory Access) controllers 121a and 121b, and the subtraction unit 122. , the noise evaluation unit 123 , the noise removal unit 124 , the filter processing unit 125 , the simple developing units 126a and 126b , the moiré determination unit 127 , and the WDMA (Write Direct Memory Access) controller 128 . is comprised of

The RDMA controllers 121a and 121b read at least two digital image signals generated by the photographing unit 102 at different times from the DDR SDRAM 116 via the bus 112 . In the example shown in Fig. 8, one RDMA controller 121a reads the digital image signal of the (N-1)-th frame from the DDR SDRAM 116 . Another RDMA controller 121b reads the digital image signal of the N-th frame, which is the next frame of the (N-1)-th frame, from the DDR SDRAM 116 . The RDMA controllers 121a and 121b may read the digital image signal of the continuous frame as described above from the DDR SDRAM 116, and read the digital image signal obtained at short intervals but not continuous from the DDR SDRAM 116. may be

The RDMA controllers 121a and 121b respectively subtract the digital image signals of the (N-1)-th frame and the N-th frame read from the DDR SDRAM 116 via the bus 112 into a subtraction unit 122 and a simple developing unit. (126a, 126b). Also, when the RDMA controllers 121a and 121b transmit the digital image signal to the subtraction unit 122 and the simple developing unit 126a and 126b, respectively, they may pass through a buffer memory (not shown).

The subtraction unit 122 performs subtraction processing on the digital image signals of the (N-1)-th frame and the N-th frame transmitted from the RDMA controllers 121a and 121b. Specifically, the subtraction unit 122 performs a subtraction process for subtracting the digital image signal of the (N-1)-th frame from the digital image signal of the N-th frame in units of pixels. The subtraction unit 122 transmits the subtracted digital image signal to the noise evaluation unit 123 .

In the present embodiment, the subtraction unit 122 performs the subtraction processing of the digital image signal of temporally successive frames, so that the noise evaluation unit 123 at the subsequent stage evaluates noise, specifically, determines whether noise is generated, and The noise generating part can be detected.

The noise evaluation unit 123 performs noise evaluation on the digital image signal after subtracting the digital image signal of the (N-1)th frame from the digital image signal of the Nth frame transmitted from the subtraction unit 122 in units of pixels. carry out

In the noise evaluation, specifically, the noise evaluation unit 123 generates noise in the digital image signal after subtracting the digital image signal of the (N-1)th frame from the digital image signal of the Nth frame in units of pixels. Judgment of the presence or absence and a search for a noise generating part are performed. A basic way of thinking about 3D noise reduction is to determine that if there is a significant difference in the digital image signal of two consecutive frames in time, that part is considered as noise. The noise evaluation unit 123 performs noise evaluation on the digital image signal obtained by subtracting the digital image signal of the (N-1)th frame from the digital image signal of the Nth frame in units of pixels, and determines the result of the evaluation as noise. It is sent to the removal unit 124 .

The noise removal unit 124 performs noise removal processing on the digital image signal of the Nth frame according to the noise evaluation result transmitted from the noise evaluation unit 123 . The noise removing unit 124 transmits the digital image signal of the Nth frame (hereinafter, referred to as an N′th frame for convenience) after noise is removed to the filter processing unit 125 .

In addition, the noise removal processing in the noise removal unit 124 is not limited to a specific processing, and various processing corresponding to 3D noise reduction may be applied.

The filter processing unit 125 performs filter processing on the digital image signal of the N′-th frame after noise is removed according to a result of color moiré determination by the moire determination unit 127 . The filter processing according to the color moiré determination result performed by the filter processing unit 125 is performed by adding color moiré, in particular, for a portion in which green and magenta color moiré occurs, for pixels adjacent to a 45 ^O oblique direction. Filter processing is performed to remove color moiré so that the development result becomes black and white. The direction in which color moiré is occurring can be detected, for example, by comparing the pixel values of each pixel. In addition, in the filter processing by the filter processing unit 125, all processing for removing the color moiré of green and magenta is possible, so it is not limited to the above-described addition processing for pixels adjacent to the 45 ^O oblique direction.

The filter processing unit 125 performs filter processing according to the color moiré determination result by the moiré determination unit 127 on the digital image signal of the N′-th frame after noise has been removed, so that the digital image of the N′-th frame is performed. It is possible to remove the color moiré that occurs in the signal, especially the color moiré that appears repeatedly in green and magenta. The filter processing unit 125 transmits the digital image signal of the N'th frame after color moiré has been removed to the WDMA controller 128 .

The simple developing units 126a and 126b generate image data by performing simple developing processing on the digital image signals of the (N-1)th frame and the Nth frame transmitted from the RDMA controllers 121a and 121b. The simple developing units 126a and 126b generate image data by performing simple developing processing on the digital image signal, respectively, and send the generated image data to the moire determination unit 127 .

The moiré determination unit 127 determines whether or not color moiré, particularly green and magenta color moiré, is generated from the image data of the (N-1)th frame and the Nth frame transmitted from the simple developing units 126a and 126b. perform processing. Although the moiré determination processing according to the present embodiment will be described in detail later, for example, the moiré determination unit 127 divides the image data after simple development into a predetermined number of areas, and provides color moiré for each of the areas. to determine whether the occurrence of

In the present embodiment, when the occurrence or non-occurrence of color moiré is determined, the probability of color moiré may be determined in a plurality of steps, for example, 10 steps of 1 to 10. In this case, the one with a larger number may be made into a thing with a high probability of color moirail. The moiré determination unit 127 transmits to the filter processing unit 125 the result of color moiré determination performed on the video data of the (N-1)th frame and the Nth frame in association with information identifying each area. .

For example, the information for identifying the region increases in the right direction with the upper-left region of the reduced image data being 0, and moves down one stage when reaching the rightmost region, and moves from the left end to the right. The value increases again towards the end, and this may be repeated until the lower-right region is reached.

The WDMA controller 128 writes the digital video signal of the N'-th frame from which the color moiré is removed by the filter processing unit 125 to the DDR SDRAM 116 via the bus 112 .

The noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention has a configuration as shown in FIG. 7 , and thus uses two temporally consecutive digital image signals to reduce noise. In addition to the removal, it is determined whether or not color moiré has occurred from image data obtained by developing the two digital image signals, and if color moiré has occurred, the color moiré can be removed.

An example of the detailed configuration of the noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention has been described with reference to FIG. 8 . Next, an example of the operation of the photographing apparatus 100 according to an embodiment of the present invention will be described.

[2.2. Example of operation]

9 shows an example of the operation of the photographing apparatus 100 according to an embodiment of the present invention. Here, the photographing apparatus 100 according to an embodiment of the present invention determines whether or not color moiré occurs from image data obtained by developing two temporally consecutive digital image signals, and when color moiré occurs, the Remove color moiré. Hereinafter, an example of the operation of the photographing apparatus 100 according to an embodiment of the present invention will be described with reference to FIG. 9 .

The photographing apparatus 100 reads two digital image signals that are temporally different, particularly temporally consecutive, from the DDR SDRAM 116 via the bus 112 (step S101). The read processing of this step S101 is performed, for example, by the RDMA controllers 121a and 121b.

When two digital image signals that are different in time are read from the DDR SDRAM 116 in step S101, the photographing device 100 simply develops the two digital image signals to generate two image data ( step S102). The simple developing processing of this step S102 is executed by, for example, the simple developing units 126a and 126b.

When two digital image signals are simply developed in step S102 and two image data are generated, the photographing apparatus 100 determines a portion in which color moiré occurs from the two image data (step S103). The determination processing of this step S103 is executed, for example, by the moire determination unit 127 .

The determination processing in step S103 will be described in detail. In this embodiment, image data is divided into a predetermined number of regions, and in each of the regions, it is determined whether or not color moiré occurs. In the present embodiment, when determining whether or not the occurrence of color moiré occurs, the probability of color moiré may be determined using a plurality of steps, for example, 10 steps of 1 to 10. In this case, the one with a larger number may be made into a thing with a high probability of color moirail.

10 shows an example in which a part of image data of the (N-1)-th frame and the N-th frame is cut. 10 shows an example in which the same part is cut out with respect to the image data of the (N-1)-th frame and the N-th frame.

In the example shown in Fig. 10, green and magenta patterns are generated in a part of the image data of the (N-1)-th frame and the N-th frame. In the present embodiment, it is determined as follows whether the green and magenta patterns are really caused by color moiré.

In the example shown in FIG. 10 , it can be seen that the positions of green and magenta shapes occurring in a part of the image data of the (N-1)th frame and the Nth frame are different. As shown in FIG. 9 , when a change occurs between images of two consecutive frames, in this embodiment, the color moa range is evaluated based on the steepness of the pattern and the degree of change in the pattern. In addition, in the present embodiment, whether a region in which a color significantly different from that of other regions appears is also used as a material for evaluating whether a color moire range is present.

Also in the present embodiment, the color moirail probability may be determined by, for example, the size of a range in which the position of the shape is different between images of two consecutive frames.

If it is determined in step S103 that a portion in which color moiré occurs, the photographing apparatus 100 continuously detects the Nth digital image signal, particularly noise, according to the determination result of the portion in which color moiré occurs in step S103. Color moiré is removed on the digital image signal of the N'-th frame after being removed (step S104). The color moiré removal processing in step S104 is performed, for example, by the filter processing unit 125 .

The filter processing unit 125, for example, with respect to a portion in which color moiré, particularly green and magenta color moiré, occurs, adds color moiré to pixels adjacent to the 45 ^O oblique direction so that the development result becomes black and white. Perform filter processing to remove. In addition, as long as the filter process in step S104 is a process for removing green and magenta color moiré, it is not limited to the above-described addition process for pixels adjacent to the 45 ^O oblique direction.

The photographing apparatus 100 according to the present exemplary embodiment may effectively remove color moiré, particularly green and magenta color moiré, and generate an image without color moiré by executing a series of operations as described above.

[2.3. Variation example]

In the above-described photographing apparatus 100 , the noise removing unit 104 removes color moiré in addition to noise removal, but the present invention is not limited to this example. For example, the noise removal unit 104 may only perform a determination process on the color moiré generating portion, and the interpolation processing unit 106 may generate image data by performing interpolation processing while removing the color moiré according to the determination result. .

11 shows an example of the configuration of the noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention. Hereinafter, an example of the configuration of the noise removing unit 104 included in the photographing apparatus 100 according to an embodiment of the present invention will be described with reference to FIG. 11 .

The noise removing unit 104 shown in FIG. 11 does not include the filter processing unit 125 unlike the noise removing unit 104 shown in FIG. 8 .

Accordingly, the noise removal unit 124 performs noise removal processing on the digital image signal of the Nth frame according to the evaluation result of the noise transmitted from the noise evaluation unit 123 , and removes the noise from the Nth frame A digital image signal of (hereinafter, referred to as an N'-th frame for convenience) is transmitted to the WDMA controller 128 .

Further, the moiré determination unit 127 determines whether or not color moiré, particularly green and magenta color moiré, is generated from the image data of the (N-1)th frame and the Nth frame transmitted from the simple developing units 126a and 126b. The process of determining is performed, and the determination result of color moiré is sent to the WDMA controller 128 .

Then, the WDMA controller 128 calculates the digital video signal of the N'th frame after the noise sent from the noise removal unit 124 has been removed and the color moiré determination result for each area sent from the moiré determination unit 127 to determine the area. In association with the identifying information, it is written to the DDR SDRAM 116 via the bus 112 .

As the noise removing unit 104 has the configuration as shown in FIG. 11 , in the interpolation processing unit 106 of the photographing apparatus 100 , the digital image signal of the N′th frame recorded in the DDR  SDRAM 116 . and reading the determination result of color moiré, and performing interpolation while removing moiré to generate image data.

Further, in the interpolation processing unit 106, the digital image signal of the N'th frame and the color moiré determination result recorded in the DDR SDRAM 116 are read, and the interpolation processing is performed while removing the moiré to generate image data. . In this case, it is possible to generate an image that is clear and has no color moiré compared to the case where the filter processing is performed.

<3. Summary>

As described above, according to an embodiment of the present invention, the occurrence of color moiré, in particular, occurrence of green and magenta color moiré, can be determined by comparing digital image signals acquired at different temporally, in particular temporally, consecutive timings. A photographing apparatus 100 for determining is provided. Also, according to an embodiment of the present invention, there is provided a photographing apparatus 100 that determines whether or not color moiré occurs, particularly green and magenta color moiré, and performs a process of removing color moiré according to the determination result. do.

The photographing apparatus 100 according to an embodiment of the present invention uses digital image signals acquired at different temporally, in particular, temporally continuous timings as described above, without increasing the memory capacity for color moiré determination. It is possible to determine whether or not moiré occurs, in particular, the occurrence of green and magenta color moiré. In particular, the photographing apparatus 100 according to an embodiment of the present invention can determine green and magenta color moiré caused by thin lines near an inclined pixel pitch, which has been difficult until now, and can remove the color moiré.

Each step in the processing executed by the apparatus of the present specification is not necessarily processed in a time series according to a flowchart or an order described as a flowchart. For example, each step in the processing executed by each device may be processed in a sequence different from the sequence described in the flowchart, or may be processed in parallel.

In addition, in order to exhibit the functions equivalent to the configuration of each device described above, hardware (not shown) such as CPU (Central Processing Unit), ROM (Read Only Memory) and RAM (Random Access Memory) embedded in the device of the present specification. A computer program may also be written. In addition, a storage medium in which the computer program is stored may be provided. Further, by configuring each functional block shown in the functional block diagram with hardware, a series of processing can be realized with hardware.

In the above, preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to these examples. It is clear that various changes or modifications can be derived by those of ordinary skill in the art to which the present invention belongs, within the scope of the technical idea described in the claims, and of course the present invention It is understood to be within the technical scope of

The present invention is highly likely to be used in image processing apparatuses having various functions.

100 shooting devices
102 Cinematography
104 noise canceling unit
106 interpolation processing unit
108 codec part
110 Ministry of Communications
112 bus
114 control
116 SDRAM
121a RDMA controller
121b RDMA controller
122 Production Reduction Department
123 Noise Evaluator
124 noise canceling unit
125 filter processing unit
126a simple developing unit
126b simple developing unit
127 Moiré Judgment Division
128 WDMA controller

Claims (6)

At least two image signals output at different timings in time from a sensor array excluding filters, in which a plurality of light receiving elements that receive light through the optical system and photoelectrically convert them are arranged in a two-dimensional Bayer array, an image developing unit that generates original image data;
a moire determination unit which determines an area in which moire is generated from the at least two original image data generated by the image developing unit; and
and a moiré removing unit configured to remove moiré from the image signal according to the determination result of the moiré generation region by the moiré determining unit;
The moiré determination unit,
The steepness of the pattern and the degree of change in the pattern for the region where the difference occurs in the repeating patterns of green, which has the largest number of pixels in the Bayer array, and magenta, which is not green, in the Bayer array of the at least two original image data , an image processing apparatus for determining a moiré generation region based on whether a region in which a color is significantly different from that of other regions is present, and a size of a range in which a shape is displaced in images of successive frames. delete delete The method according to claim 1, wherein the moire removal process performed by the moire removal unit comprises:
and interpolation processing performed on the original image data according to the determination result. At least two image signals output at different timings in time from a sensor array excluding filters, in which a plurality of light receiving elements that receive light through the optical system and photoelectrically convert them are arranged in a two-dimensional Bayer array, an image developing step of generating original image data;
a moire determination step of determining an area in which moire occurs from the at least two original image data generated in the image developing step; and
a moiré removal step of removing moiré from the image signal according to the determination result of the moiré generation region by the moiré determination step;
The moiré determination step is
The steepness of the pattern and the degree of change in the pattern for the region where the difference occurs in the repeating patterns of green, which has the largest number of pixels in the Bayer array, and magenta, which is not green, in the Bayer array of the at least two original image data , an image processing method for determining a moiré generation region based on whether a region in which a color significantly different from that of other regions is displayed, and a size of a range in which a shape is displaced in images of successive frames. a photographing unit having a sensor array excluding a filter, in which a plurality of light receiving elements that receive light through an optical system and photoelectrically convert them are disposed in a two-dimensional Bayer array;
an image developing unit which develops at least two image signals output from the photographing unit at different timings to generate at least two original image data;
a moire determination unit which determines an area in which moire is generated from the at least two original image data generated by the image developing unit; and
and a moiré removing unit configured to remove moiré from the image signal according to the determination result of the moiré generation region by the moiré determining unit;
The moiré determination unit,
The steepness of the pattern and the degree of change in the pattern for the region where the difference occurs in the repeating patterns of green, which has the largest number of pixels in the Bayer array, and magenta, which is not green, in the Bayer array of the at least two original image data , a photographing apparatus for determining a moiré generation region based on whether a region in which a color significantly different from that of other regions is present, and a size of a range in which a shape is displaced in images of successive frames.

Images