JP2015159396A - Imaging apparatus, image generation method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To eliminate moire caused by the difference between the display update frequency of a display apparatus and the image reception update frequency of an imaging device.SOLUTION: An imaging apparatus 1 acquires preview images obtained from an imaging device 11, successively with a predetermined rate. Next, the imaging apparatus 1 detects moire included in each of the acquired preview images. Next, the imaging apparatus 1 extracts each of portions equivalent to a predetermined phase of the moire from each of the acquired preview images. Next, the imaging apparatus 1 combines the extracted portions to a captured image and generates one piece of image data.

Description

  The present invention relates to an imaging apparatus, an image generation method, and a program.

  Moire (interference fringe) that moves in one direction when content displayed on a display device such as a cathode ray tube or LCD (Liquid Crystal Display) or a display device such as a projector is captured using a camera function such as a digital camera or a smartphone. ) May occur. This is caused by the difference between the display update frequency of the display device and the image reception update frequency of the image sensor. In addition, when flicker is generated in the backlight of the display device, when the display device is imaged using a CMOS sensor that uses a rolling shutter, the same phenomenon occurs due to a difference between the flicker frequency and the scanning speed of the rolling shutter. Moire occurs.

  Note that Patent Document 1 discloses a method for eliminating moiré caused when the pixel interval of the display device does not match the pixel interval of the image sensor.

JP 2013-239917 A

Moire generated when the pixel interval of the display and the pixel interval of the image sensor do not coincide with each other appears at a high spatial frequency, and can be eliminated by fraudulently like the method described in Patent Document 1. . On the other hand, the moire caused by the difference between the display update frequency of the display device and the image receiving update frequency of the image sensor has a low spatial frequency and cannot be eliminated by frowning the image.
The objective of this invention is providing the imaging device, the image generation method, and program which solve the subject mentioned above.

  One aspect of the present invention is an image acquisition unit that sequentially acquires image data obtained from an image sensor at a predetermined rate, a moire detection unit that detects moire included in each image data acquired by the image acquisition unit, An extraction unit that extracts a portion corresponding to a predetermined phase of the moire detected by the moire detection unit from each of the image data acquired by the image acquisition unit and a portion extracted by the extraction unit are combined into one image. And an image generation unit that generates data.

  According to the above aspect, it is possible to eliminate moire caused by the difference between the display update frequency of the display device and the image reception update frequency of the image sensor.

It is a schematic block diagram which shows the hardware constitutions of the imaging device which concerns on some embodiment. It is a schematic block diagram which shows the software structure of the imaging device which concerns on some embodiment. It is a figure which shows the outline of the image generation method by the imaging device which concerns on some embodiment. It is a flowchart which shows operation | movement of the imaging device which concerns on some embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings.
<< First Embodiment >>
FIG. 1 is a schematic block diagram illustrating a hardware configuration of an imaging apparatus 1 according to some embodiments.
The imaging device 1 is a device that converts light received from an optical system into image data that is electronic information. The imaging device 1 includes an imaging device 11, a quantizer 12, a shutter button 13, a CPU 14 (Central Processing Unit), a main storage device 15, an auxiliary storage device 16, an LCD 17, and an LCDC 18 (LCD Controller).

The imaging element 11 is an element that receives light via an optical system and accumulates electric charges according to the amount of light. The image sensor 11 outputs the charges accumulated at a predetermined refresh rate (image receiving update frequency) to the quantizer 12.
The quantizer 12 quantizes the charge output from the image sensor 11 and sequentially generates image data.
The shutter button 13 is a button for determining an imaging timing when pressed.
The LCD 17 sequentially displays images obtained from the quantizer 12 as preview images.
The LCDC 18 is a control device that controls display on the LCD 17.

  In the imaging apparatus 1 according to the present embodiment, the CPU 14 reads out a program from the auxiliary storage device 16, develops it in the main storage device 15, and executes processing according to the program, thereby removing moire included in the image data.

FIG. 2 is a schematic block diagram illustrating a software configuration of the imaging apparatus 1 according to some embodiments.
The CPU 14 includes a preview image acquisition unit 101, a focus processing unit 102, a shutter determination unit 103, a captured image acquisition unit 104, a moire detection unit 105, an extraction unit 106, and an image generation unit 107 by executing a program. Further, the CPU 14 secures storage areas of the preview image holding unit 108 and the captured image holding unit 109 in the main storage device 15 according to the program. Note that the preview image acquisition unit 101 and the captured image acquisition unit 104 are examples of an image acquisition unit.

The preview image acquisition unit 101 acquires the preview image displayed on the LCD 17 from the LCDC 18 according to the refresh rate, and records the preview image in the preview image holding unit 108.
The focus processing unit 102 executes an autofocus process for the optical system.
The shutter determination unit 103 determines whether or not the shutter button 13 has been pressed.
The captured image acquisition unit 104 acquires the captured image captured by the image sensor 11 when the shutter button 13 is pressed from the quantizer 12 and records the captured image in the captured image holding unit 109.

The moiré detection unit 105 detects moiré included in the preview image acquired by the preview image acquisition unit 101.
The extraction unit 106 extracts a luminance peak portion of moire from each preview image held by the preview image holding unit 108.
The image generation unit 107 generates image data without moire by combining the luminance peak portions extracted from the plurality of preview images by the extraction unit 106 with the captured image held by the captured image holding unit 109.

Here, an outline of the image generation method according to the present embodiment will be described.
FIG. 3 is a diagram illustrating an outline of an image generation method by the imaging apparatus 1 according to some embodiments.
As illustrated in FIG. 3, when the imaging apparatus 1 captures an image of a display apparatus, parallel-wave moire may occur in the captured image and the preview image. The moire is caused by a difference between the refresh rate of the display device and the refresh rate of the image sensor 11. At this time, the phase of the moire included in the time-series preview image is displaced by a certain amount. This is because the refresh rate that causes moire is a constant speed.
The imaging device 1 extracts the moiré luminance peak portion from each preview image and combines it with the captured image to generate image data without moiré as shown in FIG.

Next, details of the operation of the imaging apparatus 1 of the present embodiment will be described.
FIG. 4 is a flowchart illustrating the operation of the imaging apparatus 1 according to some embodiments.
When the image pickup apparatus 1 is activated and the optical system is directed toward the subject, the focus processing unit 102 performs auto-focus processing of the optical system and focuses the optical system on the subject (step S1). At this time, the LCDC 18 displays the image data generated by the quantizer 12 on the LCD 17 as a preview image.

  Next, the moire detection unit 105 acquires the preview image displayed on the LCD 17 from the preview image acquisition unit 101, and determines whether or not moire has occurred in the preview image (step S2). Specifically, the moiré detection unit 105 specifies the spatial frequency of the preview image by performing Fourier transform on the time-series preview image acquired by the preview image acquisition unit 101, and a certain amount of change between the time-series preview images. If there is a spatial frequency at which the phase changes, it is determined that moire has occurred in that portion.

  When the moiré detection unit 105 determines that moire has occurred in the preview image (step S2: YES), the preview image acquisition unit 101 records the acquired preview image in the preview image holding unit 108 (step S3). When the preview image is recorded, or when the moire detection unit 105 determines that no moire has occurred in the preview image (step S2: NO), the shutter determination unit 103 determines whether the shutter button 13 has been pressed. (Step S4). If the shutter determination unit 103 determines that the shutter button 13 has not been pressed (step S4: NO), the process returns to step S2 to determine whether or not moire has occurred in the preview image. When moire occurs in the preview image, the preview image is additionally recorded in the preview image holding unit 108 sequentially. On the other hand, the storage area secured as the preview image holding unit 108 in the main storage device 15 is limited. Therefore, when a new preview image cannot be recorded in the storage area secured as the preview image holding unit 108, the preview image acquisition unit 101 discards the oldest preview image and records a new preview image.

  On the other hand, when the shutter determination unit 103 determines that the shutter button 13 has been pressed (step S4: YES), the captured image acquisition unit 104 acquires the image data captured at the timing from the quantizer 12, and Image data is recorded as a captured image in the captured image holding unit 109 (step S5). That is, when the captured image includes moire, the preview image storage unit 108 stores the preview image captured immediately before the captured image. Next, the extraction unit 106 determines whether or not the preview image captured immediately before the captured image is stored in the preview image storage unit 108 (step S6).

  When the extraction unit 106 determines that the preview image captured immediately before the captured image is stored in the preview image storage unit 108 (step S6: YES), the moire detection unit 105 determines the preview image, the captured image, and the like. And a portion where the same moire as the captured image is generated is detected from the preview image (step S7). Specifically, the moire detection unit 105 detects a portion having a spatial frequency in which the phase changes with a constant change amount between the captured image and the time-series preview image as a portion where moire is generated. . At this time, the moire detection unit 105 specifies the moire spatial frequency and the moire phase in each preview image.

  Next, the extraction unit 106 extracts, from each of the preview images held by the preview image holding unit 108, portions corresponding to the luminance peak of the moire detected by the moire detection unit 105 (step S8). Specifically, the extraction unit 106 specifies a luminance peak portion from the preview image based on the moire spatial frequency specified by the moire detection unit 105 in step S7 and the moire phase in each preview image.

  Next, the image generation unit 107 corresponds to a portion of the captured image held by the captured image holding unit 109 that corresponds to the luminance peak of the moire extracted from the preview image by the extraction unit 106 in the portion detected by the moire detection unit 105. Are combined to generate image data without moire (step S9). Then, the image generation unit 107 records the generated image data in the auxiliary storage device 16 (step S10), and ends the process.

  On the other hand, if the extraction unit 106 determines that the preview image is not held in the preview image holding unit 108 (step S6: NO), moire does not occur in the captured image held by the captured image holding unit 109. The captured image is recorded in the auxiliary storage device 16 without performing the synthesis process (step S10), and the process ends.

  By the above processing, the imaging apparatus 1 can obtain image data in which moire caused by the difference between the display update frequency of the display apparatus and the image reception update frequency of the imaging element 11 is eliminated.

<< Second Embodiment >>
In the first embodiment, the case where the image generation unit 107 generates image data without moire by combining the luminance peak portions extracted by the extraction unit 106 has been described, but the present invention is not limited to this.
In the imaging apparatus 1 according to the second embodiment, the extraction unit 106 has a predetermined phase (for example, 60 degrees or 120 degrees) of the moire detected by the moire detection unit 105 from each preview image held by the preview image holding unit 108. Etc.) are extracted. In this case, although the luminance is lower than that in the first embodiment, an image without moire can be obtained as in the first embodiment.

<< Third Embodiment >>
In the first and second embodiments, the extraction unit 106 extracts a portion corresponding to one phase of the moire detected by the moire detection unit 105 from each preview image held by the preview image holding unit 108. Although explained, it is not limited to this.
In the imaging apparatus 1 according to the third embodiment, the extraction unit 106 includes two phases (for example, 60 degrees and 120 degrees, 0 degrees and 180 degrees, etc.) having the same luminance among the moire detected by the moire detection unit 105. ) Are extracted. As a result, the extraction unit 106 can extract an image for synthesis twice that of the first embodiment and the second embodiment from one preview image.

<< Fourth Embodiment >>
In the first embodiment, a case has been described in which the extracting unit 106 extracts a portion corresponding to a luminance peak of moire in a plurality of preview images. However, the present invention is not limited to this.
In the imaging apparatus 1 according to the fourth embodiment, the extraction unit 106 has a predetermined phase width including a luminance peak of moire in a plurality of preview images (for example, from 30 degrees to 60 degrees when the luminance peak is 90 degrees). ) Part is extracted. Specifically, the extraction unit 106 extracts a portion having a phase width corresponding to the amount of moiré phase shift between two consecutive preview images. Then, the image generation unit 107 generates an image without moire by combining the portions corresponding to the last one cycle of the temporal displacement of the moire phase among the plurality of portions extracted by the extraction unit 106.
Thus, the imaging apparatus 1 can obtain an image without moire if the preview image holding unit 108 holds at least the number of preview images corresponding to one cycle of the temporal displacement of the moire phase. That is, the imaging apparatus 1 according to the fourth embodiment can obtain image data without moiré from fewer preview images than in the first to third embodiments.

<< Fifth Embodiment >>
In the fourth embodiment, a case has been described in which the extraction unit 106 extracts a portion having a phase width corresponding to the amount of moiré phase shift between two consecutive preview images. However, the present invention is not limited to this.
In the imaging apparatus 1 according to the fifth embodiment, the extraction unit 106 extracts a portion having a phase width corresponding to twice the amount of moiré phase shift in two consecutive preview images. In this case, when the extraction unit 106 combines the extracted parts, an overlap occurs between the parts. The image generation unit 107 generates image data using the pixel value of the overlapping portion as the average value of the two overlapping pixels. Thereby, the imaging device 1 can obtain more natural image data.

<< Sixth Embodiment >>
In the first embodiment, the case has been described in which the imaging storage removes moire from image data of a still image, but the present invention is not limited to this.
In the imaging device 1 according to the sixth embodiment, the captured image acquisition unit 104 acquires a moving image based on the image data acquired from the quantizer 12. Therefore, the captured image holding unit 109 holds a moving image including a plurality of frame images. Then, the extraction unit 106 extracts a portion including a predetermined phase of moire from the preview image held by the preview image holding unit 108, and the image generation unit 107 uses the captured image holding unit 109 to extract the portion extracted by the extraction unit 106. A moving image is generated by combining each frame image of the moving image to be held. Thereby, the imaging device 1 can obtain a moving image without moire.

<< Seventh Embodiment >>
In the sixth embodiment, the case where the extraction unit 106 extracts a portion including a predetermined phase of moire from the preview image held by the preview image holding unit 108 has been described. However, the present invention is not limited to this.
In the seventh embodiment, the extraction unit 106 extracts a portion including a predetermined phase of moire from a plurality of frame images of a moving image held by the captured image holding unit 109. Thereby, the imaging apparatus 1 does not need to have the preview image holding unit 108.

<< Eighth Embodiment >>
In the first embodiment, the case where the extraction unit 106 extracts a moire luminance peak portion from a preview image captured immediately before the captured image has been described, but the present invention is not limited thereto.
In the eighth embodiment, the preview image acquisition unit 101 continues to record a preview image in the preview image holding unit 108 after the captured image is captured. Then, the extraction unit 106 extracts a moire luminance peak portion from the preview image captured immediately after the captured image. Also in this case, the imaging apparatus 1 can obtain an image without moire as in the first embodiment.

<< Ninth embodiment >>
In the first embodiment, the case of extracting the moire brightness peak portion from the preview image captured immediately before the captured image and the preview image captured immediately after the captured image in the eighth embodiment has been described. However, it is not limited to this.
In the ninth embodiment, the preview image acquisition unit 101 continues to record a preview image in the preview image holding unit 108 after the captured image is captured. Then, the extracting unit 106 extracts a moire luminance peak portion from both the preview image captured immediately before the captured image and the preview image captured immediately after the captured image. Thereby, the imaging device 1 can obtain an image without moiré more accurately than in the first embodiment or the eighth embodiment.

<< Tenth Embodiment >>
In the first embodiment, the case where the CPU 14 performs the moire removal process has been described. However, the present invention is not limited to this. The imaging apparatus 1 according to the tenth embodiment includes a GPU (Graphic Processing Unit), and the GPU performs a moire removal process. Also in this case, the imaging apparatus 1 can obtain an image without moire as in the first embodiment.

  In at least one embodiment described above, the auxiliary storage device 16 is an example of a tangible medium that is not temporary. Other examples of the tangible medium that is not temporary include a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, and a semiconductor memory connected via an interface. When this program is distributed to a computer via a communication line, the computer that has received the distribution may develop the program in the main storage device 15 and execute the above processing.

  The program may be for realizing a part of the functions described above. Further, the program may be a so-called difference file (difference program) that realizes the above-described function in combination with another program already stored in the auxiliary storage device 16.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device 11 ... Imaging element 12 ... Quantizer 13 ... Shutter button 14 ... CPU 15 ... Main memory device 16 ... Auxiliary memory device 17 ... LCD 18 ... LCDC 101 ... Preview image acquisition part 102 ... Focus processing part 103 ... Shutter Determination unit 104 ... Captured image acquisition unit 105 ... Moire detection unit 106 ... Extraction unit 107 ... Image generation unit 108 ... Preview image storage unit 109 ... Captured image storage unit

Claims (5)

An image acquisition unit that sequentially acquires image data obtained from the image sensor at a predetermined rate;
A moire detection unit that detects moire included in each image data acquired by the image acquisition unit;
An extraction unit for extracting each portion corresponding to a predetermined phase of the moire detected by the moire detection unit from each of the image data acquired by the image acquisition unit;
An image generation apparatus comprising: an image generation unit that generates a piece of image data by combining portions extracted by the extraction unit. The said moiré detection part detects the part with the spatial frequency from which the phase changes with a fixed variation | change_quantity between each image data which the said image acquisition part acquired as a part which the moiré has produced. Imaging device. The imaging device according to claim 1, wherein the extraction unit extracts a portion corresponding to a luminance peak of the moire detected by the moire detection unit from each of the image data acquired by the image acquisition unit. . Sequentially acquiring image data obtained from the image sensor at a predetermined rate;
Detecting a portion where moire occurs from each of the acquired image data;
Extracting each portion corresponding to a predetermined phase of the detected moire from each of the acquired image data;
Generating a piece of image data by combining the extracted portions. Computer
An image acquisition unit that sequentially acquires image data obtained from the image sensor at a predetermined rate;
A moiré detection unit that detects moiré included in each image data acquired by the image acquisition unit;
An extraction unit that extracts a portion corresponding to a predetermined phase of the moire detected by the moire detection unit from each of the image data acquired by the image acquisition unit,
The program for functioning as an image generation part which combines the part which the said extraction part extracted, and produces | generates one image data.