JP2015111775A - Imaging device, method for controlling the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow for generation of a high-quality monitor output video while securing peaking performance without causing an increase in a circuit scale and an increase in a memory band.SOLUTION: An imaging device comprises: a first frame memory 104 which records RAW data of a video signal obtained by an imaging element 101; a second memory 105 which records development data developed from the RAW data by signal processing; and a peaking signal generation unit 118 which generates a peaking signal from the RAW data recorded in the first frame memory 104. The imaging device switches whether to use the RAW data recorded in the first frame memory 104 or the development data recorded in the second frame memory 105 according to a magnification of a focus assist function as a video signal for generating a monitor output video using a main controller 111 and a changeover switch 113.

Description

  The present invention relates to an imaging apparatus having a focus assist function for superposing a peaking signal on a video signal, a control method thereof, and a program.

  Conventionally, in order to facilitate focus adjustment, an imaging apparatus such as a digital video camera is provided with a focus assist function for detecting a high-frequency component during focus adjustment and superimposing a peaking signal on a video signal. The peaking signal is a signal generated by extracting and amplifying a high frequency component from a luminance signal constituting a video signal. By adding this peaking signal to the video signal and displaying it on the viewfinder or output monitor, the contour portion in the video is emphasized and displayed. Thereby, it is possible to visually notify the user of the focus application, and the focus adjustment can be easily performed. This focus assist function is called peaking, contour correction, edge enhancement, enhancer, or the like.

For example, in Patent Document 1, as a “focus adjustment signal generation device”, any one of absolute values obtained by converting a luminance signal into an RGB signal and applying LPF and HPF in the vertical direction or horizontal direction to each of RGB is obtained. It is disclosed that an RGB signal for focus adjustment is generated by addition.
Patent Document 2 discloses a “viewfinder, imaging device, and display signal generation circuit” that detects a high-frequency component for each of the three primary colors of a video signal, extracts a peaking signal, performs addition by weighting the signal, It is disclosed to add to the three primary colors of the video signal.

Japanese Patent No. 4474641 Japanese Patent No. 4245019

Some image pickup apparatuses can record both raw output data from an image sensor (hereinafter referred to as RAW data) and development data developed by signal processing from RAW data.
In this type of imaging apparatus, a configuration that improves peaking performance by detecting a high-frequency component from RAW data with a large amount of information and generating a peaking signal can be considered.

However, when the recording frame rate and the display frame rate are different, it is necessary to pass RAW data to the monitor output circuit and perform development processing again, which increases the circuit scale and power consumption.
Also, by passing both RAW data and development data to the monitor output circuit, it is not necessary to perform development processing on the monitor output circuit side, but the amount of data increases, leading to an increase in memory bandwidth.

  The present invention has been made in view of the above points, and it is possible to generate a high-quality monitor output video while ensuring peaking performance without causing an increase in circuit scale and memory bandwidth. The purpose is to do so.

  An image pickup apparatus according to the present invention includes an image pickup element that converts a subject optical image formed through an optical system into a video signal, and a first recording unit that records RAW data of the video signal obtained by the image pickup element. A second recording unit that records development data developed by signal processing from the RAW data, a peaking signal generation unit that generates a peaking signal from the RAW data recorded in the first recording unit, and a monitor output video Whether to use RAW data recorded in the first recording means or development data recorded in the second recording means in accordance with the magnification rate of the focus assist function as a video signal for generating And switching control means.

  According to the present invention, it is possible to generate a high-quality monitor output video while ensuring peaking performance without causing an increase in circuit scale and memory bandwidth.

It is a block diagram which shows the structure of the imaging device which concerns on 1st Embodiment. It is a figure for demonstrating the outline | summary of the process in a Debayer circuit. It is a figure for demonstrating the outline | summary of a focus assist function. It is a figure which shows the structure of a general peaking signal generation part. It is a figure for demonstrating the data reading from a frame memory, and its process. It is a figure for demonstrating the outline | summary of the process in a simple Debayer circuit. It is a figure which shows the reading of a data by an expansion rate, and a zone | band. 6 is a flowchart illustrating processing performed by the imaging apparatus according to the first embodiment. It is a block diagram which shows the structure of the imaging device which concerns on 1st Embodiment. It is a figure for demonstrating the data reading from a frame memory, and its process. It is a flowchart which shows the process by the imaging device which concerns on 2nd Embodiment. It is a figure for demonstrating the example of the system which detects a high frequency component from RAW data, and produces | generates a peaking signal. It is a figure for demonstrating the example of the system which detects a high frequency component from RAW data, and produces | generates a peaking signal. It is a figure for demonstrating the example of the system which detects a high frequency component from RAW data, and produces | generates a peaking signal.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.
<First Embodiment>
FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus according to the first embodiment.
Reference numeral 101 denotes an image sensor that converts a subject optical image formed through an optical system into a video signal. Reference numeral 102 denotes a Debayer circuit. Reference numeral 103 denotes a signal processing circuit. Reference numeral 104 denotes a first frame memory, which functions as a first recording unit that records RAW data of a video signal obtained by the image sensor 101. Reference numeral 105 denotes a second frame memory, which functions as a second recording unit that records development data developed by signal processing from RAW data. Reference numeral 106 denotes a video signal recording unit 1. Reference numeral 107 denotes a video signal recording unit 2. Reference numeral 110 denotes a display block. Reference numeral 111 denotes a memory controller.
In the display block 110, reference numeral 112 denotes a simple Debayer circuit. Reference numeral 113 denotes a changeover switch. Reference numeral 114 denotes a peaking signal superimposing circuit. Reference numeral 115 denotes a scaling circuit. Reference numeral 116 denotes a monitor signal processing circuit. Reference numeral 117 denotes a display device. Reference numeral 118 denotes a peaking signal generator.
An enlargement ratio control signal 119 is input as a control signal to the memory controller 111 and the changeover switch 113 that control an input signal to the display block 110.

Light enters the image sensor 101 from a subject through a lens system (not shown). The signal output AD-converted by the image sensor 101 is an R, Gr, Gb, B Bayer array, and is stored in the first frame memory 104 as it is to record RAW data.
The imaging apparatus according to the present embodiment can simultaneously record RAW data and development data developed from the RAW data by signal processing. That is, the RAW data is converted into R, G, and B signal planes by the Debayer circuit 102, subjected to signal processing by the signal processing circuit 103, and then stored in the second frame memory 104.
Although there are various known examples of processing methods of the Debayer circuit 102, as shown in FIG. 2, when generating R, G, and B pixels from one Bayer 201, a plurality of pixels of the same color plane are used to correct pixel center-of-gravity displacement. In general, a method of generating a pixel at the position 203 by calculation using the. FIG. 2B shows the concept of the Debayer process in consideration of gravity center correction. Since it is desirable to generate RGB pixels on the same point position 203, it is obtained by calculation in consideration of the pixel distance from a plurality of pixels. FIG. 2C shows a concept of generating one R pixel 209 from four R pixels 205, 206, 207, and 208. Since the pixel distance is 1: 3, the R pixel 209 can be obtained by weighting calculation of each pixel. The B pixel is calculated in the same manner as in FIG. 2C, and the G pixel can be obtained by (Gr + Gb) / 2.

  The RAW data stored in the first frame memory 104 is recorded in the video signal recording unit 1 (106). The development data stored in the second frame memory 105 is recorded in the video signal recording unit 2 (107).

Next, the focus assist function in the display block 110 will be described.
FIG. 3 is a diagram for explaining the outline of the focus assist function. FIG. 3A shows an output image of the display device 117 when the focus assist function is OFF and the enlargement ratio is equal. When the enlargement ratio is equal, a signal with the same angle of view flows through the video signal recording and display device 117. Here, when the user wants to focus on a part of the subject, by turning on the focus assist function, as shown in FIG. 3B, the video with the edge information 302 added by the peaking signal is displayed on the display device 117. Is displayed. As a result, it is possible to visually inform the user of the focus condition. In addition, as shown in FIG. 3C, an enlargement ratio is set and a predetermined range 301 (see FIG. 3A) is enlarged and displayed, thereby informing the user of the lighting condition with high accuracy. Is possible.

  FIG. 4 is a diagram illustrating a basic configuration of the peaking signal generation unit. The peaking signal generation unit includes an FIR filter 401, and a luminance signal, an RGB development signal, or a Bayer Raw input signal is input as an input signal. Further, the intensity and frequency of the peaking signal can be adjusted by the gain adjustment signal and the frequency adjustment signal.

  Here, a peaking signal that is highly accurate edge information can be generated by using a signal input to the peaking signal generation unit 118 as RAW data of multiple pixels instead of development data. An example of a method for generating a peaking signal by detecting a high-frequency component from RAW data will be described later.

Hereinafter, a method of superimposing a peaking signal on the monitor output video will be described. Since the monitor output video to the display block 110 already has development data developed in the second frame memory 105, it is ideal to use this development data.
However, in this case, it is necessary to transmit both the RAW data and the development data to the display block 110, which causes a problem of increasing the memory bandwidth.

Therefore, the generation method of the monitor output video on the display block 110 is changed in accordance with the magnification ratio of the focus assist.
FIG. 5 is a diagram for explaining data reading from the frame memory and its processing. The memory controller 111, the simple Debayer circuit 112, the changeover switch 113, the peaking superimposing circuit 114, and the peaking signal generation unit 118 are extracted and illustrated. Is.

  When the magnification rate of the focus assist function is equal, the RAW data 501 is read from the first frame memory 104 by the memory controller 111 and the changeover switch 113 as shown in FIG. Input to the simple Debayer circuit 112.

  FIG. 6 shows an outline of processing in the simple Debayer circuit 112. Unlike the processing in the Debayer circuit 102 shown in FIG. 2, the simplified Debayer circuit 112 does not perform pixel center of gravity shift correction. Pixel center-of-gravity correction requires a line memory because a plurality of pixels are used in the vertical direction, and increases the circuit scale because one pixel is obtained from a plurality of pixels by calculation. Specifically, as shown in FIGS. 6A and 6B, R, B, and G pixels are generated from one Bayer 601 respectively. The R, B, and B planes can be developed as indicated by 602, 603, and 604 by using the R and B pixels as they are and the G pixel using the average value of the Gr and Gb pixels. However, in this case, since the center-of-gravity shift of the pixels is not taken into consideration, the centers of gravity of the R, G, and B pixels are different as shown in FIG. For this reason, a false color is generated depending on the image, and this false color problem tends to be noticeable by enlarging the display image. However, when the enlargement ratio is equal, the false color problem due to the deviation of the center of gravity is not conspicuous, and can be simplified by the simple Debayer.

Next, a case where the magnification rate of the focus assist function is twice or more will be described. When the enlargement ratio increases, the false color problem of the center of gravity shift becomes noticeable in the processing by the simple Debayer circuit 112 as described above.
Therefore, as shown in FIG. 5B, the RAW data 501 is read from the first frame memory 104 by the memory controller 111 and the changeover switch 113 and input to the peaking signal generation unit 118. Further, the development data 502 is read from the second frame memory 105 by the memory controller 111 and the changeover switch 113 and the peaking signal generated by the peaking signal generation unit 118 is superimposed without passing through the simple Debayer circuit 112.
When the enlargement ratio is 2 times or more, only the necessary range 301 (see FIG. 3A) is read from the frame memory. Therefore, as shown in FIG. 7, the read band from the frame memory is enlarged. If the Bayer Raw is 1 when the rate is equal, the Bayer Raw is 1/4 when the enlargement rate is 2 and 1/16 for each color of developed RGB, so the total is 7/16, and the memory bandwidth is the enlargement rate. Is less than or equal to RAW data reading at the same magnification. Therefore, the reading method as described above does not lead to an increase in memory bandwidth.
Since the development data 502 read from the second frame memory 105 has been corrected for the center of gravity deviation by the Debayer circuit 102, the false color problem does not occur.

FIG. 8 is a flowchart illustrating processing performed by the imaging apparatus according to the first embodiment.
The monitor output enlargement ratio is determined (step S801), and if the enlargement ratio is equal to or larger than a predetermined magnification of 2 times, the RAW data from the first frame memory 104 and the development data from the second frame memory 105 are desired. Is read out (step S804). Next, a peaking signal is generated from the raw data by the peaking signal generation unit 118 (step S805), and the process proceeds to step S806. In step S806, it is determined whether or not the focus assist function is ON. If it is ON, a peaking signal is superimposed on the development data read from the second frame memory 105 (step S807).

  On the other hand, if the enlargement ratio is less than 2 which is a predetermined magnification, RAW data is read from the first frame memory 104 (step S802). Next, the RAW data is developed by the simple Debayer circuit 112, a peaking signal is generated from the RAW data by the peaking signal generation unit 118 (step S803), and the process proceeds to step S806. In step S806, it is determined whether or not the focus assist function is ON. If it is ON, a peaking signal is superimposed on the development data developed by the simple Debayer circuit 112 (step S807).

  After step S807, if necessary, the scaling circuit performs enlargement / reduction processing (step S808), and monitor output is performed (step S809).

  As described above, it is possible to generate a high-quality monitor output video while ensuring peaking performance without causing an increase in circuit scale and memory bandwidth.

<Second Embodiment>
FIG. 9 is a block diagram illustrating a configuration of an imaging apparatus according to the second embodiment. The difference from the first embodiment is that a focus assist function ON / OFF control signal in addition to the enlargement ratio control signal 119 is used as a control signal to the memory controller 111 and the changeover switch 113 for controlling an input signal to the display block 110. 901 is input.

  FIG. 10 is a diagram for explaining data reading from the frame memory and its processing. The memory controller 111, the simple Debayer circuit 112, the changeover switch 113, the peaking superimposing circuit 114, and the peaking signal generation unit 118 are extracted and illustrated. Is.

  When the focus assist function is OFF, it is not necessary to generate a peaking signal. Therefore, as shown in FIG. 10, the memory controller 111 and the changeover switch 113 always provide high quality from the second frame memory 105 regardless of the enlargement ratio. The development data 502 is read out. As a result, when the focus assist function is used when the focus assist function is OFF, it is possible to always output an image that has been corrected for gravity center deviation to the display device 117. When the focus assist function is ON, the same control as in the first embodiment is performed.

FIG. 11 is a flowchart illustrating processing performed by the imaging apparatus according to the second embodiment.
It is determined whether or not the focus assist function is ON (step S1101). If the focus assist function is ON, the process proceeds to step S801. The processing in steps S801 to S809 is the same as that in the first embodiment, and description thereof is omitted here.
On the other hand, if the focus assist function is OFF, the development data is read from the second frame memory 105 in a desired range regardless of the enlargement ratio (step S1102), and the process proceeds to step S808.

Below, the example of the system which detects a high frequency component from RAW data and produces | generates a peaking signal is demonstrated.
An example of a method for generating a peaking signal will be described with reference to FIG. As shown in FIG. 12A, the imaging device is a Bayer-array CMOS sensor in which R, Gr, Gb, and B are repeatedly arranged in a square shape on the sensor in a square shape, and the number of pixels is 3840 ×. 2160. As shown in FIG. 12B, each color pixel data in the Bayer array is distributed. Then, as shown in FIG. 12C, the magnitudes of the high-frequency components filtered for each color pixel are compared in Bayer units, and the largest one is selected and output as peaking information. In the example of FIG. 12C, Gb11 ′ is the largest among the high frequency components of R11 ′, Gr11 ′, Gb11 ′, and B11 ′. As a result of the comparison, one peaking information is selected from the high-frequency components of the Bayer 4 pixels, so that the resolution of the edge information is 1920 × 1080, which is halved both vertically and horizontally.
In addition, although the magnitude of the high frequency component of Bayer4 pixel was compared, you may make it compare the magnitude of the high frequency component of 2 pixels of Gr and Gb among Bayer4 pixels.

An example of a method for generating a peaking signal will be described with reference to FIG. FIG. 13 shows contents corresponding to FIG.
As shown in FIG. 13C, the high-frequency component filtered for each color pixel is calculated in Bayer units.
When the luminance equation Y = eG + fB + gR (e, f, and g are coefficients), for example, here, the NTSC luminance equation Y = 0.487G + 0.114B + 0.299R will be described as an example. As an approximation, R: Gr: Gb : B at a ratio of 3: 3: 3: 1
Y11 ′ = (3 · Gr11 ′ + 3 · Gb11 ′ + B11 ′ + 3 · R11 ′) · α (1)
α: Add as an arbitrary constant, and output the result as peaking information. As a result of the calculation, one peaking information is selected from the high-frequency components of the Bayer 4 pixels, so that the resolution of the edge information is 1920 × 1080, which is halved both vertically and horizontally.
Although the high frequency components of the Bayer 4 pixels are added at a predetermined ratio, the high frequency components of 2 pixels of Gr and Gb among the Bayer 4 pixels may be added. That is,
Y11 ′ = (Gr11 ′ + Gb11 ′) · α (2)
α: Add as an arbitrary constant, and output the result as peaking information.

An example of a method for generating a peaking signal will be described with reference to FIG. FIG. 14 has contents corresponding to FIG.
As shown in FIG. 14 (c), the high frequency components filtered for each color pixel are calculated in Bayer units, compared, and the largest one is selected and output as peaking information. In the example of FIG. 14 (c), (Gr11 ′ + Gb11 ′) and (R11 ′ + B11 ′) and (R11 ′ + B11 ′) in which the directions connecting the computation targets are orthogonal to each other are compared in level, and A11 ′, which is the larger one, is selected. . As a result of the calculation and comparison, one peaking information is selected from the high-frequency components of the Bayer 4 pixels, so that the resolution of the edge information is 1920 × 1080, which is halved both vertically and horizontally.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

  101: Image sensor, 102: Debayer circuit, 103: Signal processing circuit, 104: First frame memory, 105: Second frame memory, 106: Video signal recording unit 1, 107: Video signal recording unit 2, 110: Display block 111: Memory controller 112: Simple Debayer circuit 113: Changeover switch 114: Peaking signal superimposing circuit 115: Scaling circuit 116: Monitor signal processing circuit 117: Display device 118: Peaking signal generator

Claims (5)

An image sensor that converts an optical image of a subject formed through an optical system into a video signal;
First recording means for recording RAW data of a video signal obtained by the imaging device;
Second recording means for recording development data developed by signal processing from the RAW data;
A peaking signal generator for generating a peaking signal from the RAW data recorded in the first recording means;
As the video signal for generating the monitor output video, the RAW data recorded in the first recording unit is used according to the magnification rate of the focus assist function, or the development data recorded in the second recording unit is used. An imaging apparatus comprising: control means for switching whether to use.   The control means uses the development data recorded in the second recording means when the magnification ratio of the focus assist function is equal to or greater than a predetermined magnification, and applies the development data recorded in the second recording means to the first recording means when the magnification is less than the predetermined magnification. The imaging apparatus according to claim 1, wherein recorded RAW data is used. Only when the focus assist function is ON, switching by the control means is performed,
3. The imaging apparatus according to claim 1, wherein when the focus assist function is OFF, the development data recorded in the second recording unit is used as a video signal for generating a monitor output video. An image sensor that converts an optical image of a subject formed through an optical system into a video signal;
First recording means for recording RAW data of a video signal obtained by the imaging device;
A control method of an imaging apparatus comprising: a second recording unit that records development data developed from the RAW data by signal processing;
Generating a peaking signal from RAW data recorded in the first recording means;
As the video signal for generating the monitor output video, the RAW data recorded in the first recording unit is used according to the magnification rate of the focus assist function, or the development data recorded in the second recording unit is used. And a step of switching whether to use the imaging apparatus. An image sensor that converts an optical image of a subject formed through an optical system into a video signal;
First recording means for recording RAW data of a video signal obtained by the imaging device;
A program for controlling an imaging apparatus comprising: a second recording unit that records development data developed from the RAW data by signal processing;
Processing for generating a peaking signal from RAW data recorded in the first recording means;
As the video signal for generating the monitor output video, the RAW data recorded in the first recording unit is used according to the magnification rate of the focus assist function, or the development data recorded in the second recording unit is used. A program for causing a computer to execute a process of switching whether to use.

Images