JP2017139646A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To ensure real time property of scene discrimination which is conventionally damaged because estimation of imaging scene consumes time when a high-level algorithm such as a convolution neural network is boarded.SOLUTION: An imaging apparatus comprises: a photometric sensor for performing photometry of a field; estimation means which estimates the classification of a subject in the field on the basis of photometric data outputted by the photometric sensor; setting means which sets imaging conditions on the basis of the estimated classification; and imaging means for imaging the subject under the set imaging conditions.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a photographing apparatus.

  An algorithm for analyzing image data using a deep learning technique is known. For example, Patent Document 1 describes a method for estimating the classification of image data using a convolutional neural network which is a kind of deep learning.

  In the technical field of photographing devices such as digital cameras, this kind of method is used to analyze real-time live view (live view) and recorded image data to estimate classification (shooting scene) and to estimate the shooting scene Techniques for photographing a subject under conditions according to the conditions are known.

Japanese Patent Laying-Open No. 2015-32308

  In the imaging device, basic processing (various processing such as imaging, storage, live view display, etc.) needs to be performed on the captured image. It is done. For this reason, when a sophisticated algorithm such as a convolutional neural network is implemented, it takes a long time to estimate a shooting scene, and the real-time property of scene discrimination is impaired.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a photographing apparatus suitable for executing a photographing scene estimation process at high speed.

  An imaging apparatus according to an embodiment of the present invention includes a photometric sensor that performs photometry of an object scene, and an estimation unit that estimates a classification of a subject in the object scene based on photometric data output from the photometry sensor. Setting means for setting shooting conditions based on the classification, and imaging means for imaging a subject under the set shooting conditions.

  Further, an imaging device according to an embodiment of the present invention includes a photometric sensor that measures a scene, and an estimation unit that estimates a classification of a subject in the scene based on photometric data output from the photometric sensor; A setting unit configured to set an image parameter based on the estimated classification; an imaging unit configured to capture an image of the subject; and a processing unit configured to process image data output from the imaging unit using the set image parameter.

  In one embodiment of the present invention, the estimation means may be configured to execute subject classification estimation processing in a pipeline.

  In one embodiment of the present invention, the estimation means may be configured to execute subject classification estimation processing in a time-sharing manner.

  In one embodiment of the present invention, the estimation means may be configured to estimate the subject classification using a convolutional neural network.

  In one embodiment of the present invention, the estimation means obtains a confidence vector having each classification candidate of the subject as an element using a convolutional neural network, and causes the elements of the obtained confidence vector to interact with each other. A configuration may be adopted in which the certainty factor vector is corrected by performing calculation, and the classification of the subject is estimated based on the corrected certainty factor vector.

  According to an embodiment of the present invention, a photographing apparatus suitable for executing a photographing scene estimation process at high speed is provided.

It is a block diagram which shows the structure of the imaging device which concerns on one Embodiment of this invention. It is a figure which shows the 1st operation | movement flow performed in one Embodiment of this invention. It is a figure which shows the execution timing of the estimation process by the image classification part with which the imaging device which concerns on one Embodiment of this invention is equipped. It is a figure which shows the 2nd operation | movement flow performed in one Embodiment of this invention.

  An imaging apparatus according to an embodiment of the present invention will be described below with reference to the drawings. In the following, a digital single-lens reflex camera equipped with a quick return type mirror will be described as an embodiment of the present invention.

[Configuration of the photographing apparatus 1]
FIG. 1 is a block diagram showing a configuration of a photographing apparatus 1 according to an embodiment of the present invention. As shown in FIG. 1, the photographing apparatus 1 includes a photographing lens 101, a diaphragm mechanism 102, a main mirror 103, a shutter mechanism 104, an image sensor 105, an image sensor driver 106, an AE (Automatic Exposure) sensor 107, and an AE sensor. Driver 108, image conversion unit 109, shooting condition setting unit 110, image parameter setting unit 111, aperture control unit 112, shutter control unit 113, mechanism control unit 114, image classification unit 115, image storage unit 116, image display unit 117, and An operation unit 118 is provided. Although the photographing lens 101 has a plurality of lenses, it is shown as a single lens for convenience in FIG. Further, in FIG. 1, connection between blocks is omitted as appropriate for the sake of avoiding the complexity of the drawing.

  A luminous flux from the subject (subject luminous flux) is reflected by the main mirror 103 via the photographing lens 101 and the diaphragm mechanism 102 toward a finder optical system (not shown). The subject luminous flux is reflected on each reflecting surface of the pentaprism of the finder optical system to become an erect image and is incident on the eyepiece. The subject luminous flux is re-imaged by the eyepiece into a virtual image suitable for the user's observation. The user can observe the subject image (virtual image) re-formed by the eyepiece lens by looking through the viewfinder window.

  The operation unit 118 includes various switches necessary for the user to operate the photographing apparatus 1, such as a power switch, a release switch, and a photographing mode switch. When the user operates the power switch, power is supplied from the battery (not shown) to the various circuits of the photographing apparatus 1 through the power line.

  A part of the pentaprism is a half mirror region. Part of the subject light beam reflected by each reflecting surface of the pentaprism is received by the AE sensor 107 via the half mirror region.

  The AE sensor 107 is driven by an AE sensor driver 108 and divides the object field into a plurality of photometric areas, and measures the light. For example, the plurality of photometric areas (pixels) are arranged in a matrix. A two-dimensional sensor (for example, CCD (Charge Coupled Device)) equipped with a color filter.

  The AE sensor 107 takes in the subject luminous flux and outputs photometric data. Specifically, the AE sensor 107 accumulates an optical image formed by each pixel on the photometric surface as a charge corresponding to the amount of light, and generates and outputs photometric data having color information. Photometric data of each pixel output from the AE sensor 107 is input to the image conversion unit 109 via the AE sensor driver 108.

[First operation flow]
In the present embodiment, the classification of the subject in the object scene (in other words, the shooting scene) is estimated based on the photometric data of each pixel output from the AE sensor 107. Examples of the shooting scene include “flower”, “dog”, “night view”, “cooking”, “sea”, and the like.

  FIG. 2 shows a first operation flow. The first operation flow shown in FIG. 2 is a flowchart showing a series of operations from estimating a shooting scene to shooting and storing / displaying shot image data. The first operation flow illustrated in FIG. 2 is started when, for example, the photometric data of each pixel output from the AE sensor 107 is input to the image conversion unit 109.

[S11 in FIG. 2 (Conversion of Image Signal Output from AE Sensor 107)]
In this processing step S11, the image conversion unit 109 performs a photometric calculation and acquires a photometric result. Specifically, the image conversion unit 109 performs predetermined signal processing on the photometric data (RAW format image signal) of each pixel, so that the information has a low resolution but can determine the shape and color of the subject. Is converted into an image signal in RGB format (or YCbCr format).

[S12 in FIG. 2 (Shooting Scene Estimation)]
In this processing step S12, the image classification unit 115 estimates the shooting scene of the scene captured by the AE sensor 107 based on the RGB (or YCbCr format) image signal input from the image conversion unit 109. Is done.

  In the present embodiment, a convolutional neural network is used to estimate a shooting scene. In the convolutional neural network implemented in the image classification unit 115, an object in the object scene captured by the AE sensor 107 is abstracted by repeating a convolution layer and a pooling layer. Abstracted image data is generated. The generated abstract image data is converted into one-dimensional vector data and input to a full connection layer.

ｉ ：一次元に変換された入力ベクトルデータの各要素を示す値
ｊ ：対象の撮影シーン候補を認識する場合とそれ以外を認識する場合とを区別するための値（例えば前者の場合ｊ＝１で後者の場合ｊ＝２）
ｘ_ｉ ：入力ベクトルデータの各要素ｉの値
ｙ_ｉ ：入力ベクトルデータの各要素ｉについての積和演算結果
ｂ_ｉ ：事前の学習結果から与えられる値であって、入力ベクトルデータの各要素ｉの調整値
ｗ_ｉｊ :事前の学習結果から与えられる値であって、入力ベクトルデータの各要素ｉの係数（対象の撮影シーン候補を認識する場合の係数ｗ_ｉ１と、それ以外を認識する場合の係数ｗ_ｉ２） In the fully connected layer, a set of values (adjustment value b _i and coefficient w _ij ) is given for each shooting scene candidate from the learning result, and a product-sum operation using the following equation is performed for each shooting scene candidate.
(Multiply-accumulate expression)

i: Value indicating each element of input vector data converted to one dimension j: Value for distinguishing between the case of recognizing a target photographing scene candidate and the case of recognizing the other (for example, j = 1 in the former case) In the latter case, j = 2)
x _i : the value of each element i of the input vector data y _i : the product-sum operation result b _i for each element i of the input vector data b _i : the value given from the previous learning result, and each element i of the input vector data Adjustment value w _ij : A value given from a prior learning result, which is a coefficient of each element i of the input vector data (coefficient w _i1 for recognizing the target photographing scene candidate and other cases for recognizing the other Coefficient w _i2 )

The product-sum operation result y _i for each shooting scene candidate is normalized in the softmax layer, which is the final layer. The image classification unit 115 estimates a photographic scene based on the value normalized in the softmax layer.

Note that the user can estimate by the image classification unit 115 by rewriting the spatial filter and value set (adjustment value b _i and coefficient w _ij ) of the convolution layer with data suitable for another shooting scene candidate. You can change the setting of the shooting scene. The user can also update the spatial filter and value set (adjustment value b _i and coefficient w _ij ) of the convolution layer for existing shooting scene candidates.

ｘ ：確信度ベクトル（softmax層から出力されるベクトル）
ａ_ｉｊ : 確信度ベクトル相互作用マトリックス
ｙ ：確信度ベクトル（最終値） In addition, a confidence vector having each photographing scene candidate of the subject as an element is obtained from the output of the softmax layer. In the present embodiment, in order to increase the finally obtained certainty factor, a matrix operation for causing the elements of the certainty factor vector to interact with each other may be performed (see, for example, JP-A-4-1870). An example of matrix calculation is shown below.
(Matrix calculation example)

x: confidence vector (vector output from softmax layer)
a _ij : Certainty vector interaction matrix y: Certainty vector (final value)

  For example, the certainty factor vectors of “flower”, “night view”, “dish”, and “sea” output from the softmax layer are “0”, “1”, “2”, and “3”, respectively. Consider a case. In this case, when the above-described matrix calculation is performed, the certainty factor of “flower” is handled with a larger value in estimating the shooting scene. Note that the photographic scene may be estimated with higher accuracy by further improving the accuracy of the certainty factor in consideration of the image magnification and the focal length obtained from the focal length.

  In general, a huge amount of calculation is required to execute an advanced algorithm such as a convolutional neural network. For this reason, there is a possibility that resources are occupied for execution of the processing, which hinders execution of other processing. Therefore, in this embodiment, the shooting scene estimation process is executed in a time-sharing manner. FIG. 3 is a diagram illustrating the execution timing of the estimation process performed by the image classification unit 115.

  As shown in FIG. 3, the capturing process and the photometric calculation by the AE sensor 107 are executed every frame. The shooting scene estimation processing by the image classification unit 115 is time-shared using a period from the photometry calculation by the AE sensor 107 to the capture processing of the next frame (hereinafter referred to as “resource idle period” for convenience of explanation). Executed.

  In the example of FIG. 3, when the first convolution (“convolution 1” in FIG. 3) and the first pooling (“pooling 1” in FIG. 3) are performed in the resource free period, the second time in the resource free period of the next frame. ("Convolution 2" in FIG. 3) and the second pooling ("pooling 1" in FIG. 3) are executed, and the convolution in FIG. Is executed. Next, a product-sum operation (“FC (Full Connection)” in FIG. 3) and a normalization process (“Softmax” in FIG. 3) are executed in the resource empty period of the next frame, and a shooting scene is acquired in the resource empty period of the next frame. (The “classification” in FIG. 3)) is executed. In the present embodiment, these series of processes are repeatedly executed at a cycle of 5 frames. That is, in the present embodiment, the shooting scene is estimated at a cycle of 5 frames.

  By adopting the time-sharing process, it is possible to avoid a situation in which the resources of the photographing apparatus 1 are occupied in the photographing scene estimation process. In addition, by using photometric data that does not require processing related to display, the resource vacant period is longer than when using live view or recorded image data, so the shooting scene can be made in a short time (higher speed than before). Can be estimated. Therefore, even when an advanced algorithm such as a convolutional neural network is implemented, the real-time property of scene discrimination is ensured.

  Further, the shooting scene estimation process may be executed in a pipeline instead of the time division process or in addition to the time division process. Similarly, when pipeline processing is employed, a situation in which the resources of the photographing apparatus 1 are occupied by the estimation processing can be avoided and real-time performance of scene determination is ensured.

[S13 in FIG. 2 (Setting of Shooting Conditions)]
In this processing step S13, the shooting condition setting unit 110 sets shooting conditions based on the shooting scene estimated in the processing step S12 (shooting scene estimation). The shooting conditions set here include, for example, shutter speed, aperture value, ISO sensitivity, countermeasures to be taken before shooting, and the like.

  For example, consider a case where “cooking” is estimated as a shooting scene. In this case, the entire image may be darkened due to the influence of the white dish being highlighted. Therefore, the shutter speed, aperture value, and ISO sensitivity are set so that the entire image does not become dark. When “flower” is estimated as the shooting scene, autofocus is set so that the background is not in focus.

  Further, the action to be taken before photographing means dealing with a matter that is difficult to be corrected or reflected in the image after photographing. For example, consider a case where “blue sky” is estimated as a shooting scene. In this case, the countermeasure to be taken before photographing is to give a correction value (under correction) to the light control in order to prevent the cloud from being blown out.

[S14 in FIG. 2 (shooting with setting conditions)]
When the release switch is pressed by the user, the aperture control unit 112 controls the aperture of the aperture mechanism 102 in accordance with the aperture value set in the processing step 13 (setting of shooting conditions). Further, the shutter mechanism 104 is driven and controlled, and the main mirror 103 is returned quickly. Specifically, the shutter mechanism 104 is a focal plane shutter, for example, and is driven and controlled by the shutter control unit 113 at the shutter speed set in the processing step 13 (setting of shooting conditions). The main mirror 103 is raised by the mechanism control unit 114 and retracted from the optical path parallel to the optical axis of the photographing lens 101 only during the period immediately before the start of the front curtain travel of the shutter mechanism 104 and immediately after the end of the rear curtain travel.

  The image sensor 105 is installed at the subsequent stage of the shutter mechanism 104. Therefore, the subject light flux that has passed through the shutter mechanism 104 is imaged on the imaging surface of the image sensor 105. The image sensor 105 is an image forming image sensor driven by an image sensor driver 106. The image sensor 105 accumulates an optical image formed by each pixel on the imaging surface as a charge corresponding to the amount of light, and performs amplification processing according to the ISO sensitivity set in the processing step 13 (setting of shooting conditions). , Generate and output an image signal. The image sensor 105 may be a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[S15 in FIG. 2 (Conversion of Image Signal Output from Image Sensor 105)]
In this processing step S15, the image conversion unit 109 converts the RAW format image signal input from the image sensor 105 into an RGB format (or YCbCr format) image signal.

[S16 in FIG. 2 (Storing Captured Image Data and Displaying Captured Image)]
In this processing step S16, the image signal (captured image data) in RGB format (or YCbCr format) converted in processing step S15 (conversion of the image signal output from the image sensor 105) is stored in the image storage unit 116. The The image storage unit 116 may store the RAW format image signal input from the image sensor 105 as it is.

  In this processing step S16, the RGB (or YCbCr format) image signal converted in the processing step S15 (conversion of the image signal output from the image sensor 105) is converted into a video signal of a predetermined format, The image is displayed on the display screen of the image display unit 117 provided on the back surface of the photographing apparatus 1. Thereby, the user can visually recognize the live view through the display screen of the image display unit 117.

[Second operation flow]
FIG. 4 shows a second operation flow. The second operation flow shown in FIG. 4 is a flow chart showing a series of operations from taking a picture under arbitrarily set conditions and storing / displaying the taken image data. The second operation flow illustrated in FIG. 4 is started when, for example, photometric data of each pixel output from the AE sensor 107 is input to the image conversion unit 109.

[S21 to S22 in FIG. 4]
Since the processing steps S21 to S22 are substantially the same as the processing steps S11 to S12 of FIG.

[S23 in FIG. 4 (setting of shooting conditions)]
In this processing step S23, the photographing condition setting unit 110 sets arbitrary photographing conditions (for example, according to an operation input by the user).

[S24 in FIG. 4 (shooting with setting conditions)]
The processing step S24 is substantially the same as the processing step S14 in FIG.

[S25 in FIG. 4 (Conversion of Image Signal Output from Image Sensor 105)]
In this processing step S25, the image to be applied to the RAW image signal input from the image sensor 105 based on the shooting scene estimated in the processing step S22 (shooting scene estimation) by the image parameter setting unit 111. The parameter is set.

  For example, when “flower (colorful)” is estimated as a shooting scene, a parameter with a little suppressed saturation is set. In addition, for example, when “natural scenery (scenery in which blue sky and green are mixed)” is estimated as a shooting scene, the parameters are set so that the contrast is effective.

  Next, in this processing step S25, the image conversion unit 109 uses the parameters set by the image parameter setting unit 111, and the RAW image signal input from the image sensor 105 is in RGB format (or YCbCr format). It is converted into an image signal (photographed image data).

  The image parameters that can be set in this processing step S25 include at least various parameters that can be reflected when converting from the RAW format to the RGB format (or YCbCr format). Illustratively, the image parameters that can be set in this processing step S25 include not only saturation and contrast but also sharpness settings and special effect settings.

[S26 in FIG. 4 (Storing Captured Image Data and Display of Captured Image)]
The processing step S26 is substantially the same as the processing step S16 in FIG.

  The above is the description of the exemplary embodiments of the present invention. Embodiments of the present invention are not limited to those described above, and various modifications are possible within the scope of the technical idea of the present invention. For example, the embodiment of the present invention also includes contents appropriately combined with embodiments or the like clearly shown in the specification or obvious embodiments.

DESCRIPTION OF SYMBOLS 1 Shooting device 101 Shooting lens 102 Aperture mechanism 103 Main mirror 104 Shutter mechanism 105 Image sensor 106 Image sensor driver 107 AE sensor 108 AE sensor driver 109 Image conversion unit 110 Shooting condition setting unit 111 Image parameter setting unit 112 Aperture control unit 113 Shutter control unit 114 Mechanism control unit 115 Image classification unit 116 Image storage unit 117 Image display unit 118 Operation unit

Claims (6)

A photometric sensor for metering the object field;
Estimating means for estimating a classification of a subject in the object field based on photometric data output from the photometric sensor;
Setting means for setting shooting conditions based on the estimated classification;
Imaging means for imaging the subject under the set imaging conditions;
Comprising
Shooting device. A photometric sensor for metering the object field;
Estimating means for estimating a classification of a subject in the object field based on photometric data output from the photometric sensor;
Setting means for setting image parameters based on the estimated classification;
Imaging means for imaging the subject;
Processing means for processing image data output from the imaging means using the set image parameters;
Comprising
Shooting device. The estimation means includes
Performing the subject classification estimation process in a pipeline;
The imaging device according to claim 1 or 2. The estimation means includes
Performing the subject classification estimation process in a time-sharing manner;
The imaging device according to any one of claims 1 to 3. The estimation means includes
Estimating the classification of the subject using a convolutional neural network;
The imaging device according to any one of claims 1 to 4. The estimation means includes
Using a convolutional neural network to obtain a certainty vector having each classification candidate of the subject as an element;
Correcting the certainty factor vector by performing a matrix operation that causes the elements of the obtained certainty factor vector to interact with each other;
Estimating the classification of the subject based on the corrected confidence vector;
The imaging device according to any one of claims 1 to 5.

Images