JP2019153972A - Imaging apparatus, control method thereof, and program - Google Patents

Abstract

To ensure visibility in an image while reducing a possibility in which the bit rate is excessively increased by a WDR function.SOLUTION: An imaging apparatus includes an imaging unit that images a subject under a plurality of exposure conditions and generates a plurality of pieces of image data, encoding means for encoding the image data, determination means for determining the magnitude of the bit rate of the encoded image data, and combining means for combining the plurality of pieces of image data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging device, an imaging device control method, and a program.

  One of the functions of an imaging apparatus is to manually switch between enabling and disabling a wide dynamic range (hereinafter referred to as WDR) function according to the bit rate. This WDR function can be realized by synthesizing images with different exposure amounts. When the WDR function is enabled, whiteout and blackout can be improved and visibility can be improved. However, when the WDR function is enabled, the visibility is improved. However, since the area to be visualized increases, the amount of data increases, and the bit rate for moving images tends to increase. Further, when recording, there is a problem that the recording capacity is pressed and cost is increased.

  Therefore, a technique for determining the brightness of an image at the time of image synthesis or encoding in consideration of the data amount has been proposed. In Patent Document 1, first, reference image data is generated from image data obtained by imaging a subject, and on the other hand, the amount of image data is reduced to generate composite image data having a smaller data amount than the reference image data. Then, the image data is synthesized using the two types of generated image data. In Patent Document 2, a plurality of frames of image data are captured and encoded with a plurality of exposures. At that time, a reference frame for a reference frame to be encoded in inter-frame predictive encoding is determined according to the exposure of each frame. Then, the brightness of the determined reference source frame and reference destination frame are matched.

JP, 2015-162851, A Japanese Patent Laid-Open No. 2015-23498

  However, with the technique disclosed in Patent Document 1, if the reference image data increases, the bit rate at the time of moving image shooting increases. Further, the technique disclosed in Patent Document 2 relates to frame reference for encoding and does not perform image synthesis, and therefore cannot be expected to improve visibility.

  In view of the above-described problems, the present invention provides a technique that can ensure visibility in an image while reducing the possibility that the bit rate is excessively increased by the WDR function.

The present invention for solving the above problems is an imaging apparatus,
An imaging unit that images a subject under a plurality of exposure conditions and generates a plurality of image data;
Encoding means for encoding image data;
Determination means for determining the magnitude of the bit rate of the encoded image data;
Combining means for combining the plurality of image data,
The determination unit determines whether the bit rate of the first image data of the plurality of image data is smaller than a first threshold;
When the determination unit determines that the bit rate is smaller than the first threshold value, the combining unit calculates the first image data and the second image data of the plurality of image data. A third image data is generated by combining, and the encoding means encodes the third image data.

  According to the present invention, it is possible to ensure visibility in an image while reducing the possibility that the bit rate is excessively increased by the WDR function.

1 is a block diagram illustrating an example of a functional configuration of an imaging apparatus corresponding to Embodiment 1 of the invention. The flowchart which shows an example of the process corresponding to Embodiment 1 of invention. The block diagram which describes an example of the function structure of the imaging device corresponding to Embodiment 2 of invention. The flowchart which shows an example of the process corresponding to Embodiment 2 of invention. The block diagram which shows an example of the hardware constitutions of the imaging device corresponding to embodiment of invention.

  Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

[Embodiment 1]
The configuration of the imaging apparatus corresponding to the first embodiment of the invention will be described below. The imaging apparatus corresponding to the present embodiment can prevent the bit rate from being excessively increased by switching the wide dynamic range (WDR) function between valid and invalid according to the magnitude of the bit rate of the captured image.

  FIG. 1A is a block diagram illustrating an example of a functional configuration of the imaging apparatus 100 corresponding to the embodiment of the invention. The imaging apparatus 100 corresponding to the present embodiment operates so as to prevent the bit rate from being excessively increased by switching between enabling and disabling the WDR function. First, the imaging apparatus 100 is configured to include an imaging unit 101, an encoding unit 102, a measurement unit 103, a synthesis setting unit 104, a synthesis unit 105, and an image processing unit 106. FIG. 1B is a diagram illustrating an example of a hardware configuration of the imaging unit 101. The imaging unit 101 includes an imaging lens 111, an aperture and shutter 112, an AE processing unit 113, a focus lens 114, an AF processing unit 115, an imaging element 116, an A / D conversion unit 117, a development processing unit 118, and the like. .

  The configurations illustrated in FIGS. 1A and 1B are merely examples, and components other than those illustrated in FIG. 1A may be added. Further, in the imaging apparatus 100 of FIG. 1A, each block may be configured by hardware using a dedicated logic circuit or a memory, except for physical devices such as an imaging element, a lens, a diaphragm, and a shutter. . Alternatively, the processing program stored in the memory may be configured by software by a computer such as a CPU executing the processing program. Hereinafter, components and functions of the imaging apparatus 100 will be described.

  The imaging unit 101 captures a subject to generate image data, and outputs the generated image data to the encoding unit 102 and the synthesis unit 105. The aperture and shutter 112 controls the amount of incident light and the charge accumulation time of the incident light (reflected light from the subject) incident on the image sensor 116 via the photographing lens 111 according to an instruction from the AE processing unit 113. To do. The AE processing unit 113 controls the operation of the aperture and shutter 112 and also controls the A / D conversion unit 117. The focus lens 114 focuses on the light receiving surface of the image sensor 116 and forms an optical image in accordance with a control signal from the AF processing unit 115.

  The image sensor 116 converts an optical image formed on the light receiving surface into an electrical signal by a photoelectric conversion unit such as a CCD element or a CMOS element, and outputs the electrical signal to the A / D conversion unit 117. The A / D converter 117 converts the received electrical signal (analog signal) into a digital signal. The A / D converter 117 can include a CDS circuit that removes noise from the received electrical signal, and a nonlinear amplifier circuit that performs nonlinear amplification before converting the received electrical signal into a digital signal.

  The development processing unit 118 performs predetermined development processing on the digital image signal output from the A / D conversion unit 117. The development processing can include, for example, processing such as debayer processing, white balance processing, gradation conversion processing, edge enhancement correction processing, scratch correction, noise removal, enlargement / reduction processing, and color conversion into the YCbCr format.

  In the present embodiment, the imaging unit 101 continuously captures images under different exposure conditions under the control of the AE processing unit 113 to generate a plurality of image data. The AE processing unit 113 detects the subject brightness based on the image data acquired by the image sensor 116, and calculates a shutter speed, an aperture value, and the like that are predetermined first exposure conditions based on the subject brightness. Of the plurality of image data, an image captured under the first exposure condition is referred to as first image data. Among the plurality of image data, the other second image data is image data captured by the AE processing unit 113 by changing the exposure amount from the first exposure condition. These are imaged, for example, by changing the exposure condition sequentially to −1 stage and +1 stage with respect to the first exposure condition (in this embodiment, the exposure conditions after the change are collectively referred to as the second exposure condition). Exposure conditions). The imaging unit 101 outputs the first image data to the encoding unit 102, and outputs the first image data and the second image data to the synthesis unit 105.

  The encoding unit 102 generates image data by compressing and encoding the image data output from the imaging unit 101 or the image data output from the combining unit 105. The image compression method can be based on standards such as H.264, H.265, and MPEG, for example. Furthermore, image data in any format including mp4 or avi format may be generated.

  The measuring unit 103 measures the bit rate of the image data encoded by the encoding unit 102 for each frame, and outputs the obtained bit rate to the synthesis setting unit 104. The synthesis setting unit 104 determines whether the bit rate measured by the measurement unit 103 is equal to or higher than a predetermined first threshold (Th1), and switches the state setting of the synthesis unit 105. If the bit rate value of the image data is equal to or greater than the first threshold Th1, the synthesis setting unit 104 transmits a control signal to the synthesis unit 105, and invalidates the synthesis in the synthesis unit 105. On the other hand, if the bit rate value is less than the first threshold Th1, the synthesis setting unit 104 transmits a control signal to the synthesis unit 105, and validates the synthesis in the synthesis unit 105. The synthesis setting unit 104 also outputs the control signal to the encoding unit 102. The encoding unit 102 switches image data to be output according to the value of the control signal.

  The synthesizing unit 105 switches the synthesizing process for the image data in accordance with the control signal transmitted from the synthesizing setting unit 104. When a control signal for enabling composition is received from the composition setting unit 104, composition processing is performed to generate the third image data by combining the first image data and the second image data acquired with different exposure amounts. The third image data obtained as a result of the synthesis is output to the encoding unit 102. On the other hand, when the control signal for invalidating the synthesis is received from the synthesis setting unit 104, the synthesis unit 105 does not perform the synthesis process of the image data received from the imaging unit 101. In this case, for the image data, the encoding result of the first image data previously performed by the encoding unit 102 is output.

  Next, a flow of processing executed by the imaging apparatus 100 corresponding to the present embodiment will be described with reference to the flowchart of FIG. FIG. 2 is a flowchart illustrating an example of processing executed by the imaging apparatus 100 corresponding to the embodiment of the invention. The processing corresponding to the flowchart can be realized, for example, by executing a program (stored in a ROM or the like) corresponding to one or more processors functioning as processing blocks.

  In S201, the imaging unit 101 performs an imaging process. At this time, the imaging unit 101 generates a plurality of image data obtained by imaging under a plurality of exposure conditions including the first exposure condition. In subsequent S202, the encoding unit 102 performs the first image obtained by imaging under the first exposure condition among the plurality of image data obtained by imaging under the plurality of exposure conditions of the imaging unit 101. Encode data. In subsequent S203, the measurement unit 103 measures the bit rate (Br) of the encoded image data based on the encoding processing result, and determines the bit rate obtained by the synthesis setting unit 104 and the first threshold value. By comparing, the magnitude of the bit rate is determined. If the bit rate Br is equal to or higher than the first threshold (“YES” in S204), the process proceeds to S205. On the other hand, if the bit rate Br is less than the first threshold (“NO” in S204), the process proceeds to S206.

  In S205, the synthesis setting unit 104 outputs the WDR OFF control signal to the synthesis unit 105 and the encoding unit 102, and does not perform the synthesis process. In S209, the encoding unit 102 encodes the first image data in S202. Output the result. On the other hand, in S206, the synthesis setting unit 104 outputs a WDR ON control signal to the synthesis unit 105 and the encoding unit 102, and executes the synthesis process in S207. In this combining process, the first image data and the second image data generated under a plurality of exposure conditions are aligned by partial portions and combined to generate third image data. Since the synthesis process is a known technique, details are omitted. By this combining process, an image having a wider dynamic range than that of a single shot can be obtained. In subsequent S208, the encoding unit 102 encodes the third image data generated in the synthesizing unit 105, and outputs the encoded result in S209.

  In the above embodiment, the first threshold value Th1 is used when determining the magnitude of the bit rate of the image captured under the first exposure condition. For example, the first threshold value may be the maximum possible bit. It can be half the rate. For example, when the maximum value of the bit rate obtained under a predetermined condition is 50 Mbps (bit per second), the first threshold Th1 can be set to 25 Mbps.

  In the above embodiment, the magnitude of the bit rate obtained by actually encoding the first image data to be processed is determined. However, the embodiment of the invention is not limited to this. For example, when an average value of bit rates of a plurality of image data encoded before the first image data to be processed is obtained, and the first image data is encoded based on the average value The bit rate may be estimated. In the estimation method, for example, the average value is corrected in the increasing direction or the decreasing direction according to the tendency of the change in the bit rate of the encoded image data, and the corrected value is the estimated value of the first image data bit rate. It is good. Alternatively, the above process may be executed using the average value of the bit rate instead of the bit rate of the first image data.

  As described above, in the present embodiment, when the bit rate of the image data to be processed is greater than or equal to a predetermined value based on the encoding result, the WDR function is controlled to be invalidated, thereby generating the generated moving image bits. It is possible to prevent the rate from increasing excessively. At the same time, when the bit rate is smaller than the predetermined value, the WDR function can be validated to improve the image visibility. Therefore, according to the present embodiment, it is possible to adaptively improve the visibility of an image while preventing an increase in the bit rate. In addition, according to the present invention, since the state of the WDR function corresponding to the bit rate of each image data is set, it is possible to transmit an image without causing a frame drop even when the bandwidth is insufficient. Is possible.

[Embodiment 2]
Next, a second embodiment of the invention will be described. In this embodiment, even when the WDR function is turned off, a predetermined correction process is performed under a certain condition in order to improve visibility.

  With reference to FIG. 3, the configuration of the imaging apparatus 300 corresponding to the present embodiment will be described. The basic configuration of the imaging apparatus 300 is almost the same as that of the imaging apparatus 100 shown in FIG. Therefore, the same reference numerals are used for components that perform substantially the same operation, and descriptions thereof are omitted. The imaging apparatus 300 of the present embodiment is different from the configuration of the imaging apparatus 100 in that a determination unit 301 is included instead of the synthesis setting unit 104 and an image processing unit 302 is included instead of the synthesis unit 105.

  In addition to the function of the combination setting unit 104, the determination unit 301 further determines whether or not the bit rate value measured by the measurement unit 103 is equal to or greater than a predetermined second threshold (Th2), and further determines the histogram. Processing is performed, and execution of luminance value correction processing in the image processing unit 302 is controlled according to these determination results. If the bit rate value of the image data is greater than or equal to the second threshold Th2, the determination unit 301 transmits a control signal to the image processing unit 302 to invalidate the luminance value correction processing in the image processing unit 302. On the other hand, if the bit rate value is less than the second threshold Th2, the determination unit 301 further performs a histogram determination process to determine whether or not the ratio of the low-luminance area in the image data is greater than a predetermined ratio. To do. Here, the low luminance area refers to an area where the luminance value of each pixel is less than a predetermined luminance value. If the ratio of the low luminance area in the image data is larger than the predetermined ratio, a control signal is transmitted to the image processing unit 302, and the luminance value correction processing in the image processing unit 302 is validated.

  In addition to the function of the synthesizing unit 105, the image processing unit 302 performs a process of correcting the luminance value of the first image data. This luminance value correction processing includes, for example, dodging correction. Dodge correction refers to correction processing for raising the luminance level of a low luminance region to a predetermined luminance value in a processing target image.

  In the present embodiment, the image processing unit 302 executes either the synthesis process described in the first embodiment or the brightness correction process. The encoding unit 102 is input with either the third image data after the synthesis process output from the image processing unit 302 or the fourth image data obtained by performing luminance correction on the first image data. . The encoding unit 102 encodes and outputs any of the input image data.

  Next, a flow of processing executed by the imaging apparatus 300 corresponding to the present embodiment will be described with reference to the flowchart of FIG. FIG. 4 is a flowchart illustrating an example of processing executed by the imaging apparatus 300 corresponding to the embodiment of the invention. The process corresponding to the flowchart of FIG. 4 is partially the same as the process of the flowchart of FIG. Therefore, the processing steps corresponding to the flowchart of FIG. 2 are assigned the same reference numerals and description thereof is omitted. Specifically, the processing from S201 to S209 is the same as that in FIG.

  In the first embodiment, when it is determined in the determination result in S203 that the bit rate is equal to or higher than the first threshold Th1 (“YES” in S204), the process proceeds to S205 to perform WDR OFF setting. On the other hand, in this embodiment, the WDR OFF setting is not performed immediately, but the bit rate Br determined by the determination unit 301 in S203 is compared with the second threshold Th2 in S401. Here, the second threshold Th2 is a value larger than the first threshold Th1. If the bit rate Br is equal to or higher than the second threshold (“YES” in S402), the process proceeds to S205. In S205, as in the first embodiment, the determination unit 301 performs WDR OFF setting, and in S209, the encoding unit 102 outputs the encoding result of the first image data in S202. On the other hand, if the bit rate Br is less than the second threshold (“NO” in S402), the process proceeds to S403.

  In S403, the determination unit 301 further performs a histogram determination process. Specifically, the ratio of the low luminance area in the first image data is determined, and it is determined whether the ratio of the low luminance area is larger than a predetermined ratio. The predetermined ratio can be, for example, 30%, 40%, or 50%, but is not limited to these. As a result of the histogram determination process, when it is determined that the low-luminance area is greater than the predetermined ratio (“YES” in S404), the process proceeds to S405. On the other hand, when it is determined that the low luminance area is equal to or less than the predetermined ratio (“NO” in S404), the process proceeds to S205.

  In step S <b> 405, the image processing unit 302 performs the luminance value correction process by the above-described dodging correction on the first image data, and raises the luminance level of the low luminance area. The image processing unit 302 outputs the fourth image data obtained as a processing result of the dodging correction to the encoding unit 102. In subsequent S406, the encoding unit 102 executes encoding processing of the fourth image data obtained from the image processing unit 302, and outputs it as an encoding result in S209.

  In the above embodiment, the second threshold value Th2 is, for example, 35 Mbps when the maximum value of the bit rate obtained under a predetermined condition is 50 Mbps (bit per second) and the first threshold value Th1 is 25 Mbps. It can be. In the present embodiment, the case where the image processing unit 302 mainly performs luminance value correction has been described. However, other than this, for example, image processing such as sharpness processing and noise reduction can be performed.

  In the above-described embodiment, the ratio of the low-brightness area is determined in the histogram determination process in S403, and the dodging correction is performed when there are many low-brightness areas. However, the embodiment of the invention is in this form. It is not limited. For example, in the histogram determination process of S403, the magnitude of the contrast may be determined instead of the ratio of the low luminance area. The contrast determination method can be realized using a known technique. Here, for an image having a small contrast, in the correction process of S405, a known gradation correction process may be performed to improve the visibility by increasing the contrast.

  As described above, in the present embodiment, when the bit rate of the image data to be processed is equal to or higher than the first threshold based on the encoding result, the WDR function is invalidated while the bit rate is lower than the second threshold. For an image having a large proportion of low-luminance regions, visibility in the low-luminance regions can be ensured by performing luminance value correction instead of the WDR function. Also in this embodiment, when the bit rate is smaller than a predetermined value, the WDR function can be validated to improve the image visibility. Therefore, according to the present embodiment, it is possible to further improve the visibility of an image while adaptively preventing an increase in the bit rate.

  An imaging device corresponding to each of the above-described embodiments and modifications thereof can be realized with a hardware configuration as shown in FIG. FIG. 5 is a block diagram illustrating a hardware configuration example of the imaging apparatus corresponding to the embodiment of the invention. The imaging device can be an arbitrary information processing terminal such as a personal computer, a digital camera, a mobile phone, a smartphone, a PDA, or a tablet terminal.

  The imaging apparatus 100 includes a CPU 501, a RAM 502, and a ROM 503. The ROM 503 stores a basic control program for the information processing apparatus. This control program is read into the RAM 502 when the apparatus is activated and executed by the CPU 501. The secondary storage device 504 is actually a hard disk or a memory disk. The secondary storage device 504 includes a high-level control program (for example, an operation system), a browser, a database for managing information related to processing data, an application for importing data acquired by the function unit 505 into the device 100, and capture Stored data is stored. These pieces of software are read into the RAM 502 in a timely manner and executed by the CPU 501.

  The functional unit 505 is a functional unit having a configuration for executing a specific function corresponding to the type of device. In the present embodiment, the function unit 505 can function as the imaging unit 101. A specific hardware configuration of the imaging unit 101 is as illustrated in FIG. A network interface (I / F) 506 is an interface for connecting to an external network. The operation unit 507 can accept an instruction operation from a user, and includes a button, a touch panel, a dial, and the like. The display unit 508 displays an image or a graphical user interface (GUI) screen according to the display control of the CPU 501, and includes a monitor. The bus 509 is used when the information 501 to 508 exchange information.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (8)

An imaging unit that images a subject under a plurality of exposure conditions and generates a plurality of image data;
Encoding means for encoding image data;
Determination means for determining the magnitude of the bit rate of the encoded image data;
Combining means for combining the plurality of image data,
The determination unit determines whether the bit rate of the first image data of the plurality of image data is smaller than a first threshold;
When the determination unit determines that the bit rate is smaller than the first threshold value, the combining unit calculates the first image data and the second image data of the plurality of image data. A third image data is generated by synthesizing, and the encoding means encodes the third image data. Image processing means for image processing the first image data;
When the determination unit determines that the bit rate is equal to or higher than the first threshold,
The determination means further determines whether or not the bit rate is smaller than a second threshold that is larger than the first threshold,
When the determination unit determines that the bit rate is smaller than the second threshold, the image processing unit corrects the first image data to generate fourth image data, and the encoding The imaging apparatus according to claim 1, wherein the unit encodes the fourth image data.   The image processing means performs a process of correcting a luminance value of a low luminance area of the first image data in which the bit rate is determined to be smaller than the second threshold value. The imaging device described.   The said image processing means performs the process which correct | amends a gradation so that contrast may be emphasized about the said 1st image data determined that the said bit rate is smaller than the said 2nd threshold value. 2. The imaging device according to 2.   The determining means determines the bit rate magnitude of the encoded first image data by determining the bit rate magnitude of the image data encoded before the first image data. The imaging apparatus according to claim 1, wherein the imaging apparatus is determined. The first image data is image data generated by imaging under a first exposure condition among the plurality of exposure conditions,
6. The second image data is image data generated by imaging under a second exposure condition among the plurality of exposure conditions in which the first exposure condition is changed. The imaging device according to any one of the above. An imaging unit that images a subject under a plurality of exposure conditions and generates a plurality of image data;
Encoding means for encoding image data;
Determining means for determining a bit rate of the encoded image data;
An image pickup apparatus control method comprising: a combining unit that combines the plurality of image data,
An imaging step in which the imaging unit generates the plurality of image data; and
A determination step for determining whether or not the bit rate of the first image data among the plurality of image data is smaller than a first threshold;
When it is determined in the determination step that the bit rate is smaller than the first threshold,
A generating step in which the combining unit combines the first image data and the second image data of the plurality of image data to generate third image data;
An image pickup apparatus control method, wherein the encoding means includes an encoding step of encoding the third image data.   The program for operating a computer provided with an imaging part as each means which the imaging device of any one of Claim 1 to 6 has.

Images