JP2023145155A - Electronic device, control method for the same, image processing device, method for the same, program, and storage medium - Google Patents

Abstract

To make it possible to adjust image data collectively and automatically so that the noise sense of the image data becomes equal regardless of the device that generated the image data or the image processing in which the image data was generated.SOLUTION: Included are an imaging element that performs photoelectric conversion of incoming light and outputs an image signal, image processing means that performs image processing on the image signal and outputs image data, acquisition means for acquiring a noise amount generated in each of the processes performed by the imaging element and the image processing means, generating means for generating noise amount information indicating a total of the noise amount acquired from the acquisition means, and associating means for associating the generated noise amount information with the image signal or the image data.SELECTED DRAWING: Figure 1

Description

The present invention relates to an electronic device and its control method, an image processing device and method, a program, and a storage medium, and particularly relates to a technique for adjusting noise superimposed on image data obtained by photographing through image processing.

Conventionally, in an imaging device such as a digital still camera or a digital camcorder, various types of noise information are superimposed on an image signal by an analog gain circuit of an image sensor, a digital gain circuit of an image processing engine, and various types of image processing. When images are taken with different imaging devices or with different ISO sensitivity value settings, it is common practice to edit the resulting images after taking them to match the noise (graininess) of each image. . In this case, the ISO sensitivity value is recorded in the meta information and Exif information in order to adjust the noise of each captured image using the ISO sensitivity information recorded in the meta information and Exif information set at the time of shooting as a guide. That is important.

However, although ISO sensitivity value information is useful for exposure value, it is often not useful for noise amount information. This is because the amount of noise varies depending on the manufacturer and model of the imaging device even if the ISO sensitivity value is the same. Specifically, the ISO sensitivity value is generated from a combination of the analog gain of the image sensor and the digital gain of the image processing engine, but generally the digital gain increases the amount of noise more than the analog gain. Since the distribution of analog gain and digital gain differs depending on the manufacturer and model, the amount of noise relative to the ISO sensitivity value differs depending on the manufacturer and model, as shown in FIG.

On the other hand, in Patent Document 1, instead of the set ISO sensitivity value, noise amount information superimposed on the output of the image sensor is recorded on a recording medium together with the captured image, and based on the noise amount information, A method of adjusting the amount of noise in an image by image editing has been proposed.

Japanese Patent Application Publication No. 2005-303803

In the method described in Patent Document 1, the amount of noise in the output of an image sensor is measured, and noise amount information is recorded together with the captured image. However, in recent imaging devices, there are many controls that affect noise amount information, such as image processing engines and codec processing for generating still image and video files. Furthermore, in addition to recording on a recording medium with an imaging device, it is also common to record images output from an external output unit such as SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interace) (registered trademark). is being carried out. As described above, since recorded images pass through various processing systems, noise amount information only from the output of the image sensor is insufficient to use as a guideline for adjusting the noise feeling during image editing.

The present invention has been made in view of the above problems, and it is an object of the present invention to enable appropriate association of information regarding noise in image data.

In order to achieve the above object, an electronic device of the present invention includes an image sensor that photoelectrically converts incident light and outputs an image signal, and an image sensor that performs image processing on the image signal and outputs image data. a processing means, an acquisition means for respectively acquiring the amount of noise generated in each process performed by the image sensor and the image processing means, and generating noise amount information indicating a total of the amount of noise acquired by the acquisition means. It has a generating means and an associating means for associating the generated noise amount information with the image signal or the image data.

According to the present invention, information regarding the noise feeling of image data can be appropriately associated.

1 is a block diagram showing an example of a schematic configuration of an imaging device according to an embodiment of the present invention. FIG. 1 is a block diagram showing the configuration of a general computer that can perform image editing processing according to an embodiment. 5 is a flowchart showing a control procedure in the embodiment. 5 is a flowchart showing an image editing procedure in the embodiment. A conceptual diagram for explaining ISO sensitivity values and noise amounts that differ depending on the manufacturer.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

FIG. 1 is a block diagram showing an example of a schematic configuration of an imaging device 100 according to an embodiment of the present invention. As shown in FIG. 1, an imaging device 100 shows the configuration of an imaging device such as a so-called digital still camera or digital camcorder. However, the present invention is not limited to this, and may be applied to various electronic devices having a camera function. For example, the imaging device according to the present invention may be a mobile communication terminal with a camera function such as a mobile phone or a smartphone, a portable computer with a camera function, a portable game machine with a camera function, or the like.

The lens unit 101 forms an image of light incident from outside the imaging device 100 on an image sensor of the imaging unit 103 . Although the lens section 101 is represented by one lens here, it is usually composed of a plurality of lenses. The light attenuation unit 102 adjusts the amount of light incident through the lens unit 101, and includes an aperture blade, an ND (Neutral Density) filter, and the like.

The light whose intensity has been adjusted by the light attenuation unit 102 is transmitted to the imaging unit 103, which converts the light into charges using a known photoelectric conversion method, such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor. incident on the image sensor. The imaging unit 103 includes an imaging device, a driver for driving the imaging device, a timing generation circuit, a CDS/AGC circuit, an A/D converter, and the like. A driver and a timing generation circuit for driving the image sensor supply drive pulses and the like for driving the image sensor to the image sensor, read out charges accumulated in the image sensor, and adjust exposure time.

The CDS/AGC circuit performs sampling and amplification on an image signal corresponding to the charge read out from the image sensor. Note that correlated double sampling (CDS) is performed for sampling, and automatic gain control (AGC) is performed for amplification. The A/D converter converts the image data (analog signal) output from the CDS/AGC circuit into a digital signal and outputs the digital signal.
A gain control unit 106 controls the analog gain of the image sensor.

The image processing unit 104 performs various signal processing such as AWB (Auto White Balance) processing, gamma conversion, and digital gain processing on the image information (digital signal) output from the A/D converter, and performs color processing on the image information (digital signal) output from the A/D converter. and brightness, etc., to generate final image data. Note that the digital gain controlled by the image processing unit 104 is increased or decreased by control of each module of the image processing unit 104, and therefore, unlike the gain control unit 106, it is intentionally not described as an individual configuration.

The codec control unit 107 converts the image data processed by the image processing unit 104 into an arbitrary recording format. Here, the codec control unit 107 supports multiple recording formats, and can convert not only to one type of recording format but also to still image and video formats of multiple recording formats that exist in the settings. can. Further, the codec control unit 107 is also capable of converting into a plurality of different recording formats at the same time.

The recording unit 109 records the image data converted into a predetermined recording format by the codec control unit on a storage medium.
Further, the image data processed by the image processing section 104 is displayed on a display device by the display section 110.

The microcomputer 105 is a circuit called a microcontroller, and comprehensively controls the operations of the imaging apparatus 100, including the control described above. Further, the microcomputer 105 sets the ISO sensitivity value, the gain control unit 106 calculates the analog gain setting amount corresponding to the set ISO sensitivity value, and sends the calculated value to the imaging unit 103. Controls the analog gain of the image sensor.

The noise amount calculation unit 108 obtains the noise amount of each image data from the image data output from the imaging unit 103, the image processing unit 104, and the codec control unit 107. The amount of noise may be obtained by measuring the amount of noise from the image data itself, or may be obtained in the following manner. In other words, by communicating control details regarding noise fluctuations in processing performed by each section, a table indicating how much noise amount fluctuates in each processing is set in advance and stored in a memory (not shown). demand.

The meta information generation unit 111 generates meta information including the calculated noise amount information and sends it to the recording unit 109. The recording unit 109 records the sent meta information together with the image data as an image file. Further, meta information may be transmitted to the display section 110 and displayed on the display section 110.

Furthermore, the output unit 112 may add the meta information generated by the meta information generation unit 111 to the image data output from the codec control unit 107 and output the image data to an external device through wired or wireless communication.

Through the above processing, it becomes possible to individually obtain the amount of noise superimposed in each part and to individually add noise amount information to image data recorded through different control systems and images displayed.

In this embodiment, as shown in FIG. 1, each part has an independent configuration, but all or part of the configuration may be realized by the microcomputer 105.

Next, with reference to FIG. 2, the computer 206 for processing image data obtained by the imaging apparatus 100 will be described. FIG. 2 shows a general hardware configuration of computer 206. The computer 206 includes a ROM 207, a RAM 208, a central processing unit (CPU) 209, a disk device 210, a bus 211, an I/O port 212, an I/F 213, and an external storage device 214.

The external storage device 214 is for driving an external storage medium, and reads out from the storage medium an image file in which image data recorded by the recording unit 109 and noise amount information are recorded. Further, the I/O port 212 allows communication with an external device including the imaging device 100 directly or via a network or the like to obtain an image file from the imaging device 100 through wired or wireless communication.

ROM 207 holds various application programs, and RAM 208 provides a storage area necessary for the operation of the programs. The CPU 209 performs processing according to the program stored in the ROM 207. The CPU 209 loads a program stored in the ROM 207 that implements the control procedure shown in FIG. processing can be realized.

The bus 211 connects each of the above configurations and enables data to be exchanged between each configuration. The I/F 213 is connected to an input unit 220 (operation means) such as a mouse, keyboard, or touch panel.

Next, the control performed by the microcomputer 105 in the imaging apparatus 100 of this embodiment will be described with reference to the flowchart in FIG. 3.

In step S<b>101 , the gain control unit 106 calculates a gain value to be set as an analog gain of the image sensor from the ISO sensitivity value set by the microcomputer 105 and communicates it to the image capture unit 103 . The imaging unit 103 sets the communicated gain value to an analog gain circuit within the imaging element. Next, in step S102, the noise amount calculation unit 108 obtains the amount of noise superimposed on the image signal output from the imaging unit 103. Here, the amount of noise is obtained from a table showing the amount of noise for the gain value set in the analog gain circuit of the image sensor.

Next, in step S103, it is determined whether or not the image processing unit 104 is to be used. If it is determined that the image processing unit 104 is to be used here, in step S104, the noise amount calculation unit 108 calculates the amount of noise superimposed on the image data in each image processing in the image processing unit 104 compared to the processing in step S102. Similarly, get it from the table. For example, since noise is superimposed on image data by controlling the digital gain with AWB or gamma conversion, the amount of noise for each processing setting value is acquired.

Next, in step S105, the codec control unit 107 determines whether the image data has been converted into a still image or a moving image in the set recording format. Here, if it is determined that the image data has been converted to the recording format set by the codec control unit 107, in step S106, the noise amount calculation unit 108 calculates the amount of noise superimposed on the image data after image format conversion. get. Here, as in step S102, the amount of noise for conversion to each recording format is obtained from the table.

Then, in step S107, the noise amount calculation unit 108 sums up the noise amounts acquired in steps S102, S104, and S106, and calculates the total amount of noise superimposed on the image data before conversion by the codec control unit 107.

In step S108, the meta information generation unit 111 converts the image data into a format of meta information to be superimposed on the image data in the recording format converted by the codec control unit 107. Then, in step S109, the recording unit 109 records the image file together with the image data on a storage medium such as an SD card.

On the other hand, if it is determined in step S103 that the image processing unit 104 is not being used, the output image data is a RAW image from the image sensor. Therefore, in step S113, the noise amount calculation unit 108 calculates the noise amount acquired in step S102 as the total noise amount superimposed on the image data. Then, in step S114, it is determined whether the recording unit 109 records the image data on the storage medium. If it is determined that the data is to be recorded on the storage medium, in step S115, similarly to step S108, the meta information generation unit 111 converts the data into a meta information format to be superimposed on the image data in the RAW image format. Then, in step S109, the recording unit 109 records the image file together with the image data on a recording medium such as an SD card.

Further, in step S114, if the recording unit 109 determines that the captured image is not to be recorded on the storage medium, similarly to step S108, the meta information generating unit 111 changes the format of the meta information to be superimposed on the image data in the RAW image format. Convert. Then, in step S112, an image file including image data and meta information is output to the outside of the imaging device 100 from the output unit 112 such as SDI or HDMI (registered trademark).

If the codec control unit 107 determines in step S105 that the image data has not been converted into a still image or a moving image in the set recording format, the process advances to step S110. In step S110, the noise amount calculation unit 108 sums up the noise amounts obtained in steps S102 and S104, and calculates the total amount of noise superimposed on the image data before image processing. Then, in step S111, similarly to step S108, the meta information generation unit 111 converts the image data into a meta information format corresponding to the image format. Then, in step S112, an image file including image data and meta information is output to the outside of the imaging device 100 from the output unit 112 such as SDI or HDMI (registered trademark).

Note that in the above-described process, a case has been described in which the noise amount according to the process is acquired from the table, but the noise amount may be measured from the image signal or image data itself.

Furthermore, in the above example, a case has been described in which the imaging device 100 acquires the amount of noise to be superimposed through analog gain processing using an analog gain circuit, image processing, and image data recording format conversion processing. However, the present invention is not limited to these, and any method may be used as long as the amount of noise for a process that generates noise is quantitatively acquired and stored in association with image data.

Next, the image editing process in this embodiment will be described with reference to the flowchart in FIG. Note that in the image editing process of this embodiment, the amount of noise is adjusted based on the amount of noise information set in the flowchart shown in FIG. As described above, the CPU 209 can implement the image editing process by loading the application program stored in the ROM 207 and implementing the control procedure shown in FIG. 4 into the RAM 208 and executing it. Note that although the description here assumes that the image editing process is performed by the computer 206, which is an external device of the imaging device 100, if the imaging device 100 has an image editing function that allows editing when image data is played back, the imaging device 100 You can also do it inside.

First, in step S201, an image file having meta information including image data and noise amount information is acquired. Here, the image file recorded on the recording medium by the recording unit 109 may be read by the external storage device 214, or the image file output from the output unit 112 may be input via the I/O port 212. . Furthermore, here, the image files to be read are not only image files of images photographed by the same imaging device, but also image files of images photographed by different imaging devices. It is also possible to read image data in different formats, such as image data in a RAW image format that has not been subjected to image processing, or image data of a moving image converted by the codec control unit 107.

Next, in step S202, the user edits the image data of one of the read image files. In the following description, the image data on which image editing is performed will be referred to as a "representative image." At this time, the user uses a user interface provided by the image editing application to adjust the amount of noise in the representative image.

In step S203, it is determined whether noise amount information exists in the meta information of all the read image files. Here, if noise amount information exists in the meta information of all the read image files, in step S204, the noise amount information of the read image files is acquired and analyzed. Then, in step S202, the noise amount adjusted by the user is set as the target noise amount, and a difference from the target noise amount is calculated. Based on the result, in step S205, the noise amount of the image data is sensitized or desensitized by the difference amount calculated in step S204 so as to match the target noise amount. Thereby, the amount of noise in the image data of all the read image files can be adjusted (corrected) at once.

On the other hand, in step S203, if there is even one image file whose meta information does not include noise amount information, the user is notified in step S206 that batch processing is not possible. Furthermore, the user is allowed to select whether to collectively adjust only the image data of the image files in which noise amount information exists, or to manually adjust the image data of all the read image files.

If the user selects to collectively adjust only the image data of image files in which noise amount information exists, in step S207, the image data of image files in which noise amount information exists is adjusted in the same manner as in step S204. , obtain and analyze the noise amount information of the image file. Then, in step S202, the noise amount adjusted by the user is set as the target noise amount, and a difference from the target noise amount is calculated. Thereafter, in step S208, similarly to step S205, the noise amount of the image data is sensitized or desensitized by the difference amount calculated in step S207 so as to match the target noise amount. Thereby, the noise amount of the image data of the image file in which noise amount information exists is adjusted (corrected) all at once.

On the other hand, if it is determined in step S206 that batch processing is not to be performed, the user uses a user interface provided by the image editing processing application to make adjustments to each image data in step S209.

Note that in the above example, the process of adjusting the amount of noise through image editing processing has been described, but image editing of color, brightness, etc. may also be performed at the same time.

Furthermore, in the above-described processing, the noise amount of other image data is made to match the noise amount of the representative image, but the present invention is not limited to this, and the processing is performed so that the noise amount of the representative image approaches the noise amount of the representative image. Just go. Further, by determining the target noise amount in advance, it is possible to bring the noise amount of image data closer to each other without adjusting the noise amount of the representative image. Further, the target noise amount may be configured such that the user selects the target noise amount from a plurality of predetermined noise amounts.

As described above, according to this embodiment, by acquiring the amount of noise generated when image data is generated and storing it in association with the image data, the feeling of noise is reduced regardless of the device that generated the image data. You can edit them all at once to make them equal.

<Other embodiments>
Further, the present invention provides a system or device with a program that implements one or more functions of the above-described embodiments via a network or a storage medium, and one or more processors in a computer of the system or device executes the program. This can also be realized by reading and executing processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

101: Lens section, 102: Light reduction section, 103: Imaging section, 104: Image processing section, 105: Microcomputer, 106: Gain control section, 107: Codec control section, 108: Noise amount calculation section, 109: Recording section , 110: Display unit, 111: Meta information generation unit, 112: Output unit, 206: Computer, 207: ROM, 208: RAM, 209: Central processing unit (CPU), 210: Disk device, 211: Bus, 212: I/O port, 213: I/F, 214: External storage device

Claims (17)

an image sensor that photoelectrically converts incident light and outputs an image signal;
image processing means for performing image processing on the image signal and outputting image data;
acquisition means for respectively acquiring the amount of noise generated in each process performed by the image sensor and the image processing means;
generating means for generating noise amount information indicating the total of the noise amounts acquired by the acquiring means;
An electronic device comprising: associating means for associating the generated noise amount information with the image signal or the image data. The image sensor performs analog gain processing,
The electronic device according to claim 1, wherein the acquisition means acquires an amount of noise according to a gain used in the analog gain processing. The image processing means performs digital gain processing,
3. The electronic device according to claim 1, wherein the acquisition means acquires an amount of noise according to a gain used in the digital gain processing. further comprising a converting unit that performs a conversion process to convert at least one image format of the image signal output from the image sensor and the image data processed by the image processing unit into a predetermined image format;
4. The electronic device according to claim 1, wherein the acquisition means further acquires an amount of noise generated in the conversion process by the conversion means. further comprising a storage means for holding a table indicating the amount of noise corresponding to a predetermined process,
5. The electronic device according to claim 1, wherein the acquisition means acquires the amount of noise corresponding to each of the processes from the table. 5. The electronic device according to claim 1, wherein the acquisition means measures the amount of noise in the image signal and the image data. 7. The apparatus according to claim 1, further comprising a storage means for storing the image signal or the image data associated with each other by the association means and the noise amount information in a storage medium. electronic equipment. 8. The apparatus according to claim 1, further comprising an output means for outputting the image signal or the image data associated by the association means and the noise amount information to the outside. Electronics. an acquisition means for acquiring an image file;
determining means for determining whether the image file includes noise amount information indicating the amount of noise in the image data;
When a plurality of image files are acquired by the acquisition means, the image data of an image file including the noise amount information among the acquired plurality of image files is corrected so as to approach a predetermined amount of noise. An image processing device comprising means and. further comprising an operation means for a user to adjust the amount of noise of a representative image that is image data of one image file including the noise amount information when a plurality of image files are acquired by the acquisition means;
The image processing apparatus according to claim 9, wherein the predetermined amount of noise is the amount of noise of the representative image adjusted by the operating means. further comprising selection means for selecting one of a plurality of predetermined noise amounts,
The image processing apparatus according to claim 9, wherein the predetermined noise amount is a noise amount selected by the selection means. When a plurality of image files are acquired by the acquisition means, and if there is an image file that does not include the noise amount information among the acquired image files, the image data of the image file that includes the noise amount information is , further comprising determining means for determining whether or not to perform correction by the correcting means,
12. The image processing apparatus according to claim 9, wherein when the determination means determines that the correction means does not perform the correction, the correction means does not perform the correction. 13. The correcting means matches image data of an image file including the noise amount information among the plurality of acquired image files to the predetermined noise amount. The image processing device according to item 1. A method for controlling an electronic device including an image sensor that photoelectrically converts incident light and outputs an image signal, the method comprising:
an image processing step in which the image processing means performs image processing on the image signal and outputs image data;
an acquisition step in which the acquisition means acquires the amount of noise generated in each process performed in the image sensor and the image processing step;
a generating means for generating noise amount information indicating a total of the noise amounts acquired in the acquiring step;
A method for controlling an electronic device, comprising: an associating step in which the associating means associates the generated noise amount information with the image signal or the image data. an acquisition step in which the acquisition means acquires an image file;
a determining step in which the determining means determines whether or not the image file includes noise amount information indicating the amount of noise in the image data;
When a plurality of image files are acquired in the acquisition step, the correction means makes the image data of the image file including the noise amount information closer to a predetermined amount of noise among the plurality of acquired image files. An image processing method comprising: a correction step of correcting; A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 9 to 13. A computer readable storage medium storing the program according to claim 16.

Images