JP2021035005A - Imaging apparatus and method for controlling the same - Google Patents

Abstract

To make it possible to relatively reduce a noise component due to heat in a target region by reducing heat evolution in the vicinity of the target region.SOLUTION: An imaging apparatus includes: photoelectric conversion means for generating image data by photoelectric conversion; a plurality of image processing means for performing processing of processing units obtained by dividing image data generated by the photoelectric conversion means; and control means. The control means controls so that in first image processing means and second image processing means among the plurality of image processing means, if the first image processing means is located closer to a part of the photoelectric conversion means which corresponds to a target region of the image data than the second image processing means, dissipation power of the first image processing means becomes smaller than dissipation power of the second image processing means.SELECTED DRAWING: Figure 5

Description

The present invention relates to an image pickup apparatus and a control method for the image pickup apparatus.

In recent years, with the increasing functionality of digital devices, there is an increasing demand for chips mounted on digital devices to be compact, thin, and highly integrated. Even in a laminated LSI in which a plurality of LSIs are three-dimensionally laminated, it is required to reduce the thickness of the chip. The thinned chip has a low thermal conductivity, and a portion where the temperature rises locally becomes apparent. Further, in a laminated LSI, the heat generated by the stacked chips cannot be sufficiently dissipated and the temperature inside the LSI rises, which may cause the chips to malfunction or malfunction. As for the sensor unit, the noise component of the sensor output signal increases due to the local heat generation, which causes deterioration of the sensor signal.

Patent Document 1 discloses a method in which a temperature sensor is installed in a laminated LSI to shut off the operation of chips when heat generation becomes large at a portion where the chips overlap.

Further, Patent Document 2 discloses a method of suppressing a temperature rise by controlling so as to stop a part of image processing based on the temperature detected by the temperature detection unit.

Japanese Unexamined Patent Publication No. 2010-171169 Japanese Unexamined Patent Publication No. 2018-19325

However, in a semiconductor integrated circuit in which a sensor unit and a processing circuit unit that processes input / output signals of the sensor unit are integrated in three dimensions, it is difficult to dissipate locally generated heat from the sensor unit, and a necessary image is required. It is also not preferable to stop the process.

An object of the present invention is to suppress heat generation in the vicinity of a region of interest so that a noise component caused by heat can be relatively reduced in the region of interest.

The image pickup apparatus of the present invention includes a photoelectric conversion means for generating image data by photoelectric conversion, a plurality of image processing means for processing a processing unit in which the image data generated by the photoelectric conversion means is divided, and the plurality of image processing means. In the first image processing means and the second image processing means in the image processing means, the photoelectric conversion means in which the first image processing means corresponds to the region of interest of the image data from the second image processing means. When it is close to the portion of, it has a control means for controlling the power consumption of the first image processing means to be smaller than the power consumption of the second image processing means.

According to the present invention, by suppressing heat generation in the vicinity of the region of interest, the noise component caused by heat in the region of interest can be relatively reduced.

It is a figure which shows the configuration example of the image pickup apparatus. It is a figure which shows the relationship between a photoelectric conversion sensor part, an image processing circuit, and a region of interest. It is a figure which shows the relationship between a power supply voltage, a drive frequency, and a computing capacity. It is a figure which shows the division of a processing unit, and the transfer destination allocation in a data transfer part. It is a flowchart which shows the control method of the control part.

FIG. 1 is a diagram showing a configuration example of the image pickup apparatus 100 according to the present embodiment. The image pickup apparatus 100 is a semiconductor integrated circuit. The image pickup apparatus 100 is configured by joining the first surface 110 and the second surface. A photoelectric conversion sensor unit 101 is provided on the first surface 110. The second surface 120 is provided with a first control area 102a, a second control area 102b, a data transfer unit 103, a control unit 105, a selection unit 106, and an input / output unit 107. The first control region 102a has image processing units 104a and 104b. The second control region 102b has image processing units 104c and 104d.

The photoelectric conversion sensor unit 101 is a photoelectric conversion unit, and generates image data by photoelectric conversion. Specifically, the photoelectric conversion sensor unit 101 converts an optical image of a subject into an image signal (electrical signal), and converts an analog image signal into digital image data.

The data transfer unit 103 conducts with the photoelectric conversion sensor unit 101 at the junction surface between the first surface 110 and the second surface 120, inputs image data from the photoelectric conversion sensor unit 101, and inputs image data to the photoelectric conversion sensor unit 101. Send and receive control signals to each other.

The power supply voltage and the drive frequency of the first control region 102a and the second control region 102b can be set independently by the control unit 105. The control unit 105 supplies the same power supply voltage and drive frequency to the image processing units 104a and 104b included in the first control area 102a, and supplies the same power supply voltage and drive frequency to the image processing units 104c and 104d included in the second control area 102b. Supply the same power supply voltage and drive frequency.

The data transfer unit 103 has a connection terminal for connecting to the photoelectric conversion sensor unit 101, inputs image data from the photoelectric conversion sensor unit 101, and transfers the image data to the image processing units 104a to 104d. The data transfer unit 103 can switch the image processing units 104a to 104d of the transfer destination.

The image processing units 104a to 104d each process the image data transferred by the data transfer unit 103.

The control unit 105 has the image processing unit 104a so that the processing time required for the image processing units 104a to 104d is about the same according to the processing performance of the image processing units 104a to 104d, which is determined depending on the power supply voltage and the drive frequency. Controls the power supply voltage and drive frequency of ~ 104d.

Further, the data transfer unit 103 transfers the image data for which the processing of the image processing units 104a to 104d has been completed to the input / output unit 107, which is an external interface.

The image processing units 104a to 104d can individually switch between the operating state and the standby state. For example, in the first control area 102a, one image processing unit 104a is in the standby state, and the other image processing unit 104b is in the operating state. The image processing unit 104a can reduce the power consumption as compared with the image processing unit 104b.

The selection unit 106 selects a region of interest with respect to the image data output by the photoelectric conversion sensor unit 101 by a predetermined method. The control unit 105 uses any of the image processing units 104a to 104d (for example, 104a) arranged in the vicinity of the region of the photoelectric conversion sensor unit 101 corresponding to the region of interest selected by the selection unit 106. select. The control unit 105 determines the power consumption amount so that one of the image processing units (for example, 104a) has a smaller estimated power consumption than at least one of the other image processing units (for example, 104b to 104d). Controls at least one of the drive frequencies.

FIG. 2A is a diagram showing the positions of the first control area 102a and the second control area 102b and the image processing units 104a to 104d of the broken line with respect to the front view of the solid line photoelectric conversion sensor unit 101. ..

FIG. 2B is a diagram showing an example of the image data 202 output by the photoelectric conversion sensor unit 101 and the region of interest 201. For example, the photoelectric conversion sensor unit 101 acquires the image data 202 prior to the main shooting, and the selection unit 106 selects the region of interest 201 of the image data 202. Alternatively, for example, in a surveillance camera having a fixed position, the selection unit 106 may set the entrance / exit as the area of interest 201 in advance.

FIG. 2 (c) is a diagram in which the region of interest 201 of FIG. 2 (a) and FIG. 2 (b) are superimposed. FIG. 2C shows the image processing units 104a to 104d in order of priority, that is, an order in which power consumption is desired to be suppressed, according to the distance from the center of gravity position of the region of interest 201.

For example, the image processing unit 104b has the highest priority among the image processing units 104a to 104d because it is closest to the position of the center of gravity of the region of interest 201. Since the image processing unit 104a is second closest to the position of the center of gravity of the region of interest 201 among the image processing units 104a to 104d, the image processing unit 104a has the second highest priority. Since the image processing unit 104d is the third closest to the position of the center of gravity of the region of interest 201 among the image processing units 104a to 104d, the image processing unit 104d has the third highest priority. Since the image processing unit 104c is the fourth closest to the position of the center of gravity of the region of interest 201 in the image processing units 104a to 104d, the priority is fourth.

The selection unit 106 selects the region of interest 201 in which the noise increase due to the temperature rise of the photoelectric conversion sensor unit 101 is desired to be suppressed when the photoelectric conversion sensor unit 101 is photographed. For example, the selection unit 106 inputs the focusing area of the image data in the autofocus process from the camera control unit (not shown) connected via the input / output unit 107, and selects the focusing area as the attention area 201. .. Further, the selection unit 106 may input a subject discrimination result from the camera control unit (not shown) and select a region matching a predetermined subject (for example, a person, a face, etc.) as the region of interest 201. Further, the selection unit 106 may select an area designated by the user by the input means (not shown) as the area of interest 201.

The control unit 105 determines the priority order based on the positional relationship of the image processing units 104a to 104d with respect to the area of interest 201 in FIG. 2C. For example, the control unit 105 determines the priority order in ascending order of the distance from the position of the center of gravity of the region of interest 201 to the image processing units 104a to 104d. Further, the control unit 105 may determine the priority order in the image processing units 104a to 104d in descending order of the area overlapping the attention area 201. Further, the control unit 105 may determine the priority order according to the temperature measured in the vicinity of each image processing unit 104a to 104d.

The control unit 105 determines the power supply voltage and drive frequency of the first control area 102a and the second control area 102b, and each image processing unit according to the determined priority and the computing power of the image processing units 104a to 104d. It is determined to switch between the operating state and the standby state of 104a to 104d.

Each image processing unit 104a to 104d has a relationship of power supply voltage, drive frequency, state, computing power, and power consumption, as shown in FIG. 3A, for example. For example, when the image processing units 104a to 104d require the computing power 7 as a whole, the control unit 105 has higher computing power with respect to the image processing units 104a to 104d in descending order of priority in FIG. 2C. To assign. For example, as shown in FIG. 3B, the control unit 105 has the power supply voltage and drive frequency of the first control area 102a and the second control area 102b, and the state and computing power of the image processing units 104a to 104d. To determine. The first control area 102a controls the power supply voltage and the drive frequency of the image processing unit 104a and the image processing unit 104b. The second control region 102b controls the power supply voltage and the drive frequency of the image processing unit 104c and the image processing unit 104d. The total computing power of the image processing units 104a to 104d is "7". Since the image processing units 104a and 104b are close to the area of interest 201 in the image processing units 104a to 104d, the power consumption can be reduced. Details of the control unit 105 allocating the above-mentioned computing power will be described later.

FIG. 4 is a diagram showing an example in which the image data generated by the photoelectric conversion sensor unit 101 is divided into 16 processing units 403. The control unit 105 assigns the data transfer unit 103 a number of processing units 403 according to the ratio of the computing powers of the first control area 102a and the second control area 102b to the first control area 102a and the second control area 102a. The transfer of the data transfer unit 103 is set so as to be processed in the control area 102b of. In the example of FIG. 3B, the computing power of the first control region 102a is 1, and the computing power of the second control region 102b is 6. The control unit 105 allocates the processing unit 403 to the first control area 102a and the second control area 102b so as to be close to the ratio of the computing power of the first control area 102a and the second control area 102b. For example, as shown in FIG. 4, the control unit 105 allocates two processing units 401 to the first control area 102a and 14 processing units 402 to the second control area 102b. Controls the transfer of 103. The data transfer unit 103 transfers a number of processing units 403 according to the computing power (power consumption) of each of the plurality of image processing units 104a to 104d to the plurality of image processing units 104a to 104d. The image processing units 104a to 104d process the processing unit 403.

FIG. 5A is a flowchart for explaining a control method of the control unit 105. Step S501 is an operation mode acquisition step. In step S501, the control unit 105 acquires the operation mode from the camera control unit (not shown). The operation mode includes, for example, the setting of the number of output pixels (resolution), the presence / absence of the continuous shooting setting, the number of pixels to be processed, or the restriction of the processing time, and affects step S521 in FIG. 5 (b).

Step S502 is a region of interest acquisition step. In step S502, the control unit 105 acquires the information of the region of interest 201 selected by the selection unit 106. The area of interest 201 is, for example, a pixel position representing the area of interest 201 such as the center of gravity, a predetermined figure such as an ellipse or a rectangle circumscribing the area of interest 201, or a predetermined boundary line of the area of interest 201. It may be a figure. As a result, the control unit 105 can efficiently process the region of interest 201.

Step S503 is a priority determination step. In step S503, the control unit 105 determines the priority order for suppressing the heat generation amount (power consumption amount) of each image processing unit 104a to 104d. For example, the control unit 105 determines the priority order of the image processing units 104a to 104d in ascending order from the position of the center of gravity of the region of interest 201 to the image processing units 104a to 104d. Further, the control unit 105 may determine the priority order of the image processing units 104a to 104d in descending order of the area where the image processing units 104a to 104 overlap with the area of interest 201. Further, the control unit 105 may determine the priority order of the image processing units 104a to 104d according to the temperature measured in the vicinity of the image processing units 104a to 104d. Further, the control unit 105 weights the image processing units 104a to 104d according to the amount of heat generated, and obtains the center point of the heat generation amount of the image processing units 104a to 104d. Then, the control unit 105 determines the priority order of the image processing units 104a to 104d according to the distance from the position of the center of gravity of the region of interest 201 to the center point of the calorific value of each image processing unit 104a to 104d. Further, the control unit 105 may convert the above-mentioned plurality of scales into scores corresponding to the respective numerical values, and determine the priority order based on the total score of the scores.

Step S504 is a computing power allocation step. In step S504, the control unit 105 determines the computing power of each image processing unit 104a to 104d so that the computing power of each image processing unit 104a to 104d is not insufficient for the computing power determined by the operation mode. adjust. The details will be described below.

FIG. 5B is a flowchart for explaining the details of step S504 of FIG. 5A. Step S521 is a required computing capacity value determination step. In step S521, the control unit 105 determines the required calculation capacity value as the calculation amount to be processed per unit time from the total calculation amount and the processing time determined according to the operation mode. For example, the control unit 105 prepares in advance a table for converting from the operation mode to the required computing power value, and determines the required computing power value with reference to the table. Hereinafter, an example in which the required computing capacity value is “7” will be described.

Step S522 is a calculation capacity provisional allocation step. In step S522, the control unit 105 sets the maximum computing power value (computing capacity "3" in the example of FIG. 3A) as the initial value of the computing power for each of the image processing units 104a to 104d. .. Next, the control unit 105 selects the image processing unit 104b having the highest priority obtained in step S503 (that is, the youngest priority in FIG. 2C) as the adjustment target of the computing power in step S523. To do.

Step S523 is a computing capacity adjusting step. In step S523, the control unit 105 sets the minimum possible value for the computing power of the image processing unit to be adjusted within a range in which the total computing power of all the image processing units 104a to 104d does not fall below the required computing power value. select.

For example, when the initial value of the computing power of each of the four image processing units 104a to 104d is "3", the total computing power of the four image processing units 104a to 104d is "12", which is the required computing power. It is "5" larger than the value "7". Therefore, the control unit 105 sets the computing power of the image processing unit 104b to be adjusted to "0".

Step S524 is a repeating condition determination step. In step S524, the control unit 105 determines whether or not the adjustment of the computing power of all the image processing units 104a to 104d has been completed. When the control unit 105 determines that the adjustment of the computing power of all the image processing units 104a to 104d has not been completed, the image processing unit having the next highest priority is set as the image processing unit to be adjusted in step S523. Return to.

For example, in step S524, the control unit 105 sets the image processing unit 104a having the second highest priority as the next adjustment target, and returns to step S523. As a result of the above, the computing power of each of the three image processing units 104a, 104c, and 104d is "3", and the computing power of one image processing unit 104b is "0". The total computing power of the image processing units 104a to 104d is "9", which is "2" larger than the required computing power value "7". In step S523, the control unit 105 subtracts "2" from the computing power "3" of the image processing unit 104a to be adjusted, and changes the computing power of the image processing unit 104a from "3" to "1". As a result, the computing power of the two image processing units 104c and 104d becomes "3", the computing power of one image processing unit 104a becomes "1", and the computing power of one image processing unit 104b becomes "1". It becomes "0". The total computing power of the image processing units 104a to 104d is "7".

Next, in step S524, the control unit 105 sets the image processing unit 104d having the third highest priority as the next adjustment target, and returns to step S523. The total computing power of the image processing units 104a to 104d is "7", which is the same as the required computing power value "7". Therefore, in step S523, the control unit 105 does not change the computing power “3” of the image processing unit 104d to be adjusted.

Next, in step S524, the control unit 105 sets the image processing unit 104c having the fourth highest priority as the next adjustment target, and returns to step S523. The total computing power of the image processing units 104a to 104d is "7", which is the same as the required computing power value "7". Therefore, in step S523, the control unit 105 does not change the computing power “3” of the image processing unit 104c to be adjusted.

Next, in step S524, the control unit 105 determines that the adjustment of the computing power of all the image processing units 104a to 104d has been completed, finishes the processing of the flowchart of FIG. 5B, and finishes the processing of the flowchart of FIG. 5B. Step S505 of the above.

In step S524, the control unit 105 may end the processing of the flowchart of FIG. 5B when the total calculation capacity of the image processing units 104a to 104d is the same as the required calculation capacity value. In that case, if the total computing power of the image processing units 104a to 104d is not the same as the required computing power value, the control unit 105 returns to step S523. For example, in the above example, in the control unit 105, after the adjustment of the image processing unit 104a having the second highest priority is completed, the total computing power of the image processing units 104a to 104d becomes the same as the required computing power value. The processing of the flowchart of FIG. 5B ends.

Step S505 in FIG. 5A is an operation setting step. In step S505, the control unit 105 sets the first control area 102a, the second control area 102b, the image processing units 104a to 104d, and the data transfer unit 103. The control unit 105 sets the power supply voltage and the drive frequency of the first control area 102a and the second control area 102b, respectively, based on the computing power of the image processing units 104a to 104d. Further, the control unit 105 sets the operating state or the standby state of each image processing unit 104a to 104d based on the computing power of the image processing units 104a to 104d. Further, the control unit 105 transfers each of the processing units 403 of FIG. 4 to the image processing unit of the transfer destination so that the processing amount of the data transfer unit 103 corresponds to the computing power of each of the image processing units 104a to 104d. Allocate to 104a to 104d. As a result, each of the image processing units 104a to 104d completes the processing in about the same time.

Here, an arbitrary first image processing unit and a second image processing unit among the plurality of image processing units 104a to 104d will be described. When the first image processing unit is closer to the part of the photoelectric conversion sensor unit 101 corresponding to the center of gravity of the region of interest 201 of the image data than the second image processing unit, the control unit 105 of the first image processing unit The power consumption is controlled to be smaller than the power consumption of the second image processing unit. Specifically, the control unit 105 controls at least one of the power supply voltage and the drive frequency of the first image processing unit.

As described above, the control unit 105 can allocate the processing amount according to the computing power of the image processing units 104a to 104d. Further, the control unit 105 sets the image data divided into the plurality of processing units 403 to process the number of the processing units 403 according to the respective computing powers for the image processing units 104a to 104d. As a result, the control unit 105 can prevent the time until the processing of all the processing units 403 is completed from being extended more than necessary.

It should be noted that all of the above embodiments merely show examples of embodiment in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

101 Photoelectric conversion sensor unit, 102a, 102b control area, 103 data transfer unit, 104a, 104b, 104c, 104d image processing unit, 105 control unit, 106 selection unit, 107 input / output unit, 201 attention area

Claims (9)

A photoelectric conversion means that generates image data by photoelectric conversion, and
A plurality of image processing means for processing a processing unit in which image data generated by the photoelectric conversion means is divided, and
In the first image processing means and the second image processing means among the plurality of image processing means, the first image processing means corresponds to the region of interest of the image data from the second image processing means. An image pickup characterized by having a control means for controlling the power consumption of the first image processing means to be smaller than the power consumption of the second image processing means when the portion is close to the part of the photoelectric conversion means. apparatus. The control means controls at least one of the power supply voltage and the drive frequency of the first image processing means, so that the power consumption of the first image processing means is smaller than the power consumption of the second image processing means. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is controlled so as to be the same. The image pickup apparatus according to claim 1 or 2, further comprising a transfer means for transferring a number of processing units corresponding to the power consumption of each of the plurality of image processing means to the plurality of image processing means. The imaging apparatus according to any one of claims 1 to 3, further comprising a selection means for selecting a focusing region of the image data as the region of interest. The imaging apparatus according to any one of claims 1 to 3, further comprising a selection means for selecting a region of a predetermined subject in the image data as the region of interest. A first control region including the first image processing means and
Further having a second control area including the second image processing means,
The control means supplies the same power supply voltage or drive frequency to the image processing means included in the first control region, and the same power supply voltage or drive to the image processing means included in the second control region. The image pickup apparatus according to claim 2, wherein a frequency is supplied. The image pickup apparatus according to any one of claims 1 to 6, wherein the surfaces of the plurality of image processing means are joined to the surfaces of the photoelectric conversion means. When the first image processing means is closer to the portion of the photoelectric conversion means corresponding to the center of gravity of the region of interest of the image data than the second image processing means, the control means performs the first image processing. The image pickup apparatus according to any one of claims 1 to 7, wherein the power consumption of the means is controlled to be smaller than the power consumption of the second image processing means. A photoelectric conversion means that generates image data by photoelectric conversion, and
A control method for an image pickup apparatus having a plurality of image processing means for processing a processing unit in which image data generated by the photoelectric conversion means is divided.
In the first image processing means and the second image processing means among the plurality of image processing means, the first image processing means corresponds to the region of interest of the image data from the second image processing means. An image pickup apparatus having a control step for controlling the power consumption of the first image processing means to be smaller than the power consumption of the second image processing means when it is close to the portion of the photoelectric conversion means. Control method.

Images