JP2020145621A - Image processing system, imaging apparatus, image processing method, and program - Google Patents

Abstract

To provide an image processing system capable of solving the disadvantage that when depth composition is performed with an image processing system that has multiple processing units, there is a possibility that processing may be delayed due to a large amount of data.SOLUTION: The image processing system includes: acquisition means that acquires multiple images; first processing means that receives signals of the multiple images from the acquisition means, split the multiple images, and sets each of the multiple images to multiple subregions; and second processing means that receives signals of a part of the subregions out of the multiple subregions from the first processing means and generates an image of depth composition. A transmission rate at which the second processing means receives signals from the first processing means is lower than a transmission rate at which the first processing means receives signals from the acquisition means.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing device that synthesizes a plurality of images having different focus positions.

When shooting multiple subjects whose distances from imagers such as digital cameras are significantly different from each other, or when shooting a subject that is long in the depth direction, the depth of field is insufficient, so focus is only on a part of the subject. It may not be possible to match. Alternatively, when it is desired to acquire an image having an extremely high resolution, it is necessary to take an image with a shallow depth of field, and the entire subject may not be within the depth of field. In order to solve this problem, in Patent Document 1, a plurality of images having different focus positions are imaged, only the focusing region is extracted from each image, combined into one image, and the entire imaging region is focused. A so-called depth composition technique for generating a composite image is disclosed.

On the other hand, in Patent Document 2, a first image processing unit that receives an image signal from an imaging unit or the like is arranged in front of the second image processing unit, and the second image processing unit that cannot receive a high transmission rate is used. On the other hand, an image pickup device that realizes scalability by acting as a buffer is disclosed.

Japanese Unexamined Patent Publication No. 10-290389 JP-A-2017-153156

However, when depth synthesis is performed using the image pickup apparatus described in Patent Document 2, the first image processing unit encodes and compresses the image data once to reduce the transmission rate, so that the image quality may deteriorate. is there. Further, since the size of the image used for depth composition may be extremely large, if the first image processing unit records the image once in the storage unit and then transmits the image to the second image processing unit, the image data due to the data delay occurs. Display and recording may be slow.

The present invention has been made in view of the above problems, and when depth composition is performed using a plurality of image processing units, the processing load can be reduced by changing the information transmitted between the image processing units. It is an object of the present invention to provide an image processing apparatus capable of providing an image processing apparatus.

In order to solve the above problems, the present invention comprises an acquisition means for acquiring a plurality of images, receiving signals of the plurality of images from the acquisition means, dividing the plurality of images, and dividing the plurality of images. A first processing means set in a plurality of divided regions in each, and a second processing means that receives signals of a part of the divided regions from the first processing means and generates an image of depth composition. The transmission rate at which the second processing means receives a signal from the first processing means is lower than the transmission rate at which the first processing means receives a signal from the acquiring means. To provide an image processing apparatus characterized by the above.

According to the configuration of the present invention, it is possible to provide an image processing apparatus that reduces information transmitted to an image processing unit in a subsequent stage when synthesizing depth composition using a plurality of image processing units.

It is a block diagram which shows the hardware structure of the digital camera in 1st Embodiment. It is a flowchart for demonstrating the process of depth synthesis in 1st Embodiment. It is a figure for demonstrating the area division with respect to the image in 1st Embodiment. It is a figure for demonstrating the evaluation of an image in 1st Embodiment. It is a block diagram which shows the hardware structure of the digital camera in 2nd Embodiment. It is a flowchart for demonstrating the process of depth synthesis in 2nd Embodiment. It is a figure for demonstrating the relationship between the focus position and image magnification in the 2nd Embodiment. It is a figure for demonstrating the cutout of the image in 2nd Embodiment. It is a figure for demonstrating the process of image enlargement in 2nd Embodiment. It is a block diagram which shows the hardware structure of the digital camera in 3rd Embodiment. It is a flowchart for demonstrating the process of depth synthesis in 3rd Embodiment. It is a figure for demonstrating the division of the area in 3rd Embodiment. It is a figure for demonstrating the setting of the division area according to the magnification in 3rd Embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First Embodiment)
FIG. 1 is a block diagram showing a hardware configuration of a digital camera according to the present embodiment.

The image sensor 101 converts light passing through an optical system (not shown) into an electric signal and outputs it as an image signal. For example, a CMOS (Complementary Metal Oxide Sensor) sensor or the like.

The first signal processing unit 102 receives the image signal output from the image sensor 101 and performs signal processing. The first signal processing unit 102 is composed of a plurality of processing units. The image sensor 101 and the first signal processing unit 102 are electrically connected, and an image signal is transmitted from the image sensor 101 as an electric signal over a predetermined time.

The first control unit 103 controls the first signal processing unit 102, and also controls the image sensor 101, the image signal evaluation unit 104, the area selection unit 105, and the like.

The image signal evaluation unit 104 is one of the units constituting the first signal processing unit 102, and calculates the evaluation result for the image signal received from the image sensor 101.

The area selection unit 105 is one of the units constituting the first signal processing unit 102, and based on the evaluation result obtained from the image signal evaluation unit 104, the image signal stored in the frame memory described later is stored in the frame memory. Outputs an area selection signal for selecting the area to be read.

The frame memory 106 stores the image signal read from the image sensor 101, and the image signal is read / written via the memory control unit 107 described later.

The memory control unit 107 is one of the units constituting the first signal processing unit 102, and controls to store the image signal read from the image sensor 101 in the frame memory 106.

The second signal processing unit 108 receives the image signal output from the first signal processing unit 102 and performs signal processing. The first signal processing unit 102 and the second signal processing unit 108 are electrically connected to each other, and an image signal is transmitted as an electric signal from the first signal processing unit 102 over a predetermined time.

The second control unit 109 controls the entire second signal processing unit 108, controls the depth synthesis unit 110 and the storage unit 111, which will be described later, and transmits / receives to / from the first control unit 103.

The depth synthesis unit 110 performs depth synthesis processing on the image signal output from the first signal processing unit 102. In the depth-of-field composition process, a plurality of image signals are imaged at different focus positions, image signals in at least a part of the in-focus area are cut out from each image, and the images are combined to obtain a deeper depth of field. This is a process for generating a deep image signal. The image signal generated by the depth synthesis unit 110 is stored in the storage unit 111 described later.

The storage unit 111 is a memory including, for example, a recording process represented by a DRAM (Dynamic Random Access Memory) in its configuration.

In the configuration shown in FIG. 1, the image sensor 101, the first signal processing unit 102, and the second signal processing unit 108 are electrically connected to each other. The data transmission rate from the image sensor 101 to the first signal processing unit 102 is designed to be higher than the data transmission rate from the first signal processing unit 102 to the second signal processing unit 108. .. With such a setting, when the image sensor 101 is desired to be changed, the image sensor 101 can be changed by newly incorporating a connection circuit or the like in the first signal processing unit 102. In addition, there is an advantage that the entire configuration can be easily reconstructed while maintaining the connection with the second signal processing unit 108.

FIG. 2 is a flowchart for explaining the process of depth synthesis in the present embodiment. The flow of the present embodiment will be described with reference to FIG.

In step S201, the first control unit 103 determines the number of images to be imaged according to a predetermined number of sheets or an instruction from the user.

In step S202, imaging is started based on an instruction from the user.

In step S203, the first control unit 103 drives the focus lens. By changing the relative position of the focus lens, the focus position in the optical axis direction can be changed.

In step S204, the image sensor 101 outputs a subject image imaged via an optical system (not shown) as an electric signal.

In step S205, the image sensor 101 transmits an electric signal of the subject image to the first signal processing unit 102.

In step S206, the image signal evaluation unit 104 divides the image obtained from the image sensor 101 into a plurality of regions, and calculates the contrast value for each region. The contrast value is a numerical value of the magnitude of the fluctuation width of the pixel values constituting the image signal, and is calculated by, for example, the difference value with the adjacent pixels.

FIG. 3 is a diagram for explaining the region division for the image in the present embodiment. The image 301 is an image obtained from the image sensor 101, and the region 302 is one divided region. Subjects 30 to 305 are shown in image 301. In FIG. 3, a dotted line shows the state when one image is divided into four in the horizontal direction (horizontal direction) and three in the vertical direction (vertical direction), and is divided into 12 in total. Is shown. The image signal evaluation unit 104 transmits the average value of the contrast values for each of the 12 divided regions to the region selection unit 105 as an evaluation result. In this embodiment, the contrast value is used as the evaluation index of the image signal, but there are other parameters that can be used as the evaluation index. For example, even if frequency analysis is performed for each divided region and the evaluation result is calculated using the ratio of high-frequency components included or the amplitude value of the high-frequency components as an index from the analysis result, the degree of focus is shown. It is effective as an index.

In step S207, the memory control unit 107 temporarily stores the image signal output from the image sensor 101 by performing write access to the frame memory 106.

In step S208, the area selection unit 105 selects the area. The area selection unit 105 transmits to the memory control unit 107 the area selection information for selecting the divided area of the image signal to be transmitted to the second signal processing unit 108 together with the evaluation result received from the image signal evaluation unit 104.

FIG. 4 is a diagram for explaining the evaluation of the image in the present embodiment. FIG. 4 shows captured images 401, 402, 403. Each of the images 401, 402, and 403 is divided into four horizontal divisions and three vertical divisions. The numbers written in the divided regions of each image in FIG. 4 mean that the contrast values are quantified within the range of 0 to 127, and the higher the numerical value, the higher the contrast value. In the present embodiment, as an example, it is determined whether or not the subject is in focus based on the height of the contrast value. When the area is in focus, the contrast value of the area is generally higher than when the area is out of focus. In a plurality of images having different focus positions, the same area is judged to be in focus when the contrast value is the highest.

Specifically, the image shown in FIG. 4 has the following meanings. Image 401 shows the contrast value in a state where the subject 303 is in focus, and shows a high value in the divided regions 404 to 409. Similarly, the image 402 shows the contrast value in a state where the subject 304 is in focus, and the divided regions 410 and 411 show high contrast values. Image 403 schematically shows the contrast value in a state where the subject 305 is in focus, and shows a high contrast value in the divided regions 412 to 414.

The area selection unit 105 outputs the area selection information to the memory control unit 107 so as to read the divided area having a contrast value equal to or higher than a predetermined threshold value from the frame memory 106. For example, with respect to the image of FIG. 4, when the threshold value of the contrast value is set to 50, the area selection unit 105 reads out the divided areas 404 to 414.

In step S209, the first signal processing unit 102 transmits the divided region selected by the area selection unit 105 to the second signal processing unit 108.

In step S210, it is determined whether or not the number of images has been reached. If it has been reached, the process proceeds to step S211. If not, the process returns to step S203.

In step S211 the depth synthesizing unit 110 performs the depth synthesizing process. The depth composition unit 110 creates a composite image by preferentially replacing the in-focus region selected in step S209 with the composite image. At the same time, the depth synthesizing unit 110 changes and stores the address information at the time of storing in the storage unit 111 according to the pixel position information of the image signal, so that the plurality of areas stored in the storage unit 111 are head-addressed. By reading in order from, it becomes possible to read as one image signal.

In the present embodiment, for example, the evaluation value of the region shown by the divided region 415 of the image 403 in FIG. 4 is smaller than the threshold value for all three images, so the following processing is performed. The area selection unit 105 selects the divided area determined to have never been selected at the time of processing the third image signal as the read area regardless of the threshold determination result. This makes it possible to adjust the depth synthesizing unit 110 so that the combined image does not have any omissions.

The following effects can be obtained from the processing as described above. When the data transmission rate between the image sensor 101 and the first signal processing unit 102 is higher than the data transmission rate between the first signal processing unit 102 and the second signal processing unit 108, the following processing is performed. .. The first signal processing unit 102 transmits the divided region selected by the area selection unit 105 to the second signal processing unit 108 according to the evaluation result of the image signal evaluation unit 104. By such processing, it is possible to suppress the transmission of unnecessary data from the first signal processing unit 102 to the second signal processing unit 108, and to design a highly scalable configuration.

In the present embodiment, the same divided region may be selected for a plurality of image signals depending on the evaluation result of the image signal evaluation unit 104. In this case, when the image signal selected by the first signal processing unit 102 is transmitted to the second signal processing unit 108, the evaluation result is also transmitted. Next, the better evaluation result transmitted by the depth synthesis unit 110 is finally used for depth synthesis.

Further, the area selection unit 105 may select not only the divided area having the best evaluation value but also a predetermined number of areas having higher evaluation values in descending order. By such processing, the number of divided regions selected from one image signal becomes constant, and the depth synthesizing unit 110 further selects a region actually used for the composite image in the depth synthesizing in step S211.

According to the first embodiment, among the data transmitted to the second signal processing unit, it is possible to suppress data that is not necessary for creating a depth-combined image, and perform image processing that can further reduce the processing load. Can be done.

(Second Embodiment)
The second embodiment will be described below. The second embodiment considers a case where the image magnification of the image imaged on the image sensor changes according to the position of the focus lens due to the optical characteristics of the optical system described later. The description of the same parts as in the first embodiment will be omitted.

FIG. 5 is a block diagram showing the hardware configuration of the digital camera according to the present embodiment.

The optical system 501 forms an image of light from a subject. The optical system 501 includes at least a focus lens and a focus lens control mechanism for focusing on a subject, a zoom lens and a zoom lens control mechanism for changing the focus distance, and an aperture for adjusting the amount of incident light. Have one or more.

The optical system drive unit 502 is one unit that constitutes the first signal processing unit 102, and transmits an electric signal for driving the lens group of the optical system 501. In the present embodiment, the optical system drive unit 502 generates a control signal for controlling the focus lens for focusing on the subject in the lens group of the optical system 501. Further, the optical system drive unit 502 transmits the image magnification when driving the focus lens to the cutout area determination unit 503, which will be described later.

The cutout area determination unit 503 generates area information for the cutout unit 504, which will be described later, to cut out the image output from the image sensor 101 based on the image magnification sent from the optical system drive unit 502.

The cutout unit 504 is one unit that constitutes the first signal processing unit 102, and cuts out the image output from the image sensor 101 based on the image magnification sent from the optical system drive unit 502. Perform processing. The number of pixels of the image signal output by the cutout unit 504 is smaller than the number of pixels of the image signal input to the cutout unit 504.

The enlargement unit 505 receives the image signal output to the cutout unit 504 and performs an enlargement process of the image signal. The enlargement process is a process of increasing the number of horizontal pixels and the number of vertical pixels of an image signal by a predetermined magnification. The number of pixels of the image signal output by the enlargement unit 505 is larger than the number of pixels of the image signal input to the enlargement unit 505.

FIG. 6 is a flowchart for explaining the process of depth synthesis in the present embodiment.

In step S601, the optical system drive unit 502 drives the focus lens based on the focus lens drive control signal. The optical system drive unit 502 transmits a focus lens drive control signal to the optical system 501. The optical system drive unit 502 changes the focus position by changing the relative position of the focus lens.

FIG. 7 is a diagram for explaining the relationship between the focus position and the image magnification in the present embodiment. The curve 701 in FIG. 7 shows how the image magnification changes as the focus position changes. FIG. 7 shows how three images are captured at focus positions 702, 703, and 704, respectively.

In step S602, the cutout area determination unit 503 determines the cutout area of the image. The cutout area determination unit 503 determines the cutout area based on the image magnification when the image received from the first signal processing unit 102 is captured. For example, the image magnifications 705, 706, and 707 at the focus positions 702, 703, and 704 shown in FIG. 7 are set to αn (n = 0, 1, 2). In the following description, the image magnification 705 is α0, the image magnification 706 is α1, and the image magnification 707 is α2. The cutout area selection unit 105 calculates the cutout start position and the number of cutout pixels when the image magnification is αn (n = 0, 1, 2).

FIG. 8 is a diagram for explaining the cutting out of the image in the present embodiment. Image 801 is an image captured at image magnification α0, image 802 is an image captured at magnification α1, and image 803 is an image captured at image magnification α2. In this way, as the image field ratio increases, the area in which the subject is projected becomes narrower, and when performing depth composition, it is possible to combine images based on the image with the highest image magnification (the narrowest angle of view). desirable. Therefore, some angles of view of images other than the image having the highest image magnification are not used for the composite image. The cutout area selection unit 105 cuts out only the angle of view used for the composite image. The cutout area selection unit 105 calculates the relative ratio (image magnification ratio: βn) between the other image and the image signal having the highest image magnification, and determines the cutout start position and the number of cutout pixels based on the relative ratio (image magnification ratio: βn). In the present embodiment, the cutout area selection unit 105 calculates based on the following (Equation 1).

The cutout start position and the number of cutout pixels are obtained based on the image magnification ratio βn obtained by the cutout area selection unit 105. In the present embodiment, it is represented by a zero origin Cartesian coordinate system with the upper left pixel as the origin. Further, the horizontal direction is defined as the x-coordinate and the vertical direction is defined as the y-coordinate. Then, the x-coordinate of the cutting start position can be calculated by the following (Equation 2).

The y-coordinate of the cutting start position can be calculated by the following (Equation 3).

It should be noted that each of pixelH and pixelV represents the number of horizontal pixels and the number of vertical pixels in the effective region of the image signal.

Further, the cutout area selection unit 105 calculates the number of cutout horizontal pixels pixelH'and the number of cutout vertical pixels pixelV' by the following (Equation 4) and (Equation 5).

With respect to the images 801, 802, and 803 of FIG. 8, the cutout areas determined by the cutout area selection unit 105 are the areas 804, 805, and 806. Since the image magnification of the image 803 is the maximum, the image 803 has the same size as the cutout area 806 and the image 803 determined by the cutout area selection unit 105.

The relationship between the focus position and the image magnification as shown in FIG. 7 may be held in a discrete table in a storage unit (not shown) inside the optical system drive unit 502. Alternatively, it may be held as a discrete table in a storage unit (not shown) inside the first control unit 103 and transmitted. Further, when the relationship between the focus position and the image magnification can be approximated by a polynomial or the like, only the coefficient of the polynomial may be stored in advance in a storage unit (not shown).

In step S603, the cutting section 504 cuts out the cutting area determined by the cutting area determining section 503. When pixels outside the cutout area determined by the cutout area selection unit 105 are required in the enlargement process described later, the cutout unit 504 cuts out the image signal having the lowest image magnification so as to output the corresponding signal. By doing so, it is possible to improve the image quality.

In step S604, the enlargement unit 505 performs the enlargement process. The enlargement unit 505 receives the image magnification from the first signal processing unit 102 and increases the size of the image according to a predetermined enlargement ratio.

FIG. 9 is a diagram for explaining an image enlargement process in the present embodiment. Image 901 shows a state of enlarging an image captured at an image magnification α0, image 902 shows a state of enlarging an image captured at an image magnification α1, and image 903 shows a state of enlarging an image captured at an image magnification α2. It is shown. Images 901 to 903 do not have the same number of horizontal and vertical pixels, and cannot be combined as they are. The enlargement unit 505 calculates the enlargement ratio γn based on the image magnification ratio βn of each image according to the following (Equation 6), and performs the enlargement processing.

Since the image 903 has the largest image magnification, it is not necessary to perform the enlargement processing.

Although the enlargement ratio γn at the time of enlargement has been described as being calculated from the image magnification ratio βn, as another calculation method, for example, the enlargement is based on the number of horizontal pixels or the number of vertical pixels of each image signal. The rate γn can also be calculated.

In step S206, as in the first embodiment, the image signal evaluation unit 104 calculates the contrast value of the image output from the enlargement unit 505 and transmits it to the area selection unit 105.

In the above description, the size of the subject of each image is made uniform by the enlargement processing in the enlargement unit 505, but in the present embodiment, the size of the subject can be made uniform by the reduction processing. , The same effect can be obtained.

The following effects can be obtained from the processing as described above. When the data transmission rate between the image pickup element 101 and the first signal processing unit 102 is higher than the data transmission rate between the first signal processing unit 102 and the second signal processing unit 108, the following Can be processed. After reducing the number of pixels of the image obtained from the image sensor 101 based on the image magnification, enlargement processing necessary for depth synthesis is performed to adjust the size of the image of the subject of the image. The first signal processing unit 102 selects and transmits data used for depth synthesis processing, thereby suppressing the influence of fluctuations in the image magnification from the first signal processing unit 102 to the second signal processing unit 108. The data to be transmitted can be suppressed.

According to the second embodiment, among the data transmitted to the second signal processing unit, it is possible to suppress data that is not necessary for creating a depth-combined image in consideration of the image magnification, and the processing load is further reduced. Can perform image processing that can be performed.

(Third Embodiment)
The third embodiment is different from the first embodiment and the second embodiment, and is an embodiment for reducing variations in the size of the subject included in the divided region between the images and variations in the evaluation result. ..

Hereinafter, the third embodiment will be described. The description of the same parts as those of the first embodiment and the second embodiment will be omitted.

FIG. 10 is a block diagram showing a hardware configuration of a digital camera according to the present embodiment. The image signal evaluation unit 1001 changes the evaluation range of the image signal obtained from the image sensor 101 based on the image magnification sent from the optical system drive unit 502.

The reduction unit 1002 performs reduction processing on the image of the divided region transmitted from the first signal processing unit 102. The reduction process is a process of reducing the number of horizontal pixels and the number of vertical pixels of an image signal by a predetermined magnification. The number of pixels of the image signal output from the reduction unit 1002 is smaller than the number of pixels of the input image signal.

FIG. 11 is a flowchart for explaining the process of depth synthesis in the present embodiment.

In step S1101, the image signal evaluation unit 1001 evaluates the image signal. Similar to the first embodiment, the image signal evaluation unit 1001 divides the image obtained from the image sensor 101 into a plurality of regions, and calculates a contrast value for each of the plurality of regions.

FIG. 12 is a diagram for explaining the division of the region in the present embodiment. FIG. 12 shows how the divided areas 1202 and 1203 are set for the image 1201. In the first embodiment, the divided regions are described so as not to overlap each other, but in the present embodiment, it is shown that the divided regions 1202 and 1203 overlap.

If the division area is set in the same range for each image to be combined, an error will occur in the evaluation result of each division area due to the change in the magnification of the image signal due to the change in the image magnification caused by the optical system 501. There is. Therefore, in the present embodiment, in order to reduce the error, the flexibility with respect to the change in the image magnification is ensured by setting the divided region so as to have the overlapping region according to the fluctuation amount of the image magnification.

In step S1102, the area selection unit 105 stores the area selection information for selecting the division area of the image signal to be transmitted to the second signal processing unit 108 based on the evaluation result received from the image signal evaluation unit 1001. It is transmitted to the control unit 107. The area selection unit 105 selects an area based on the contrast value of each divided area.

In step 1103, the reduction unit 1002 performs the reduction process. The optical system drive unit 502 sends the image magnification to the first control unit 103, the first control unit 103 calculates the reduction ratio based on the image magnification, and transmits it to the second control unit 109. The reduction unit 1002, which has received the image division area output from the first signal processing unit 102, receives the reduction ratio for performing the reduction processing from the second control unit 109, and reduces the image signal. I do.

The first control unit 103 calculates the reduction ratio εn based on the following (Equation 7).

The image magnification can be obtained by the second control unit 109 without going through the first control unit 103 by the following method. The area selection unit 105 transmits the image magnification from the image signal evaluation unit 1001 to the memory control unit 107 together with the area selection information. When the data in the divided region is transmitted from the first signal processing unit 102 to the second signal processing unit 108, the image magnification is added and transmitted, so that the reduction unit 1002 can obtain the image magnification.

The storage unit 111 stores the divided area after the reduction process via the depth synthesis unit 110. Subsequent processing of this embodiment is the same as that of the first and second embodiments. However, since an area that overlaps each divided area read from the storage unit 111 is included, it is necessary to appropriately delete the area when storing.

In the above description, the size of the subject of each image is made uniform by the reduction process in the reduction unit 1002, but in the present embodiment, the size of the subject can be made uniform by the enlargement process. , The same effect can be obtained.

The following effects can be obtained from the processing as described above. When the data transmission rate between the image pickup element 101 and the first signal processing unit 102 is higher than the data transmission rate between the first signal processing unit 102 and the second signal processing unit 108, the following Can be processed.

The image signal evaluation unit 1001 can suppress the influence of the image magnification change caused by driving the optical system 501 by changing the division region based on the image magnification obtained from the optical system drive unit 502. At the same time, it is possible to suppress the data transmitted from the first signal processing unit 102 to the second signal processing unit 108.

In this embodiment, the image signal evaluation unit 1001 may change the size of the divided region according to the image magnification obtained from the optical system driving unit 502. FIG. 13 is a diagram for explaining the setting of the divided region according to the image magnification in the present embodiment. FIG. 13 shows how the division area is set for the three images 1301, 1302, and 1303. Image 1301 is an image captured at an image magnification α0, image 1302 is an image captured at an image magnification α1, and image 1303 is an image captured at an image magnification α2. The area separated by the dotted line shown in FIG. 13 indicates one divided area, and it is shown that the horizontal size and the vertical size of one divided area are changed according to the image magnification. ing. When the divided area of the image 1303 is evenly divided into four horizontal divisions and three vertical divisions, the horizontal size of the divided area of the image 1301 is set according to the ratio of the image magnification based on the divided area of the image 1303 having the largest image magnification. Make it smaller. Even if the size of the image of the subject included in the image changes, the area being evaluated does not change between the images, so that the reliability of the evaluation result can be improved.

According to the third embodiment, among the data transmitted to the second signal processing unit, it is possible to suppress the data that is not necessary for creating the image of the depth composition in consideration of the variation in the evaluation between the images. Image processing that can reduce the processing load can be performed.

(Other embodiments)
Although the above embodiment has been described based on the implementation using a digital camera, the present embodiment is not limited to the digital camera. For example, it may be carried out by a portable device having a built-in image sensor, or a network camera capable of capturing an image. Further, the captured image may be operated on a processing device other than the digital camera by the method described above.

In the present invention, a program that realizes one or more functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device program. It can also be realized by the process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Image sensor 102 First signal processing unit 103 First control unit 104 Image signal evaluation unit 105 Area selection unit 106 Frame memory 107 Memory control unit 108 Second signal processing unit 109 Second control unit 110 Depth synthesis unit 111 Storage unit 501 Optical system 502 Optical system drive unit 503 Cutout area determination unit 504 Cutout unit 505 Enlargement unit 1001 Image signal evaluation unit 1002 Reduction unit

Claims (19)

An acquisition method for acquiring multiple images,
A first processing means that receives signals of the plurality of images from the acquisition means, divides the plurality of images, and sets a plurality of divided regions in each of the plurality of images.
It has a second processing means that receives a signal of a part of the divided regions from the first processing means and generates an image of depth composition.
An image processing apparatus characterized in that the transmission rate at which the second processing means receives a signal from the first processing means is lower than the transmission rate at which the first processing means receives a signal from the acquisition means. The first aspect of claim 1, wherein the first processing means evaluates the divided region in each of the plurality of images and selects the divided region based on the result of the evaluation. Image processing device. The image processing apparatus according to claim 2, wherein the first processing means calculates an evaluation value for each of at least a part of the divided regions in the evaluation. The image processing apparatus according to claim 3, wherein the second processing means receives a signal of the divided region selected by the first processing means. The image processing apparatus according to claim 4, wherein the second processing means receives a signal in the divided region whose evaluation value is equal to or higher than a predetermined threshold value. The image processing apparatus according to claim 4, wherein the second processing means receives a predetermined number of signals of the divided region in descending order of the evaluation values. The image processing apparatus according to any one of claims 2 to 6, wherein the first processing means calculates a contrast value of the divided region and performs the evaluation based on the contrast value. The invention according to any one of claims 2 to 6, wherein the first processing means performs frequency analysis on the divided region and performs the evaluation based on the result of the frequency analysis. Image processing equipment. The second processing means receives at least one of the divided regions that is not selected as the first processing means among the plurality of divided regions at the same position. The image processing apparatus according to any one of claims 2 to 8. The image processing apparatus according to any one of claims 1 to 9, wherein the first processing means sets the divided regions so as to overlap each other. The image processing apparatus according to any one of claims 1 to 10, wherein the first processing means performs enlargement or reduction processing on the image. The image processing apparatus according to claim 11, wherein the first processing means performs the enlargement or reduction processing based on the information of the optical system in which the image is captured. The image processing apparatus according to claim 11, wherein the first processing means changes the size of the divided region based on the information of the optical system in which the image is captured. The image processing apparatus according to claim 12, wherein the information of the optical system includes a focus position when the image is captured. The image processing apparatus according to claim 12, wherein the information of the optical system includes a position of a focus lens when the image is captured. The image processing apparatus according to any one of claims 1 to 15, wherein the first processing means performs a cropping process on the plurality of images. An imaging means that captures multiple images and
A first processing means that receives signals of the plurality of images from the imaging means, divides the plurality of images, and sets a plurality of divided regions in each of the plurality of images.
It has a second processing means that receives a signal of a part of the divided regions from the first processing means and generates an image of depth composition.
An imaging device characterized in that the transmission rate at which the second processing means receives a signal from the first processing means is lower than the transmission rate at which the first processing means receives a signal from the imaging means. An acquisition step in which the acquisition method acquires multiple images,
A first processing step in which the first processing means receives signals of the plurality of images from the acquisition means, divides the plurality of images, and sets each of the plurality of images in a plurality of divided regions. When,
The second processing means has a second processing step of receiving a signal of a part of the divided regions from the first processing means and generating an image of depth composition.
An image processing method characterized in that the transmission rate at which the second processing means receives a signal from the first processing means is lower than the transmission rate at which the first processing means receives a signal from the acquisition means. A computer program that operates an image processing device on a computer.
An acquisition step in which the acquisition method acquires multiple images,
A first processing step in which the first processing means receives signals of the plurality of images from the acquisition means, divides the plurality of images, and sets each of the plurality of images in a plurality of divided regions. When,
The second processing means receives a signal of a part of the divided regions from the first processing means from the first processing means, and performs a second processing step of generating an image of depth composition.
A program characterized in that the transmission rate at which the second processing means receives a signal from the first processing means is lower than the transmission rate at which the first processing means receives a signal from the acquisition means.

Images