JP2015228113A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor capable of suppressing a change in a blurred shape of the background of segmented image data even in the case that focal distance changes during photographing.SOLUTION: The image processor includes segment means for segmenting a portion of first image data to generate second image data, and filter processing means for performing filter processing for giving blur to the second image data. The filter processing means sets a filter coefficient to be used for filter processing in accordance with a change in focal distance of an optical system used during imaging for generating first image data.

Description

  The present invention relates to a technique for cutting out a partial area of image data.

  With the increase in the number of pixels in the imaging device, it is possible to generate high-resolution image data such as 8K (7680 × 4320 pixels) Super Hi-Vision. On the other hand, full-high-definition (Full Hi-Vision) is the mainstream of home televisions that are output devices, so it is necessary to process image data in order to output a super high-definition image to a full-high-definition output device. is there. As a method for this, for example, a method of reducing the entire angle of view of Super Hi-Vision image data to the size of Full Hi-Vision, or a method of cutting out an area having the same size as Full Hi-Vision from Super Hi-Vision image data.

  For example, Patent Document 1 discloses a technique of setting a main subject within an angle of view even if the main subject moves by setting a cutout region according to the tracking result of the main subject from the original image data. . In addition to setting the cutout area and the resizing rate in accordance with the size of the main subject included in the image data as well as the cutout area, the main subject can always be set to an arbitrary size. If this technology is used, partial image data in which the main subject is always arranged in an arbitrary size and at an arbitrary position regardless of the camera work (zoom operation or panning operation) of the imaging device when generating the original image data. Can be generated.

JP 2000-261657 A

  Here, if the focal length of the imaging device is changed while a moving image is being captured, the position and size of the subject included in the extracted partial image data can be made constant, but the background of the partial image data There arises a problem that the degree of blur changes.

  FIG. 8 is a diagram for explaining a change in the degree of background blur, and shows a case where the focal length of the imaging apparatus is changed from the tele side to the wide side.

  If there is no change in the difference between the distance between the subject and the background, the longer the focal length, the more the image data is blurred. Therefore, as shown in FIG. 8, when the focal length of the imaging apparatus is changed from the tele side to the wide side, the degree of background blur gradually decreases, so that the sharpness of the background of the clipped partial image data gradually increases. Will increase.

  When displaying the original image data generated by the imaging device on the display device, not only the degree of blurring of the background but also the change of the angle of view is caused by the change of the focal length from the tele side to the wide side. It is hard to give the observer a sense of incongruity. However, when the partial image data cut out in order to keep the position and size of the subject constant is displayed on the display device, the background blur is not affected even though the position and size of the subject have not changed. Since it changes, it becomes easy to give a strange feeling to an observer.

  Therefore, there is a demand for an image processing apparatus that can suppress a specific change in background blur of cut out image data even when the focal length changes during shooting.

  In order to solve the above-described problem, the present invention provides a cut-out unit that cuts out a part of first image data to generate second image data, and a filtering process for adding blur to the second image data. The filter processing means sets the filter coefficient used for the filter processing according to the change in the focal length of the optical system used at the time of imaging for generating the first image data. An image processing apparatus characterized by the above is provided.

  According to the present invention, it is possible to provide an image processing apparatus that can suppress a specific change in the background blur of cut out image data even when the focal length changes during shooting.

1 is a system configuration diagram of an image processing apparatus according to a first embodiment of the present invention. It is a figure for demonstrating the process which produces | generates the 3rd image data which is output image data from the 1st image data which is original image data. It is a figure which shows the relationship between a subject distance and the sharpness of an image, and a figure which shows the relationship between a subject distance and the cutoff frequency of a filter process. 3 is a flowchart for explaining the operation of the image processing apparatus according to the first embodiment. It is a system configuration | structure figure of the image processing apparatus concerning the 2nd Embodiment of this invention. 10 is a flowchart for explaining the operation of the image processing apparatus according to the second embodiment. It is a figure showing a frequency on a horizontal axis and an amplitude of a filter on a vertical axis. It is a figure for demonstrating the change of the blur condition of a background.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a system configuration diagram of an image processing apparatus according to the first embodiment of the present invention.

  In this embodiment, the focal length varies from the tele side to the wide side at the time of shooting, and the optical conditions at the time of shooting are recorded together with the image.

  In FIG. 1, a cutout unit 101 cuts out a part of an image from first image data that is original image data having a large number of pixels, and sets the cutout image data to a predetermined size (number of pixels). Resizing processing is performed to generate second image data. The first image data may be obtained via a recording medium or a network, or the image processing apparatus has an imaging unit, and the image data obtained by photographing with the imaging unit is the first image data. It may be used as image data.

  The area dividing unit 102 divides the second image data output from the cutout unit 101 based on the distance information. This distance information is acquired from the imaging apparatus, and indicates distance information to the subject at a plurality of points within the angle of view when the first image data is generated. If the imaging device is equipped with a phase difference AF sensor or an image sensor having the function of a phase difference AF sensor is used, distance information at the distance measurement point corresponding to the AF sensor arrangement is acquired. Therefore, area division is performed using this distance information. For example, a hue is first obtained for each pixel of the second image data, and adjacent pixels that are similar in hue are grouped. Furthermore, by comparing the corresponding distance information between the groups, and combining the groups in which the corresponding distance information is close and the distance between the groups is equal to or less than the threshold, it is possible to divide into regions for each subject. it can.

  Also, if the imaging device is configured to perform a scanning operation for AF, the subject distance in each region (pixel) within the angle of view can be acquired based on the contrast value. Can be divided into regions for each subject.

  Alternatively, the subject distance can be acquired by comparing the edge strengths of the focused image and the non-focused image. For example, when comparing two image data picked up by changing the in-focus position, the edge intensity in the in-focus area changes abruptly, but the edge intensity in the non-in-focus area does not change much. Accordingly, distance information of each region can be acquired based on the amount of change in edge strength. However, the means for acquiring distance information is not limited in the present invention.

  The filter coefficient setting unit 103 determines a filter coefficient for each region divided by the region dividing unit 102 using distance information of the input image and optical information at the time of photographing such as an aperture value and a focal length.

  The filter processing unit 104 performs filter processing, which will be described later, on each region divided by the region dividing unit 102 using the filter coefficient determined by the filter coefficient unit 103, and outputs third image data as output image data. Generate.

  The memory 110 holds first image data 105 that is original image data, distance information 106, optical information 107, and third image data 108 that is output image data, and each processing unit via a memory interface 109. And data transfer. The memory 110 may be an internal memory provided in the image processing apparatus or an external memory accessible via the memory interface 109.

  FIG. 2 is a diagram for explaining processing for generating third image data that is output image data from first image data that is original image data. In FIG. 2, the second image data 401 is generated by cutting out and resizing an area surrounded by a dotted line in the first image data 410.

  It is assumed that the second image data 401 is divided into a region A including a main subject, a region B including a house, a region C including a tree, and another region D by the region dividing unit 102. The imaging apparatus focuses on the area A, and the distance information becomes longer in the order of the area B, the area C, and the area D.

  The image data 402 represents second image data when the optical system is set to the tele side, and the image data 403 is a third image generated by subjecting the image data 402 to filtering by region. Image data is shown.

  The image data 404 indicates second image data when the optical system is set to the wide side, and the image data 405 is generated by subjecting the image data 404 to filter processing for each region. 3 shows image data. Since the focal length when the image data 402 is generated is longer than the focal length when the image data 404 is generated, the image data 402 is more blurred in the regions B, C, and D.

  FIG. 3A is a diagram showing the relationship between the subject distance and the sharpness of the image. If the subject distance matches the distance to the in-focus surface, the sharpness of the image is high (the amount of blur is small), and the sharpness decreases (the amount of blur is large) away from the in-focus surface. FIG. 3B is a diagram illustrating the relationship between the subject distance when the focal length changes from the tele side to the wide side and the cutoff frequency of the filter processing performed by the filter processing unit 104.

  When the focal length is changed from the tele side to the wide side, the sharpness of the background area of the image increases as shown in FIG. 3A. Therefore, in order to keep the blur of the background area constant, the sharpness of the image Filter characteristics that lower the degree are required. Therefore, as shown in FIG. 3B, filter characteristics are set such that the farther away from the in-focus plane, the lower the cut-off frequency for the in-focus area. In other words, the filter characteristic is such that the area near the in-focus plane has a small amount of blur and the area away from the in-focus plane has a large amount of blur. In addition, since the depth of field is related to the degree of blur, the characteristics may be changed according to the F value.

  Next, the operation of the image processing apparatus according to the present embodiment will be described with reference to the flowchart shown in FIG.

  In step S401, the cutout unit 101 reads the original image data 105 stored in the memory 110 via the memory interface 109 as first image data.

  In step S402, the cutout unit 101 performs cutout processing and resizing processing on the input first image data to generate second image data. At this time, the cutout unit 101 performs each process at the cutout position and the resizing ratio so that the size of the subject is constant. Alternatively, an instruction to designate the main subject may be given from the outside, and a cutout position where the designated subject is the center of the angle of view and a resizing rate at which the subject becomes a predetermined size may be set. That is, when the focal length changes, the size of the subject with respect to the size of the original image data changes, so that the cutout position and the resizing ratio are changed so as to cancel this.

  In step S <b> 403, the region dividing unit 102 reads the distance information 106 obtained when the original image data 105 is captured, and performs region division on the second image data generated by the clipping unit 101.

  In step S404, the filter coefficient setting unit 103 reads the distance information 106 read from the memory 110 and the optical information 107 including the information on the F-number of the aperture and the information on the focal length, and cuts the filter process for each background region. Determine the off frequency. As shown in FIG. 3B, the area A is a focal plane, and the subject distance becomes longer in the order of the area B, the area C, and the area D. Therefore, the filter is set so that the cut-off frequency of the area A becomes the highest. The coefficient fc [A] is set. Then, the cut-off frequencies fc [B], fc [C], and fc [D] for each background region are set so that the cut-off frequencies decrease in the order of region B, region C, and region D. As the cut-off frequency becomes smaller, the image processing in which the degree of blur becomes larger is performed by the filter processing by the subsequent filter processing unit 104. Further, in order to suppress the difference in blurring of the background due to the difference in focal length, the filter coefficients H [B] and H [C are set such that the cutoff frequency with respect to the background region gradually decreases as the focal length approaches the wide end. ] And H [D] are set. Of course, if the F value of the aperture that specifically affects the background blur is changed, it is desirable to change the filter coefficient so as to suppress the influence of the change in the F value.

  In step S <b> 405, the filter processing unit 104 performs a filter process on each divided area according to the filter characteristics set by the filter coefficient setting unit 103. The image data 402 and the image data 404 in FIG. 2 are both the second image data. However, since the image data 404 is captured when the focal length is on the wide side, the image data 404 is the image data. The background area is less blurred than the data 402.

  However, as described above, the filter coefficient setting unit 103 sets the filter coefficient corresponding to the focal length so as to suppress the degree of blur due to the difference in focal length. Therefore, image data 403 that is third image data obtained by filtering the image data 402 and image data 405 that is third image data obtained by filtering the image data 404. The blur of the background area is almost equal.

  If the filter coefficients are completely separated for each area, the change in image quality may be noticeable at the boundary of the area. To make this inconspicuous, the filter coefficient for the boundary is weighted and added to the filter coefficient for each area. You may make it use what was done.

  In step S507, the third image data is output to the memory 110 as output image data 108 via the memory interface 109.

  In the present embodiment, the case where the focal length is changed from the tele side to the wide side has been described as an example, but the reverse may naturally be applied. When the focal length changes from the wide side to the tele side, a filter coefficient is set that increases the sharpness of the image in the background area (reduces the degree of blur) according to the change in the focal length.

  As described above, according to the present embodiment, the focal length of the original image data is reduced by performing the filtering process using the filter coefficient corresponding to the distance and the focal length from the focal plane for each region in the image. Even if it changes, the degree of blurring of the background area of the cut-out image data can be stabilized.

(Second Embodiment)
FIG. 5 is a system configuration diagram of an image processing apparatus according to the second embodiment of the present invention. Also in this embodiment, it is assumed that the focal length varies from the tele side to the wide side during shooting.

  In the image processing apparatus of FIG. 5, the same components as those of the image processing apparatus of FIG. The image processing apparatus in FIG. 5 differs from the image processing apparatus in FIG. 1 in that it includes a frequency analysis unit 511 and a filter coefficient setting unit 503 instead of the filter coefficient setting unit 103.

  The frequency analysis unit 511 receives the second image data divided into regions and performs frequency analysis for each region to calculate a frequency component. The calculated frequency component for each region is input to the filter coefficient setting unit 503.

  Next, the operation of the image processing apparatus according to the present embodiment will be described with reference to the flowchart shown in FIG. Of the steps described in FIG. 6, steps denoted by the same reference numerals as the steps described in FIG. 4 perform the same processing as in the first embodiment, and thus description thereof is omitted. This embodiment is different from the first embodiment in that steps S601 and S602 are executed instead of step S404 in FIG.

  In step S601, the frequency analysis unit 511 first performs frequency conversion for each divided region on the second image data when the focal length is on the tele side, and is most frequently included in each region based on the frequency characteristics. The band (excluding the DC component) is calculated. If this image processing apparatus has an imaging unit, the focal length is moved to the tele side and control is performed so as to generate image data. Good. As a frequency conversion means, there is a method in which each region is divided into blocks of N × N pixels, discrete cosine transform (DCT) is performed on each block, and a frequency band having the largest sum of absolute values of DCT coefficients is obtained. Can be mentioned.

  In step S602, the filter coefficient setting unit 503 cuts off the cut-off frequencies fc [B] and fc [for each of the regions B, C, and D so that the frequency band that is contained most in each region is attenuated by a predetermined amount. C] and fc [D] are set.

  FIG. 7 is a diagram in which the horizontal axis represents frequency and the vertical axis represents filter amplitude, and is a diagram illustrating the relationship between the frequency band most contained in each background region and the cutoff frequency. In the present embodiment, a frequency that attenuates by a predetermined amount (3 dB in the example of FIG. 7) from the DC component is set as a cutoff frequency.

  As described above, according to the present embodiment, even if the focal length of the original image data changes, the frequency characteristic is obtained for each region of the image and the filter process is performed using the filter coefficient based on the frequency characteristic. In addition, it is possible to stabilize the blur condition of the background area of the cut-out image data.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 101 Cutout part 102 Area division part 103,503 Filter coefficient setting part 104 Filter processing part 109 Memory interface 110 Memory 511 Frequency analysis part

Claims (12)

Clipping means for cutting out part of the first image data and generating second image data;
Filter processing means for performing a filter process for adding blur to the second image data;
The filter processing unit sets a filter coefficient used for the filter processing according to a change in a focal length of an optical system used at the time of imaging for generating the first image data. Processing equipment. Area dividing means for dividing the second image data into a plurality of areas;
The image processing apparatus according to claim 1, wherein the filter processing unit sets the filter coefficient for each divided region.   3. The image processing according to claim 2, wherein the region dividing unit divides the second image data into a plurality of regions based on subject distance information obtained for the second image data. apparatus.   4. The image according to claim 1, wherein the filter processing unit sets the filter coefficient so that the degree of blur becomes stronger in a region far from the in-focus plane. 5. Processing equipment.   5. The image according to claim 1, wherein the filter processing unit sets the filter coefficient so that a degree of blur increases as a focal length of the optical system becomes shorter. Processing equipment.   The image processing apparatus according to claim 1, wherein the filter processing unit sets the filter coefficient according to a diaphragm of the optical system. Frequency analysis means for analyzing frequency characteristics for each of the divided areas;
The image processing apparatus according to claim 2, wherein the filter processing unit sets the filter coefficient according to a frequency characteristic analyzed by the frequency analysis unit.   The frequency analysis means analyzes frequency characteristics for each of the divided areas in second image data generated from image data obtained when the optical system has a predetermined focal length. The image processing apparatus according to claim 7.   The image processing apparatus according to claim 7, wherein the frequency analysis unit obtains a frequency band that is contained most in each divided area.   The cutout unit performs cutout processing and resizing processing so that the size of a subject included in the first image data is constant, and generates the second image data. The image processing apparatus according to item 1. A cutting-out step of cutting out part of the first image data to generate second image data;
A filtering process for performing a filtering process for providing blur to the second image data;
In the filter processing step, a filter coefficient used for the filter processing is set according to a change in a focal length of an optical system used at the time of imaging for generating the first image data. Processing method.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 10.

Images