JP6622546B2 - Image processing apparatus and method - Google Patents

Description

  The present invention relates to an image processing apparatus and method such as a digital camera or a smartphone capable of enlarging and displaying a part of a captured image or a reproduced image, for example.

  There is an image processing apparatus such as a digital camera having a function capable of enlarging and displaying an image in order to confirm an in-focus state at the time of capturing an image, confirming reproduction after capturing. In a conventional image processing apparatus, an enlarged display of a part of a rectangular area centering on the in-focus position detected by the apparatus, and an image are equally divided (for example, 9 × 3 × 3, 4 × 16). There is an image processing apparatus that enlarges and displays a rectangular area including the highest frequency component for each divided area.

Japanese Patent Laid-Open No. 2004-186872

  In the enlarged display as in the conventional image processing apparatus, the enlarged display is not performed according to the subject, so even if only the enlarged result is viewed, only a part of the subject is displayed. In some cases, the enlarged display is difficult to recognize what the subject is.

The present invention has been made in view of the above conventional example, and an object thereof is to provide an image processing apparatus and method capable of adaptively enlarging and displaying a photographed image or a photographed image including a focus position. And

In order to achieve the above object, an image processing apparatus according to the present invention includes means for specifying a focus position in image data including distance information between a shooting position and a subject, and the subject at the focus position based on the distance image. Specifying means for specifying an area including the enlarged area and display means for displaying the enlarged area in an enlarged manner, the specifying means, based on the distance image, the distance of the subject at the in-focus position, An area including a subject with a distance such that the difference between them is within a predetermined value is further specified as the enlarged area.

  According to the present invention, a captured image or a captured image can be adaptively enlarged and displayed including the in-focus position.

Configuration diagram of image processing device (digital camera) Diagram showing image processing engine Captured image enlargement process flowchart Region division processing flowchart Explanatory drawing of area division processing Enlarged area determination process flowchart Sub-area determination processing flowchart The figure which shows the enlarged display example 1 Diagram showing enlarged display example 2 Diagram showing enlarged display example 3 The figure which shows the enlarged display example 4 Playback image enlargement process flowchart

  EMBODIMENT OF THE INVENTION Below, the form for implementing this invention with reference to an accompanying drawing is demonstrated in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed according to the configuration of the apparatus to which the present invention is applied and various conditions. The present invention is described below. It is not limited to the form.

[Example 1]
FIG. 1 shows a configuration diagram of a digital camera which is an example of an image processing apparatus of the present invention.
A control unit (CPU) 101 controls the entire digital camera. The lens 102 is a lens that can change the focal length and the state of the diaphragm. The image sensor 103 is a CMOS sensor, for example, and has a function of converting light imaged by the lens 102 into an electrical signal. It is a sensor that can also be acquired. For example, a sensor in which a Z pixel that can acquire distance information by a ToF (time-of-flight) method is stacked below a RGB pixel is known. Information in which distance information is associated with each pixel is referred to as distance map information or distance image information.
The signal processing unit 104 has a function of performing predetermined signal processing such as white balance and gamma correction on the input moving image signal and storing the image data in the memory 109 via the bus 110. At the same time, the distance information has a function of storing in the memory 109 distance image information composed only of distance information values. Further, the digital camera of this embodiment also has a focus position acquisition function for determining which position in the image is the focus position.
The JPEG encoding / decoding unit 105 has a function of encoding image data stored in the memory 109 into compressed image data in JPEG format, and conversely decoding JPEG image data held on the memory 109. .
The memory card controller 106 has a function of recording various data stored in the memory 109 on the memory card 107 and conversely reading various data recorded on the memory card 107 onto the memory 109. Various files can be recorded and read through the file system processed by the control unit 101.
The flash memory 108 is used for storing a program operated by the control unit 101 and storing various setting data, and the data recorded in the device is retained even when the power of the apparatus is turned off.
The memory 109 is a large-capacity memory capable of holding data such as image data as described above.
The bus 110 has a function of transmitting a command from the control unit 101 to each block and transmitting data between the memory 109 and each block.
The image processing engine 111 has a function of performing various image processing on the image data held on the memory 109 at high speed by setting and control from the control unit 101. For example, as shown in FIG. 2, a development processing unit 201, a distance information generation unit 202, an area division processing unit 203, an enlargement / reduction processing unit 204, a trimming processing unit 205, a face detection unit 206, an area determination processing unit 207, and other various types. It consists of a processing unit. These processing units may be configured by dedicated hardware for each processing, as in this embodiment, an image processing processor dedicated to image processing, software operating on the processor, or a control unit 101 and software operating on 101 may be used. In this embodiment, it is configured by dedicated hardware, and each processing unit is configured to be capable of parallel processing simultaneously.
The display unit 112 has a function as a display device for the decoded image data, and also has a function as a viewfinder at the time of shooting. Further, it also has a function as an operation screen when the user operates the apparatus.

<Image enlargement function>
Next, the processing content of the image enlargement function near the in-focus position at the time of imaging, which is a feature of the image processing apparatus of the present embodiment, will be described with reference to the flowchart of FIG. 3 are executed by the image processing engine 111 shown in FIG. 2, but may be executed by the control unit 101 as described above.
When enlargement of an image (referred to as a noticed image or a noticed frame) is requested by a user instruction at the time of imaging, the captured image enlargement process in step S301 is started. Note that the image of interest is an image being captured, and may be a still image or a frame of a moving image.
In step S <b> 302, image data (RAW data) and distance information acquired from the image sensor 103 are stored in the memory 109 via the signal processing unit 104. The image data acquired here is also called image data of interest.
In step S303, the signal processing unit 104 acquires the in-focus position for the image data acquired in step S302. The in-focus position is a position on the subject whose distance from the shooting position is equal to the focal length. The in-focus position can be specified, for example, by dividing an image into small regions, and among regions where the spatial frequency exceeds the threshold, the region with the highest spatial frequency can be specified as the region including the in-focus position. The in-focus position specified at the time of shooting may be used in this step. Alternatively, a pixel or region having a distance equal to the current focal length may be specified based on the distance image, and this may be specified as the in-focus position.
In step S304, development processing such as color conversion processing or noise reduction processing is performed on the image data (RAW data) stored in the memory 109 by the development processing unit 201 of the image processing engine 111. By this processing, for example, a still image (photograph) as shown in FIG. 8A is generated.
Step S305 is a face detection process. A face in the still image is detected by the face detection unit 206 of the image processing engine 111 for the still image stored in the memory 109. When a face is detected, information such as the number of faces and the image position (coordinates) and size of each detected face is temporarily stored in the memory 109.
In parallel with the processes in steps S304 and S305, the processes in steps S306 and S307 are executed. In step S <b> 306, the distance information (FIG. 8B) is generated from the distance information acquired by the distance information generation unit 202 at the same time as the image data and stored in the memory 109. The distance information is data represented by 8 bits from 0 to 255 for each pixel. In this example, the distance information is not linearly converted from the actual distance between the shooting position and the subject, but is nonlinear (for example, as the distance is shorter). This is 8-bit data converted (with high accuracy). In step S306, a distance image is generated in accordance with the image size (that is, the size of the frame of interest) using the distance information as a pixel value. In the case of FIG. 8B, the distance is expressed as white as the distance is short, and black as the distance is long.
In step S307, the region division processing unit 203 performs region division processing, which will be described later with reference to FIG. To divide.

  When the face detection process in step S305 and the area division process in step S307 are completed, an enlarged area determination process in step S308 (refer to FIG. 6 for details) based on the face detection result, the area division result, and the distance image. Are executed by the area determination processing unit 207. The enlargement area is a rectangular area to be enlarged and displayed. The determined rectangular area is enlarged and displayed on the display unit 112 by the enlargement / reduction processing unit 204 in step S309. In this way, the captured image enlargement process is executed. The captured image enlargement process is performed for each captured frame as long as an enlarged display instruction from the user continues. In addition, when the captured image enlargement process of FIG. 3 is performed on the moving image and the processing time from step S302 to S309 is simply added, it will exceed one frame time, and the enlargement is performed in real time. It is difficult to display an image. Therefore, in such a case, it is necessary to divide a series of processes into a plurality of processes that can be processed within one frame time (for example, two divisions of S302 to S307 and S308 to S309), and perform pipeline processing for each. In such a case, since the delay time from the process of S302 to the display process of S309 increases, the higher the processing capability of the image processing engine 111, the smaller the display delay. For example, in the case of a still image, the restriction of one frame time does not need to be considered.

<Details of area division processing>
Here, the region division processing in step S307 will be described with reference to FIGS.
In the area dividing process, first, an initial target pixel is set. In the area dividing process in step S307, the pixel corresponding to the in-focus position acquired in step S303 is set as the initial target pixel. FIG. 5A is an enlarged view of the distance image near the pixel corresponding to the in-focus position. A pixel written as 1 is a pixel corresponding to the in-focus position = a target pixel. After setting the target pixel, the pixel value of the target pixel = distance information value is acquired in step S403. In step S404, distance information values of 8 pixels adjacent to the target pixel are acquired. In step S405, the distance information value of the target pixel is compared with the distance information value of each adjacent pixel. If the difference between the information values is within a predetermined threshold value α, the adjacent pixel is set as a new target pixel. Set. Of course, a plurality of target pixels may be set. In FIG. 5B, a pixel described as 2 indicates a newly set target pixel. A pixel written as “x” indicates a pixel that has not been set as a new pixel of interest because it exceeds the threshold value α. The threshold α may be changed according to the shooting mode of the imaging device. For example, it is conceivable that the threshold value α is decreased in a mode where the depth of field is shallow, such as the macro shooting mode, and the threshold value α is increased in a mode where the depth of field is deep.
In step S406, the presence / absence of a newly set target pixel is determined. If there is a pixel of interest, the process returns to step S403 and the process is repeated. When a plurality of target pixels are set, the processing from step S403 is executed for each pixel. In FIG. 5C, the pixel described as 3 is the target pixel newly set for the target pixel 2. Eventually, the pixel of interest disappears, and the process proceeds to step S407, where the region division process ends. In FIG. 5D, since there is no new target pixel for the target pixel 5 (there is no pixel to be described as 6), the region division process ends in this state.
A set of pixels (pixels assigned numbers 1 to 5 in FIG. 5) set as the target pixel during processing, including the initial target pixel, is divided as one region. In FIG. 5 (e), a region with a diagonal line is a region including the initial pixel of interest = the focused pixel (reference numeral 501) divided by this region dividing process.
As described above, a group of pixels connected to the initial target pixel having a distance approximately equal to the distance of the initial target pixel is specified, and this is specified as a connected region related to the initial target pixel. If information for specifying the connected area is obtained, the image data itself may not be divided.

<Enlarged area determination processing>
Next, details of the enlargement area determination processing in step S308 described above will be described with reference to the flowchart of FIG.
When the enlargement area determination process is started in step S601, in step S602, the focus area is determined to be a face area from the focus area acquired in step S307 and the face detection information acquired in step S305 in the flowchart of FIG. Determine if. If it is determined as a face area, the process proceeds to step S605. If it is not determined as a face area, the process proceeds to step S603. In step S602, for example, if the focus area is included in the face area, it may be determined that the focus area is a face. Further, for example, if the face area and the in-focus area overlap, and the overlapped portion exceeds a certain ratio with respect to the in-focus area, the in-focus area may be determined to be a face.
In step S603, the in-focus area is determined as an enlarged area, and the process proceeds to the sub-area determination process in step S604. In step S603, for example, a rectangular area including the in-focus area is determined as an enlarged area. For this purpose, for example, a pixel group connected to the in-focus area, a pixel group having a color difference with a color in the in-focus area within a predetermined value is identified as an object including the in-focus position, and a rectangle including the object Let the region be an enlarged region. Further, for example, a pixel group that is connected to the in-focus area and that has a difference from a pixel distance in the in-focus area within a predetermined value is identified as an object including the in-focus position, and includes the object. A rectangular area is set as an enlarged area. In the latter case, for example, the region of the subject can be specified by appropriately setting the value of the threshold value α and executing the procedure of FIG. 4, and the enlarged region can be determined so as to include it. In this case, it is desirable to use a larger value for the threshold value α than in the case of specifying the focus position area, that is, the focus area.
The sub-region determination processing in step S604 determines whether or not there is an area having distance information equivalent to the in-focus area other than the in-focus area in the still image. This is a process of determining. Specific processing contents will be described below with reference to the flowchart of FIG.

When the sub-region determination process is started in step S701, the pixel A having the same distance information value as the in-focus position is searched for the distance image in step S702.
If the pixel A exists, the process proceeds to step S703, where the pixel A is set as the target pixel, and the region division process in step S704 is performed. The area dividing process is the process shown in the flowchart of FIG.
In step S705, it is determined whether or not the number of pixels constituting the area is greater than or equal to a predetermined number for the area extracted by the area dividing process. If the number is greater than or equal to the predetermined number, the process proceeds to step S706, and the extracted area is set as one of the sub areas, and the area information is held. If the predetermined number is not reached, the process returns to step S702 and the process is repeated. In step S702, the process is repeated until there is no pixel A, and the sub-region determination process ends. The sub-area is an area that is enlarged and displayed according to the user's designation. In this example, the sub-area is displayed while switching to the display of the area including the in-focus position. If there are a plurality of sub-regions, in step S309, the displayed regions are sequentially switched in accordance with the sub-region display designation operation. In order to make a sub-region a rectangular region, before registration in step S706, for example, a pixel group of a subject is specified based on a color difference or a distance difference in the manner described in step S603, and the rectangular region including the subject is defined as a sub-region. do it.

Returning to the flowchart of FIG. 6, the continuation of the enlarged region determination process will be described.
If it is determined in step S602 that the focus area is a face area and the process proceeds to step S605, it is determined whether another face exists in the vicinity of the focus area. For example, if the distance between each other is within a certain value, it can be determined to be near. Note that the face area that is the focus area is not included in the vicinity. If there is no other face area in the vicinity of the focused area, in step S606, the face area portion of the focused area (or a rectangular area including the face area) is determined as an enlarged area. On the other hand, if it is determined that there is another face area in the vicinity, the process proceeds to step S607, and distance information values are acquired from the distance image for all the face areas in the vicinity. It is determined whether or not the face area has a close distance information value. If the distance information of the face area to be compared is not close to the distance information of the focused area, the process proceeds to step S606. When the distance information is close, a rectangular area including all face areas with close distance information is specified, the rectangular area is determined as an enlarged area, and the enlarged area determining process is terminated. The aspect ratio of the enlarged region is preferably the same as the aspect ratio of the original frame. For comparison of distance information, for example, an average value of distances indicated by distance information of each pixel is obtained for each region, and if the difference between the average distances of the respective regions is within a predetermined value, the distance from the shooting position is short. It can be determined that the area. Alternatively, based on the distance information of all the pixels in the comparison target area, the pixels having the maximum distance and the minimum distance are specified, and the pixels belong to the same area or the difference is within a predetermined value. For example, it is possible to determine that the area is a subject area at a similar distance from the shooting point.

<Example of enlarged display>
Here, specific display examples of the image enlargement display using the captured image enlargement process described above are shown in FIGS.
Each of (a) and (b) in FIGS. 8 to 11 represents a still image held on the memory 109 and a distance image corresponding to the still image. Until there is an enlarged display instruction from the user, the image of (a) in each figure is displayed on the display unit 112 every time image data is captured from the sensor.

  In the case of FIG. 8, the in-focus area including the in-focus position 801 is an area where the vehicle is photographed as a subject. If there is an enlargement display instruction from the user in this state, the enlargement area determination process of FIG. 6 is executed. Since the in-focus area is not a face area, steps S602, S603, and S604 are processed, and the sub-area determination process of step S604 is performed. Executed. It is determined that there is no sub-region, and finally, a rectangular region 802 including a focused region is determined as an enlarged region. By the above processing, the rectangular area 802 is enlarged and displayed on the display unit 112, and a display state as shown in FIG.

  In the case of FIG. 9, the in-focus area including the in-focus position 902 is an area where the vehicle is photographed as a subject. If there is an enlargement display instruction from the user in this state, the enlargement area determination process of FIG. 6 is executed. Executed. In the sub-region determination process, the region 903 is determined as a region including a pixel group having a distance information value close to the in-focus region, and as a result, the rectangular region 904 including the region 903 is determined as a sub-region. Eventually, the rectangular area 902 including the in-focus area and the rectangular area 904 are determined as the enlarged areas. Through the above processing, the rectangular area 902 is enlarged on the display unit 112, and a display state as shown in FIG. At this time, a mark 905 is displayed on the display unit 112 to indicate that there is a sub area. When switching of the sub area is instructed by a user operation, the rectangular area 904 is enlarged and a display as shown in FIG. Similarly to FIG. 9C, a mark 906 indicating that a sub area exists is displayed on the screen, and when switching of the sub area is instructed by a user operation, the display returns to the display of FIG. Every time there is an instruction thereafter, the display is switched as shown in FIGS.

  In the case of FIG. 10, the focus area including the focus position 1001 is included in the face area. If there is an enlargement display instruction from the user in this state, the enlargement area determination process of FIG. 6 is executed, and since the in-focus area is included in the face area 1002, the process proceeds from step S602 to step S605. Since there is a nearby face area 1004 in the focused area, the process proceeds to step S607. At this time, since the distance information value of the nearby face area 1004 is not close to the distance information value of the face area 1002, the process proceeds to step S606. Eventually, the rectangular area 1003 is determined as the enlarged area. By the above processing, the rectangular area 1003 is enlarged and displayed on the display unit 112, and a display state as shown in FIG.

  In the case of FIG. 11, the focus area including the focus position 1101 is included in the face area. If there is an enlargement display instruction from the user in this state, the enlargement area determination process of FIG. 6 is executed, and since the focus area is included in the face area 1102, the process proceeds from step S602 to step S605. Since there are neighboring face areas 1103 and 1104 in the focus area, the process proceeds to step S607. At this time, since the distance information value of the nearby face area 1104 is not close to the distance information value of the face area 1102, while the face area 1003 has a close distance information value, the process proceeds to step S608. Eventually, the rectangular area 1105 is determined as the enlarged area. By the above processing, the rectangular area 1105 is enlarged and displayed on the display unit 112, and a display state as shown in FIG. In this way, a rectangular region that includes the subject at the in-focus position and a subject that is approximately equidistant from the subject or a rectangular region that includes both of them can be determined as an enlargement display target, and can be enlarged and displayed. In step S607, it is determined that the distance along the screen between the in-focus area and the face area is within a predetermined value. In step S608, the distance in the depth direction between the in-focus area and the face area is within the predetermined value. It can also be said that there is a certain thing. That is, in these steps, it can be determined that the subject at the in-focus position and the subject in the face area are within a predetermined range in the three-dimensional space.

  As described above, the image processing apparatus according to the present embodiment performs a combination of the region division processing result on the distance image and the face information to determine an adaptive enlarged region for the image to be displayed. It is possible to improve the convenience of enlarged display. In particular, by adaptively enlarging and displaying a region including the in-focus position, that is, the focus position in the image, both confirmation of the in-focus position and confirmation of the subject can be achieved. In addition, a plurality of in-focus positions in the image can be confirmed by magnifying and displaying a subject equidistant from the in-focus position.

  In steps S602 and S605, the face area is determined. However, if the subject is of the same type, the process may be branched to S607.

[Example 2]
In the first embodiment, the enlarged display method at the time of image shooting has been described. In this embodiment, an enlarged display method during image reproduction will be described. The image processing apparatus will be described with the configuration of the digital camera shown in FIG. 1 as in the first embodiment, but the photographing function is not necessarily required in the present embodiment.

  In the image handled in this embodiment, a still image is recorded as a JPEG file, and corresponding distance image data is associated with the JPEG file and recorded on a recording medium (memory card 107). The still image is also provided with in-focus position information and face position information as metadata, and is recorded in the JPEG file. For example, these metadata are specified by a camera at the time of shooting and attached to a still image as metadata.

Image enlargement processing during reproduction will be described with reference to the flowchart of FIG.
When the reproduction image enlargement process is started in accordance with a user instruction in step S1201, a JPEG file is read from the memory card 107 and stored in the memory 109 in step S1202. In step S1203, the focus position and face position information are acquired from the metadata in the JGEG file and stored in the memory 109.
In step S1204, the JPEG encoding / decoding unit 105 decodes the JPEG file on the memory 109, and the reproduced still image is expanded on the memory 109. This reproduced still image is a still image corresponding to (a) in FIGS. 8 to 11 of the first embodiment.
In step S1205, the distance image file is read from the recording medium, and the distance image is developed in the memory 109. This distance image is a distance image corresponding to (b) in FIGS.
In step S1206, region division processing is executed on the in-focus position acquired in step S1203 in the distance image on the memory 109. The contents of the area division processing are the same as those in the flowchart of FIG. Subsequently, an enlargement area determination process in step 1207 is executed. The enlarged region determination process may be the same as the flowchart in FIG. 6 of the first embodiment. When the enlargement area is determined, the area is enlarged and displayed on the display unit 112 in step S1208.

  Here, at the time of reproduction, in step S1208, for example, a plurality of selected images may be displayed side by side on the display unit 112. In this case, each image is displayed in a reduced size. In such a display, a reproduction image enlargement process may be applied to each individual image so that the content of the image can be easily confirmed even when a plurality of images are displayed. Although it is necessary to perform processing on a plurality of images, in the present invention, since the area division processing on a simple distance information image is used as an image, the processing on each image is light and the processing on a plurality of images is performed. Execution time can be shortened even in processing.

  As described in the first embodiment, a rectangular area including a subject at the in-focus position and a subject that is substantially equidistant from the subject is used as an enlargement display target for image data recorded in advance by the above procedure. It can be determined and enlarged.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101: Control unit, 102: Lens, 103: Image sensor, 104: Signal processing unit, 105: JPEG encoding / decoding unit, 106: Memory card controller, 107: Memory card, 108: Flash memory, 109: Memory, 111 : Image processing engine, 112: Display unit

Claims (10)

Means for specifying a focus position in image data including distance information between a shooting position and a subject;
Based on the image data , specifying means for specifying an area including the subject at the in-focus position as an enlarged area;
Have a display means for displaying an enlarged the expansion area,
The specifying unit further specifies, as the enlarged region, an area including a subject having a distance such that a difference between the focus position and the subject distance is within a predetermined value based on the image data. Image processing device. Means for identifying a face region from the image data;
The specifying unit further specifies, as the enlarged region, an area including a subject whose distance is such that a difference from the distance of the subject at the in-focus position is within a predetermined value when the in-focus subject is not a face. The image processing apparatus according to claim 1 , wherein: The specifying unit determines whether a subject of the same type as the subject at the in-focus position is within a predetermined range of the subject at the in-focus position in a three-dimensional space based on the image data. 3. The image processing apparatus according to claim 1, wherein an area further including the same type of subject as the subject at the in-focus position is specified as the enlarged region. When a plurality of enlargement areas are specified, the display means displays a mark for switching display on a screen for displaying the enlargement area, and the plurality of enlargement areas are displayed according to an instruction for the mark. the image processing apparatus according to any one of claims 1 to 3, characterized in that switching the display. Means for identifying a face region from the image data;
When the subject at the in-focus position is a face, the specifying unit determines whether another face is within a predetermined range of the face at the in-focus position in a three-dimensional space based on the image data. If it is determined to be within range, the image processing apparatus according to claim 1, wherein the identifying further comprises region the other face as the enlarged region. The specifying means includes a subject at the focus position and a subject of the same type as the subject at the focus position within a predetermined range on the image, and the subject at the focus position and the focus. Determining that the subject at the in-focus position is within the predetermined range in a three-dimensional space when a difference in distance from the shooting position between the subject at the position and the subject of the same type is within a predetermined value. The image processing apparatus according to claim 3 . By photographing a subject, according to any one of claims 1 to 6, characterized by further comprising means for acquiring image data corresponding to the distance image showing the distance between the shooting position and the object for each pixel Image processing apparatus. By reading from the recording medium, according to any one of claims 1 to 6, characterized by further comprising means for acquiring image data corresponding to the distance image showing the distance between the shooting position and the object for each pixel Image processing apparatus. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 8 . Identifying a focus position in image data including distance information between a shooting position and a subject;
A specifying step of specifying an area including the subject at the in-focus position as an enlarged area based on the image data ;
Have a display step of displaying by enlarging the enlargement area,
In the specifying step, based on the image data, an area including a subject having a distance such that a difference from the distance of the subject at the in-focus position is within a predetermined value is further specified as the enlarged region. Image processing method.

Images