JP5653509B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for performing a blurring process on an image obtained by an imaging apparatus using a solid-state imaging device.

  Conventionally, diorama-like shooting in which the top and bottom of an image is blurred has been optically realized. In order to realize optically, for example, overhead view photographing is performed using a tilt shift lens which is a special lens. Further, the blurring effect of the image is not limited to the diorama style, and can be realized by image processing on a normal captured image. Japanese Patent Application Laid-Open No. 2004-151561 discloses a technique that can obtain an effect like soft focus by mixing an image obtained by blurring a normal captured image and an original image at a certain ratio. Moreover, in patent document 2, when displaying the picked-up image of the vehicle periphery by a vehicle-mounted camera on a monitor screen, the technique which makes the sharpness of an image display different in the other area except the said partial area and the some area of a picked-up image. Is disclosed.

JP 2000-69277 A Japanese Patent No. 3944926

  In order to realize a diorama-like still image by image processing, a plurality of stages of blurring processing in which the degree of blur is changed in a predetermined direction is necessary. If the relationship between the blur processing region and direction and the composition of the photographed image is appropriate, a good image in which the effect of the diorama effect can be visually recognized can be obtained. However, if the posture of the camera is changed while the blur processing area and direction are fixed, the direction and composition of the blur effect of the photographed image also change, which may result in an image that does not exhibit the expected diorama effect.

  The present invention is intended to solve the above-described problems, and enables an image having an expected diorama effect to be obtained even when the attitude of the imaging apparatus is changed.

In order to solve the above-described problems, an image processing apparatus according to the present invention is an image processing apparatus that performs image processing on a captured image input from an imaging unit, and that performs special effects on the captured image. Processing means;
Setting means for setting a direction of a special effect by the image processing means, and the setting means sets a region where the special effect is not applied, and changes the setting of the direction of the special effect before the change. According to the setting of the area where the special effect is not applied, the changed area where the special effect is not applied is set.

  The image processing apparatus of the present invention is an image processing apparatus that performs image processing on a captured image input from an imaging unit, and includes a region that is not blurred and a region that is blurred with directionality from the captured image. An image processing unit that generates an image; and a setting unit that sets a blur direction of the image generated by the image processing unit. The setting unit sets a region that is not blurred and sets the blur direction. When changing the area, the non-blurred area after the change is set according to the setting of the non-blurred area before the change.

  The image processing apparatus of the present invention is an image processing apparatus that performs image processing on a captured image input from an imaging unit, and includes a first region and the first region in a predetermined direction from the captured image. Image processing means for generating an image having a second area that is more blurred than the area; and setting means for setting the predetermined direction and the first area. When the direction setting is changed, the changed first area is set according to the setting of the first area before the change.

  The image processing method of the present invention is an image processing method for performing image processing on a captured image obtained by an image processing apparatus, and is set by a setting step for setting a direction of a special effect and the setting step. An image processing step for applying a special effect to the photographed image according to the direction of the special effect, and in the setting step, an area where the special effect is not applied is set, and the setting of the direction of the special effect is changed. In this case, a region where the special effect after the change is not applied is set according to the setting of the region where the special effect is not applied before the change.

  The image processing method of the present invention is an image processing method for performing image processing on a captured image obtained by an imaging apparatus, and has a region that is not blurred and a region that is blurred with directionality from the captured image. An image processing step for generating an image, and a setting step for setting a blur direction of the image generated in the image processing step. In the setting step, an area not to be blurred is set, and the blur direction is set. When the setting is changed, the non-blurred area after the change is set according to the setting of the non-blurred area before the change.

  The image processing method of the present invention is an image processing method for performing image processing on a captured image obtained by an imaging apparatus, wherein the first region and the first region are defined in a predetermined direction from the captured image. An image processing step for generating an image having a second region that is more blurred than the region; and a setting step for setting the predetermined direction and the first region. In the setting step, the predetermined step When the direction setting is changed, the changed first area is set according to the setting of the first area before the change.

  According to the present invention, it is possible to obtain an image in which an expected diorama-like effect appears even when the posture of the imaging apparatus is changed.

The figure which shows the structure of the imaging device Operation flow for explaining diorama-like image processing according to an embodiment of the present invention Operation flow for explaining the photographing operation of the embodiment of the present invention Schematic diagram showing the relationship between the attitude of the imaging device and diorama processing Schematic diagram showing the relationship between the attitude of the imaging device and diorama processing

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram illustrating a configuration of an imaging apparatus according to the first embodiment.

  The optical system 101 images a subject by forming incident light on the image sensor 102. The optical control unit 111 controls exposure, zoom, focus, optical blur correction, and the like of the optical system 101. The image sensor 102 converts the formed optical image into an electrical signal. The imaging control unit 112 is a control system for driving the imaging element. The analog signal processing unit 103 performs processing such as clamping and gaining on the output of the solid-state imaging device. An analog / digital (A / D) conversion unit 104 performs digital conversion on an analog signal and outputs it as image data. In the present embodiment, the imaging unit including the optical system 101 to the A / D conversion unit 104 is used. In addition, the imaging apparatus enables still image shooting by recording image data of a captured image output from the imaging unit on the recording medium 108 in a single frame. In addition, video recording is possible by continuously recording a plurality of frames. The signal processing unit 105 performs image processing described later on the captured image of the A / D converted digital signal to generate an output image. The internal storage unit 106 temporarily stores image data when generating an output image. The signal processing unit 105 is configured to be able to read out image data stored in the internal storage unit 106 according to an instruction from the system control unit 113, and performs diorama-like image processing such as generation of a blurred image, trimming processing, and image synthesis processing. I do. Details of the diorama-style image processing will be described later.

  The recording / reproducing unit 107 records the generated image data on the recording medium 108. As the recording medium 108, various removable recording media such as an optical disk recording medium and a semiconductor recording medium can be used. The image display device 110 displays the generated image data as a live image via the display control unit 109. The display device 110 having a function as an electronic viewfinder also performs in-focus frame display, in-focus display, and out-of-focus warning display according to the execution of the program in the system control unit 113. The system control unit 113 basically controls the overall operation of the imaging apparatus, and usually includes a CPU. For example, a predetermined calculation process is executed by the signal processing unit 105 using image data obtained by the imaging unit, and the optical control unit 111 and the imaging control unit 112 are controlled based on the calculation result. The optical control unit 111 performs control to execute automatic focus control (AF). The optical control unit 111 controls the diaphragm and shutter, and the imaging control unit 112 controls the electronic shutter of the image sensor 102 to execute automatic exposure control (AE).

  The posture detection unit 115 detects the posture of the imaging apparatus using an acceleration sensor and / or a gyro sensor used for the optical blur correction. The system control unit 113 determines the posture of the imaging apparatus based on the posture detection result output from the posture detection unit 115, displays setting information according to the posture, and changes the input direction of the cursor key according to the posture. . Reference numeral 114 denotes an operation unit. The operation unit 114 includes a shutter SW1 for instructing the imaging apparatus to perform a preliminary operation before photographing, a shutter SW2 for instructing the imaging apparatus to perform a photographing operation, a cursor key for instructing a position for performing image processing, and the like. The system control unit 113 performs AF and AE by detecting the shutter SW1, and performs a photographing operation by detecting the shutter SW2. The configuration of the imaging apparatus illustrated in FIG. 1 is an example, and the configuration of the imaging apparatus according to the present invention is limited to the configuration illustrated in FIG. 1 as long as the operations described below can be performed. It is not something.

  The diorama style image processing of this embodiment will be described with reference to FIG. In the present embodiment, an image in which the effect of the diorama effect can be visually recognized can be generated by performing the diorama image processing in any of the shooting operations of still image shooting, moving image shooting, and live display. In the present embodiment, common diorama-like image processing is performed in the shooting operation. However, since the image quality and frame rate required for each shooting operation are different, the contents of the diorama-like image processing may be changed according to the shooting operation.

  In FIG. 2, an image 200 shows the entire captured image input to the signal processing unit 105. An object of the present embodiment is to obtain an image in which a diorama effect appears mainly on a person in a scene including a building, a person, a tree, and the like. A frame 201 displays a range and position for focusing specified in the angle of view of the captured image. A person is included in the frame 201, and an image showing a diorama effect is obtained by performing a process of blurring the upper and lower regions. The strength of blurring the area can be arbitrarily set by the signal processing unit 105, and can be gently blurred.

  In the present embodiment, the frame 201 is also displayed on the live image displayed on the display device 110, the vertical position thereof is fixed at a position slightly below the center line 202, and the width 203 is also constant. It has become. The signal processing unit 105 continuously performs diorama-like image processing on each frame according to the position and width of the frame 201 so as to update the live image.

  Next, an operation flow of diorama-like image processing will be described with reference to FIG. This operation flow is realized by the system control unit 113 controlling each processing block.

  First, the system control unit 113 responds to the fact that one frame of a photographic image to be processed is obtained by performing development processing such as white balance, which is image data input by the imaging unit. Is started (S201). Here, at the start of the operation flow, fixed values for the position and width of the frame 201 are already set.

  The system control unit 113 determines the amount of blur in an area other than the frame 201 based on the degree of blur set by the user (S202). In the present embodiment, the degree of blur can be selected from “strong” and “weak” even during the photographing operation.

  Next, the system control unit 113 stores the image data of the captured image obtained by performing development processing such as white balance, which is image data input by the imaging unit, in the internal storage unit 106 (S203). This is the same as the photographed image when photographing in the normal mode.

  Then, the signal processing unit 105 creates an image in which the entire image is blurred from the image data of the captured image (S204). In the present embodiment, a blurred image is generated by reducing a captured image by a blinking process and then enlarging again to the original size. For example, if the selected degree of blur is “strong”, the reduction rate of the reduction process is 1/8, and if it is “weak”, the reduction rate of the reduction process is 1/4. The present invention is not limited to this method, and a blurred image may be generated using an image that has been optically captured in advance at an out-of-focus position.

  The signal processing unit 105 trims the portion of the frame 201 from the image data of the captured image stored in the internal storage unit 106 (S205). Then, the signal processing unit 105 aligns and pastes the trimmed image to the blurred image created in step S204, thereby generating an image in which the upper and lower sides are blurred and the sense of resolution remains in the center (S206). . In addition, when pasting image data of a photographed image, the pasting boundary portion is made inconspicuous by gently mixing the blurred image and the trimmed photographed image. That is, the process of setting the combined ratio of the captured image and the blurred image to 0: 1 at the boundary portion of the frame 201 and increasing the combined ratio of the captured image toward the center line of the frame 201 is performed.

  FIG. 3A is an operation flow for changing the contents of the diorama-like image processing in accordance with the posture change of the imaging device. This operation flow is realized by the system control unit 113 controlling each processing block. The content change operation flow can also be applied to moving image shooting and still image shooting. Here, the operation from the state of performing a live image display operation to the still image shooting will be described.

  In the state where the live image is displayed, this operation flow is started in response to the setting of the diorama-like shooting mode by the user's mode designation. After the start of the operation flow, the system control unit 113 determines the attitude of the imaging device based on the output signal of the attitude detection unit 115 (S301). Next, the system control unit 113 compares the current posture detected in step S301 with the stored previous posture. Here, the current posture is stored for use as the previous posture in the next posture comparison (S302). Next, the system control unit 113 determines whether the posture has been changed according to the comparison result (S303). If it is determined in step S303 that the posture has not been changed, the process proceeds to step S305. On the other hand, if it is determined that the posture has been changed, the process proceeds to step S304.

  In step S304, as shown in FIG. 3A, FIG. 3B, and FIG. 3C, the display direction of the frame 201 and diorama-like image processing are performed according to the orientation of the imaging device determined in step S301. Switch the direction.

  If the posture has not been changed (step S303: No), the direction is as it is, and if the posture has been changed (step S303: Yes), the direction is changed in step S304 in FIG. 2B. The described diorama-style image processing is performed (S305).

  Then, it is determined whether or not the shutter button has been pressed (S306). If it is determined that the shutter button has not been pressed, the process returns to step S301, and the display of the live image and the posture detection based thereon are continued, and it is determined that the shutter button has been pressed. Performs still image shooting processing (S307). Then, the diorama-like image processing described with reference to FIG. 2B is performed on the obtained still image (S305). Here, the image quality should be prioritized over the frame rate for the still image, so that the process of steps S204 to S206, which is performed using one blurred image, is performed using two or more blurred images, so that a diorama can be obtained. A still image with a wind effect may be obtained.

  The processing direction change processing in step S304 in FIG. 3A will be described in detail. FIG. 3B is a diorama-like image displayed on the display device 110 when the user holds the imaging device in the normal position. This is the same as that described with reference to FIG. 2A, and the range in focus and the position of the display frame are the frame 201. Here, the normal position is a position where the user holds the imaging device as shown in FIG.

  FIG. 3C shows a position where the user holds the grip position of the imaging apparatus downward as shown in FIG. 4B and rotates 90 ° to the right with respect to the normal position. FIG. 4B shows the processing direction at this time and that the frame 201 is rotated 90 ° to the left with respect to the normal position.

  FIG. 3D is a position rotated 90 ° to the left with respect to the normal position. The position of the frame 201 at this time has a symmetrical relationship with FIG.

  If the frame is fixed even if the imaging device is rotated 90 degrees in the left and right directions, the composition changes with respect to the scene 200 in FIG. 2A and the frame is displayed in the vertical direction, and the diorama processing is performed. Therefore, the expected diorama effect cannot be obtained. On the other hand, as in this embodiment, by rotating the direction of the frame as shown in FIGS. 3C and 3D according to the posture change, the effect of the diorama effect can be obtained even if the composition changes. Since the direction can be maintained in the state before the rotation, a good image can be obtained.

(Second Embodiment)
In the present embodiment, during the live image display operation, the user can set the vertical position and width of the frame 201 using the operation unit 114. The signal processing unit 105 executes diorama-like image processing according to the set position and width of the frame 201 so as to update the live image.

  At this time, when the direction of the diorama-like image processing is changed due to the posture change, it is preferable that the diorama-like image is continuously displayed with the position and width of the frame 201 set by the user.

  A method for setting the frame 201 (the range and position where the image is in focus) when rotated 90 ° to the right from FIG. 4 (a) from the normal position in FIG. 4 (a) will be described with reference to FIG. Although description about the case of 90 ° rotation to the left is omitted, since it can be realized with the same configuration, in the following description, it is assumed that the rotation is 90 ° to the right or 90 ° to the left. Express.

  When the scene shown in FIG. 4C similar to that of the first embodiment is photographed to obtain a diorama effect, the user photographs with a composition of either the normal position or the vertical position. Since the in-focus range and position (frame 201) are set according to the person who is the main subject, regardless of the normal position (402) and vertical position (401), the user's specification is reflected and the width is almost the same. It is necessary to set it. Further, the position of the frame is also set so that the vertical balance is within a certain region regardless of the normal position (402) and the vertical position (401) so that the balance of the composition is not lost.

  FIG. 5 is a diagram illustrating an example in which the above setting is performed according to a change in posture.

  FIG. 5A shows an example in which the width of the frame 201 at the normal position is set to M, and shows the case of the composition 402 in FIG. 4C. Normal position When the posture is changed from the vertical position shown in FIG. 5A to the vertical position shown in FIG. 5B, the width of FIG. 5B in the vertical position according to the width M of FIG. 5A in the normal position. Set P. Here, there is a relationship of width M: vertical width X of the captured image = width P: horizontal width Y of the captured image. Using the same ratio relationship, the width O and the width Q in FIG. 5B in the vertical position are set according to the width L and the width N in FIG. Vertical Position Even when the posture is changed from FIG. 5B to the normal position FIG. 5A, the width L, the width M, and the width N are set according to the width O, the width P, and the width Q.

  FIG. 5C is an example in which the position of the frame 201 at the normal position is set to the center position S, and shows the case of the composition 402 in FIG. 4C. The center position S of the frame 201 in FIG. 5C at the normal position is set as a ratio to the height X of the entire screen. When the posture is changed from FIG. 5C to FIG. 5D in the vertical position, the center position U of the frame 201 can be easily obtained from the width Y of the entire screen by the formula: U = Y × S ÷ X. Is possible. Further, when the posture is changed from FIG. 5D in the vertical position to FIG. 5C in the normal position, it is calculated from the equation S = X × U ÷ Y. Here, an example in which the center position U, the center position S, and the like are inherited at a ratio has been shown, but the position may be calculated from the upper end of the screen or the number of pixels from the center of the screen.

  The setting of the frame 201 has been described separately for the width and the position. However, if the width M of FIG. 5A is set from the center position S of the frame 201 of FIG. 5C, FIG. ) Can be set from the center position U of FIG. In this case, as shown in FIG. 4D, when the user sets the frame 201 near the lower end of the screen at the vertical position 404 and resets it to the normal position 405, the center position U of the frame is increased when the set width P is wide. And S may correspond to each other, but the frame range P may protrude from the composition. Therefore, when the frame is set at the position that touches the lower end, the center position relationship is not inherited as it is even if the posture is changed, and the center position is set so that the lower end position is maintained while maintaining the width of the frame. To do. Although not shown, the same processing is performed when the frame 201 is set near the upper end.

  As described above, in the present embodiment, in the specification in which the user can set the position and width of the frame for performing the diorama processing, even when the orientation of the imaging apparatus changes, the expected diorama processing An image showing the effect can be obtained.

(Other embodiments)
In the above-described embodiment, the display of live images and the diorama-like image processing for still image recording have been described. However, the present invention is not limited to this, and live display in the case of performing moving image recording or for moving images to be recorded Is also applicable.

  In the above embodiment, the term “diorama-like image processing” is used as a special effect. However, this is for convenience, and the captured image and one or more blurred images are given directions and each region is changed. The present invention can be applied to other special effect processing as long as it includes processing to synthesize.

  In the above embodiment, the case where the present invention is applied to an imaging apparatus has been described. However, the present invention is not limited to this, and a personal computer, a digital television, or the like that can input a captured image from an externally connected imaging apparatus. It may be realized in the image processing apparatus. In this case, the posture is determined from the posture detection information included in the header information in the image file of the photographed image, and diorama-like image processing corresponding to the posture is performed. In the case of a personal computer, it is realized by obtaining and installing application software for executing the steps shown in FIG. 3 from a storage medium such as a CD-ROM or a server on a network.

105 signal processing unit 106 internal storage unit 113 system control unit 115 attitude detection unit

Claims (29)

An image processing apparatus that performs image processing on a captured image input from an imaging unit,
Image processing means for applying a special effect to the captured image;
Setting means for setting the direction of special effects by the image processing means,
The setting means sets the area where the special effect is not applied, and changes the special effect direction according to the setting of the area where the special effect is not applied when changing the setting of the direction of the special effect. An image processing apparatus characterized in that an area to be applied is set.   The image processing apparatus according to claim 1, wherein the setting unit sets a width of an area where the special effect is not applied.   The image processing apparatus according to claim 1, wherein the setting unit sets a position of an area to which the special effect is not applied.   When the setting means changes the setting of the direction of the special effect, the ratio of the width of the area where the special effect is not applied and the width of the direction of the special effect of the entire image is not changed before and after the change. The image processing apparatus according to claim 1, wherein a width of a region to which no special effect is applied is set.   When the setting means changes the setting of the direction of the special effect, the ratio between the distance from the image edge of the position of the area where the special effect is not applied and the width of the direction of the special effect of the entire image is before and after the change. 5. The image processing apparatus according to claim 1, wherein a position of an area where the special effect is not applied is set so as not to change. Having detection means for detecting the posture of the imaging means;
The image processing apparatus according to claim 1, wherein the setting unit sets a direction of the special effect according to an attitude detected by the detection unit.   The image according to any one of claims 1 to 6, further comprising display means for displaying on the display medium a frame corresponding to the image to which the special effect has been applied and a region to which the special effect is not applied. Processing equipment.   The image processing apparatus according to claim 1, further comprising an operation unit that allows a user to set a position or a width of an area to which the special effect is not applied. Instructing means for instructing imaging to the imaging means,
The setting means sets an area where the special effect is not applied by the operation means when the display means is sequentially displaying images input from the imaging means on the display medium, and the image processing means is determined by the instruction means. The image processing apparatus according to claim 8, wherein the special effect is applied to an image captured according to an imaging instruction according to the setting.   The image processing apparatus according to claim 1, wherein the special effect is diorama style image processing.   The image processing apparatus according to claim 1, wherein the special effect is a synthesis process having directionality.   10. The image processing according to claim 1, wherein the special effect is image processing in which an image having a first area and a second area that is more blurred than the first area in a predetermined direction is generated. The image processing apparatus according to any one of the above. An image processing apparatus that performs image processing on a captured image input from an imaging unit,
Image processing means for generating an image having a non-blurred area and a blurred area with directionality from the captured image;
Setting means for setting a direction of blur of an image generated by the image processing means,
The setting means sets a non-blurred area and sets the non-blurred area after the change according to the setting of the non-blurred area before the change when setting the blur direction and changing the setting of the blur direction. Image processing device. Having detection means for detecting the posture of the imaging means;
The image processing apparatus according to claim 13, wherein the setting unit sets a blur direction of an image generated by the image processing unit in accordance with an attitude detected by the detection unit.   The image processing apparatus according to claim 13, wherein the setting unit sets a width of the non-blurred area.   The image processing apparatus according to claim 13, wherein the setting unit sets a position of the area not to be blurred.   When the setting unit changes the setting of the blur direction, the ratio of the width of the non-blurred area and the width of the whole image in the blur direction does not change before and after the change. The image processing device according to claim 13, wherein the image processing device is set.   When the setting unit changes the setting of the blur direction, the ratio of the distance from the image edge of the position of the non-blurred region to the width of the blur direction of the entire image does not change before and after the change. The image processing apparatus according to claim 13, wherein a position of the area not to be blurred is set. An image processing apparatus that performs image processing on a captured image input from an imaging unit,
Image processing means for generating an image having a first area in a predetermined direction and a second area that is more blurred than the first area from the captured image;
Setting means for setting the predetermined direction and the first region;
The image processing apparatus, wherein when the setting in the predetermined direction is changed, the setting unit sets the changed first area according to the setting of the first area before the change. Having detection means for detecting the posture of the imaging means;
The image processing apparatus according to claim 19, wherein the setting unit sets the predetermined direction according to an attitude detected by the detection unit.   The image processing apparatus according to claim 19, wherein the setting unit sets a width of the first region.   The image processing apparatus according to claim 19, wherein the setting unit sets a position of the first region.   When the setting means changes the setting in the predetermined direction, the ratio of the width of the first area to the width of the predetermined direction of the entire image is not changed before and after the change. The image processing apparatus according to claim 19, wherein a width of the image processing apparatus is set.   When the setting unit changes the setting of the predetermined direction, the ratio between the distance from the image edge of the position of the first region and the width of the entire image in the predetermined direction does not change before and after the change. 24. The image processing apparatus according to claim 19, wherein the position of the first area is set in the image processing apparatus. An image processing method for performing image processing on a captured image obtained by an image processing apparatus, wherein a setting step for setting a direction of a special effect;
An image processing step of applying a special effect to the photographed image according to the direction of the special effect set in the setting step;
In the setting step, when the area where the special effect is not applied is set and the setting of the direction of the special effect is changed, the special effect after the change is changed according to the setting of the area where the special effect is not applied before the change. An image processing method characterized by setting a region to be not applied. An image processing method for performing image processing on a captured image obtained by an imaging device,
An image processing step for generating an image having a non-blurred area and a blurred area with directionality from the captured image;
A setting step for setting a blur direction of the image generated in the image processing step. In the setting step, when setting a region not to be blurred and changing the setting of the blur direction, An image processing method comprising: setting the non-blurred area after the change according to the setting of the non-blurred area. An image processing method for performing image processing on a captured image obtained by an imaging device,
An image processing step of generating an image having a first area in a predetermined direction and a second area that is blurred from the first area from the captured image;
Setting the predetermined direction and the first region, and
In the setting step, when the setting in the predetermined direction is changed, the changed first area is set according to the setting of the first area before the change. Each step of the image processing method according to any one of claims 25 to 27 is described, a program executable by a computer.   A computer-readable storage medium storing a program for causing a computer to execute the steps of the image processing method according to any one of claims 25 to 27.

Images