JP4513903B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  In recent years, imaging apparatuses equipped with a continuous shooting function for continuously taking a plurality of photographs have been widely used. Recently, a technique has been developed to output a higher quality photo than a pre-combination photo by combining a plurality of photos taken using the continuous shooting function into a single photo. Usually, such photo composition processing is automatically executed in the imaging apparatus, or is executed on the PC by moving the photo data to a personal computer (hereinafter, PC) or the like. For example, Patent Document 1 below describes a configuration in which a plurality of photographs are automatically combined in an imaging apparatus, and the combined photographs are displayed as one still image.

  The purpose of synthesizing photographs includes, for example, generating a panoramic photograph by synthesizing photographs in different imaging ranges. As another object, recently, a technique of generating a photograph with a resolution higher than the resolution of the image sensor by combining a plurality of photographs taken while moving with an amount of movement equal to or less than the pixel size of the image sensor. Are known. For example, a technique may be used in which a photographer selects a photograph that matches a photographer's intention from a plurality of continuously photographed photographs, without combining consecutively photographed photographs. This method is very effective in that it can avoid a photograph in which a person is meditating at the time of shooting or an unintended object is in the imaging range.

Japanese Unexamined Patent Publication No. 2000-41170

  However, when a large number of people are included in the subject as in a group photo, it is often difficult to shoot a picture in which all people are not meditated even if continuous shooting is performed. As described above, even if the continuous shooting is performed, there are cases where a plurality of continuously shot photos do not include a photo that matches the photographer's intention. In such a case, the photographer cannot obtain a photograph that matches the intention. Further, in order to move photo data to a PC or the like and perform photo correction or composition using an image processing application, a high degree of operation technique and knowledge are required, which is a heavy burden on the photographer. As a result, the photographer gives up on obtaining the intended photograph and endures by selecting a photograph relatively close to the intended photograph.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to obtain a plurality of consecutively photographed photographs with a simple operation in order to obtain a photograph intended by the photographer. It is an object of the present invention to provide a new and improved image processing apparatus and image processing method that can be partially combined.

  In order to solve the above-described problem, according to an aspect of the present invention, a display screen on which a first image included in a group of continuously captured images is displayed, and friction on the display screen rubbed by a user. A friction position detection unit that detects a position; an image composition unit that combines a pixel region of the first image corresponding to the friction position and a pixel region of a second image included in the image group; The image combining unit combines the first image with the first image so that the difference in shooting time between the first image and the second image increases as the friction amount rubbed by the user increases. An image processing apparatus is provided for switching between the two.

  The image composition unit increases the composition ratio of the second image to the first image as the friction amount increases, and the second image is displayed when the friction amount exceeds a predetermined threshold. It may be configured to switch to an image having a larger photographing time difference from the first image.

  Further, the image composition unit displays the second image before switching in the pixel area until the friction amount exceeds a predetermined threshold by a predetermined amount, and when the predetermined amount is exceeded, the second image is displayed. It may be configured to switch.

  The friction amount is an amount determined by one amount or a combination of plural selected from the number of times the pixel area is rubbed, the pressing force during friction, the friction direction, and the friction speed. Also good.

  Further, the image group may be a plurality of images that are continuously captured while changing arbitrary parameters that can be changed at the time of capturing.

  The parameter may be a control parameter for controlling a depth of field, an image effect, or a flash.

  In order to solve the above-described problem, according to another aspect of the present invention, when a first image included in a group of continuously captured images is displayed on a display screen, the user rubs. The friction position detecting step for detecting the friction position on the display screen, the pixel area of the first image corresponding to the friction position, and the pixel area of the second image included in the image group are combined. An image compositing step, wherein in the image compositing step, the photographing time difference between the first image and the second image increases as the friction amount rubbed by the user increases. An image processing method is provided in which a second image to be combined with one image is switched.

  As described above, according to the present invention, it is possible to partially synthesize a plurality of continuously photographed photographs with a simple operation. As a result, the photographer can obtain an intended photograph by partially combining a plurality of photographs.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, the duplicate description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[About the flow of explanation]
Here, the flow of explanation to be described later will be briefly described. First, with reference to FIG. 1 and FIG. 2, the object of the technology according to the embodiment of the present invention will be briefly described. Among these, specific problems to be solved by the technology according to the embodiment will also be described. Next, with reference to FIG. 3, the principle regarding the technique of the embodiment will be described. Thereafter, a functional configuration of the image processing apparatus 100 according to the embodiment capable of realizing the technology will be described with reference to FIGS. 4 to 9.

  Next, the operation of the image processing apparatus 100 will be described with reference to FIG. Furthermore, the flow of the depth calculation process included in the operation will be described with reference to FIG. Next, an application example of the technique according to the embodiment will be described with reference to FIGS. 12A to 12C. Further, another application example according to the embodiment will be described with reference to FIG. Finally, an example of the hardware configuration of the image processing apparatus 100 and the imaging apparatus 200 according to the embodiment will be described with reference to FIGS. 14 and 15. Hereinafter, description will be made sequentially.

[the purpose]
First, with reference to FIG. 1 and FIG. 2, a brief description will be given of a purpose of a technique according to an embodiment described below. FIG. 1 and FIG. 2 are explanatory diagrams schematically showing an example of a continuously captured subject image. In the following description, the example of FIG. 1 is referred to as case 1 and the example of FIG. 2 is referred to as case 2.

  First, refer to FIG. In FIG. 1, a picture P (t1) taken at time t1 and a picture P (t2) taken at time t2 are drawn. However, times t1 and t2 are continuous shooting times. Further, two subject images (subject A and subject B) are captured in the photos P (t1) and P (t2). Here, attention is paid to the “blink” of the subject A and the subject B.

  In case 1, the subject A included in the photograph P (t1) was taken at the moment of blinking. Therefore, the subject A included in the photograph P (t1) is not in a state that matches the photographer's intention. However, the subject B included in the photograph P (t1) does not blink and is in a state (OK) that matches the photographer's intention.

  Thus, the photograph P (t1) including the subject image that does not match the photographer's intention is not a photograph that matches the photographer's intention. On the other hand, the subject A and the subject B included in the photograph P (t2) do not blink and are in a state (OK) that matches the photographer's intention. Therefore, the photograph P (t2) is a photograph that matches the photographer's intention. When the photograph (photo P (t2)) that matches the photographer's intention is included in the plurality of continuously photographed photographs as in the case 1, the photographer selects the photograph P that matches the intention. (T2) may be selected.

  Reference is now made to FIG. In FIG. 2, similarly to FIG. 1, a picture P (t1) taken at time t1 and a picture P (t2) taken at time t2 are drawn. Also in case 2, times t1 and t2 are continuous shooting times. Further, two subject images (subject A and subject B) are captured in the photos P (t1) and P (t2).

  In the case 2 as well, the subject A included in the photograph P (t1) was taken at the moment of blinking. Therefore, the subject A included in the photograph P (t1) is not in a state that matches the photographer's intention. However, the subject B included in the photograph P (t1) does not blink and is in a state (OK) that matches the photographer's intention. That is, the photograph P (t1) including the subject image that does not match the photographer's intention is not a photograph that matches the photographer's intention.

  On the other hand, the subject A included in the photograph P (t2) does not blink and is in a state (OK) that matches the photographer's intention. However, the subject B included in the photograph P (t2) was taken at the moment of blinking. Therefore, the photograph P (t2) is not a photograph that matches the photographer's intention. When the photograph (photo P (t2)) that matches the photographer's intention is included in the plurality of continuously photographed photographs as in the case 1, the photographer selects the photograph P that matches the intention. (T2) may be selected. However, in case 2, this method cannot be used.

  As mentioned above, if there is a photo that matches the photographer's intention among the photos that were shot continuously like a frame-by-frame with multiple shutter releases in a short time, the photo that suits the intention A method of picking up the image can be applied. However, as in the above example, when a photograph that does not include a subject image that is not intended by the photographer is not included in the continuously photographed photographs, it is not possible to apply a technique for picking up a photograph that matches the intention. For example, when a large number of human images are included in the imaging range, such as a group photo, it is difficult to capture the moment when everyone is not blinking, and there is a high probability that the situation as in Case 2 will occur.

  In view of such a current situation, the technique according to the embodiment described below enables a photograph that matches the photographer's intention even in the case of Case 2 to be obtained. However, the technology according to the embodiment is not only applied as a measure against blinking of the subject, but can be applied to various cases as shown in application examples described later. In addition, the application range of the technology according to the embodiment is not limited to a photograph, and includes, for example, moving image frames and various still images constituting broadcast data and video data. However, in the following, for convenience of explanation, it will be referred to as “photographs” taken continuously.

<Embodiment>
Hereinafter, an embodiment of the present invention will be described. The present embodiment relates to a technique for synthesizing a plurality of continuously photographed photographs. However, this technique is also applied to a plurality of photographs taken continuously while changing predetermined parameters. In addition, the present embodiment relates to a technique for synthesizing other photographs continuously photographed in accordance with the rubbing operation by the user rubbing on the screen on which the pictures are displayed. Therefore, the technique relates to a user operation system.

[Principle explanation]
First, the principle of the method of the present embodiment will be described with reference to FIG. FIG. 3 is a principle explanatory diagram for explaining the principle of the technique according to the present embodiment.

  FIG. 3 schematically shows a touch panel and a plurality of continuously taken photographs. These photographs were taken continuously while changing the parameters. However, it should be noted that one parameter, “time”, is changing even if the parameters relating to the imaging mechanism are not changed during continuous shooting. Parameters relating to the imaging mechanism include, for example, depth of field, effects, backlight correction, flash, and the like.

  The depth of field is determined depending on the focal length, the aperture value, and the like. For example, the shorter the focal length, the deeper the depth of field, and the deeper the aperture, the deeper the depth of field. As effects, for example, there are those that apply an effect by a lens filter and various digital filters, and those that combine an arbitrary background image or change an entire photo to sepia. Examples of the effects of the lens filter include a polarizing filter, a neutral density filter, a color correction filter, and a color enhancement filter. For the flash, for example, switching between lighting / extinguishing and the amount of adjustment of the brightness are the parameter values.

  A plurality of photographs taken continuously while changing these parameters are arranged along the parameter changing direction (time-lapse direction) in FIG. Actually, the photographic data is aligned based on the parameter change direction by aligning the photographic data based on the information of the photographing time. However, it should be noted that since the photo data is usually managed side by side based on the information of the photographing time, no special operation is required from the user. The photograph located at the head of the photograph data arranged in this way is displayed on the touch panel. In other words, the first photograph taken among a plurality of continuously photographed pictures is displayed on the touch panel.

  Here, the user can synthesize the displayed photograph with another photograph for the pixel region rubbed with the finger by rubbing the surface of the touch panel with a finger while viewing the photograph displayed on the touch panel. At this time, in this embodiment, friction information such as the position or area rubbed with the finger by the user, the number of times the finger reciprocated in the position or area, the pressing force, the number of fingers, or the moving speed of the finger is combined with the photograph. Used for processing. In the composition process, for example, a photograph with a larger parameter amount is composed according to the number of times the finger has reciprocated or the pressing force. At this time, only the position or area rubbed with the finger by the user is synthesized. Therefore, the user can combine only a part of the photograph with a simple operation.

  For example, in the case 2 shown in FIG. 2, the photograph P (t1) is displayed on the touch panel. As described above, the photograph P (t1) does not match the photographer's intention in that the subject A blinks. On the other hand, in the photo P (t2), the subject B blinks, and this photo does not match the photographer's intention.

  However, if a part of the subject A (around the eyes) in the photograph P (t1) is replaced with a part of the subject A (around the eyes) in the photograph P (t2), the photograph matches the photographer's intention. Is completed. Therefore, the user performs an operation of rubbing a part of the subject A (around the eyes) with a finger while looking at the photograph P (t1) displayed on the touch panel. By this operation, only the rubbed part in the picture P (t1) is gradually combined with the picture P (t2), and finally replaced with a part of the subject A shown in the picture P (t2). As a result, a desired photograph in which neither the subject A nor the subject B blinks is obtained.

  As described above, the technique according to the present embodiment generates a composite image by a series of operations performed by the user by changing a composite position, a composite range, a composite degree, or the like intended by the user by a user's rubbing operation. That's it. Further, the user performs only the operation of rubbing the surface of the touch panel, and the photo composition process can be performed with a simple operation that is not comparable to an image processing application or the like. That is, the technology according to the present embodiment also provides such a highly convenient operation system.

  In the above description, the touch panel is used as an example of display means and input means. Further, as means other than the finger for rubbing the touch panel, for example, a stylus or the like may be used. Furthermore, the first photograph taken continuously may not be displayed on the screen of the touch panel. For example, a photograph at a photographing time designated by a user or a photograph corresponding to a predetermined photographing time may be displayed. . Moreover, when it rubs with a several finger | toe, you may be comprised so that the synthesized photograph may return to the original. Thus, the present embodiment is not limited to this, and various modifications are possible.

  The principle of this embodiment has been described above. Hereinafter, a specific functional configuration and a series of operations of the image processing apparatus 100 for realizing the technique according to the present embodiment will be described. So far, in order to emphasize the concreteness, description has been made using the expression “photograph” as the compositing process target, but in the following description, the expression “image” including a broader meaning is used.

[Functional Configuration of Image Processing Apparatus 100]
Here, the functional configuration of the image processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 4 is an explanatory diagram showing a functional configuration of the image processing apparatus 100 according to the present embodiment. In FIG. 4, an imaging apparatus 200 that provides photograph data continuously captured to the image processing apparatus 100 is also drawn. The imaging apparatus 200 and the image processing apparatus 100 form an image processing system.

  As shown in FIG. 4, the image processing apparatus 100 mainly includes a display input unit 110 and an image composition unit 130.

(Display input unit 110)
The display input unit 110 mainly includes a display unit 112 and a friction position detection unit 114. The touch panel is an example of the display input unit 110.

  The display unit 112 is a display device for displaying an image captured by the imaging device 200. Further, the function of the display unit 112 is realized by the output unit 918 among the hardware resources shown in FIG. Further, the display unit 112 is provided with a friction position detection unit 114. The friction position detection unit 114 may be formed integrally with the display unit 112 or may be formed as a separate body. However, the friction position detection unit 114 is provided on the screen of the display unit 112 so that an image displayed on the display unit 112 can be seen through.

  The friction position detector 114 detects a position rubbed by the user. In particular, the friction position detection unit 114 detects which position or region of the image displayed on the screen of the display unit 112 has been rubbed by the user. Further, the friction position detection unit 114 may be configured to detect a pressing force applied when rubbed by the user. Further, the friction position detection unit 114 has a function of detecting the presence of a user's finger or the like in the space on the screen close to the display unit 112 and recognizing it as a friction position even if the user is not directly touching. You may do it. That is, the friction position mentioned here may include position information for an operation performed by the user so as to draw a sky on the screen of the display unit 112.

  The information on the friction position thus detected is input from the friction position detection unit 114 to the friction information detection unit 132 of the image composition unit 130.

(Image composition unit 130)
The image composition unit 130 mainly includes a friction information detection unit 132, a composition ratio setting unit 134, and a composition processing unit 136. The function of the image composition unit 130 can be realized by the CPU 902 based on a computer program recorded in the ROM 904, the RAM 906, the storage unit 920, or the removable recording medium 928 among the hardware resources shown in FIG.

(Friction information detection unit 132)
The friction information detection unit 132 detects various types of friction information based on the friction position information input by the friction position detection unit 114 of the display input unit 110. The friction information includes, for example, a pixel area rubbed by the user, the number of reciprocating frictions of the pixel area, the speed of friction, the direction of friction, or the shape of the rubbed locus.

  For example, the friction information detection unit 132 accumulates information on the friction position sequentially detected from the friction position detection unit 114, and can detect the friction information by analyzing the accumulated information. . The friction information detected by the friction information detection unit 132 is input to the composition ratio setting unit 134. Hereinafter, for convenience of explanation, it is assumed that “friction frequency” and “pixel area” are detected as friction information.

(Composite ratio setting unit 134)
The composition ratio setting unit 134 sets an image composition ratio based on the friction information detected by the friction information detection unit 132. The image composition ratio here refers to the pixel value of the rubbed pixel area in the image displayed on the display unit 112 (hereinafter referred to as display image) and the pixel value of the image to be combined (hereinafter referred to as image to be combined). It means the synthesis ratio when synthesize. For example, the composition ratio setting unit 134 sets the composition ratio of pixel values so that the weight of the composite image increases as the number of frictions increases. However, when the image to be synthesized is switched according to the number of frictions, the synthesis ratio remains fixed at 1 or 0.

  Here, a method for setting the composition ratio will be described with a specific example. First, three images (image P (t1), image P (t2), image P (t3); t1 <t2 <t3) are captured by the imaging apparatus 200 and input to the image processing apparatus 100. And In addition, it is assumed that the image P (t1) is displayed on the display unit 112. At this time, when the display unit 112 displaying the image P (t1) is rubbed by the user, the friction position detection unit 114 detects the friction position, and the friction information detection unit 132 inputs the number of frictions to the composite ratio setting unit 134. Is done.

  When the number of times of friction is 1, for example, the pixel value D1 of the pixel area of the image P (t1) rubbed by the user and the pixel value D2 of the pixel area of the image P (t2) are combined at 1: 1. Then, the composition ratio is set so that the pixel value D ′ = (D1 + D2) / 2 is calculated. When the number of times of friction is 2, for example, the pixel value D1 of the pixel area of the image P (t1) rubbed by the user and the pixel value D2 of the pixel area of the image P (t2) are combined at 1: 2. Then, the composition ratio is set so that the pixel value D ′ = (D1 + 2 * D2) / 3 is calculated. With such a configuration, it is possible to express the display image gradually changing to a composite image having a large parameter as it is rubbed.

  On the other hand, when switching the image to be synthesized according to the number of frictions, it is as follows. When the number of times of friction is 1, the pixel value D1 of the pixel area of the image P (t1) rubbed by the user is replaced with the pixel value D2 of the pixel area of the image P (t2). That is, the combined pixel value D ′ corresponding to the pixel region is D ′ = D2. If the above expression is used, the synthesis ratio between the pixel value D1 and the pixel value D2 is 0: 1. When the number of frictions is 2, the pixel value D1 of the pixel area of the image P (t1) rubbed by the user is replaced with the pixel value D3 of the pixel area of the image P (t3). That is, the combined pixel value D ′ corresponding to the pixel region is D ′ = D3. If said expression is used, the synthetic | combination ratio between pixel value D1 and pixel value D3 will be 0: 1.

  The above setting method is an example, and the present embodiment is not limited to this. However, by using such a setting method, a composition ratio corresponding to friction information can be set. The image composition method will be described in detail later. The composition ratio set in this way is input to the composition processing unit 136.

(Composition processing unit 136)
The composition processing unit 136 causes the display unit 112 to display an image corresponding to the first or predetermined time point in the group of continuously captured images. Thereafter, the user performs an operation of rubbing on the screen of the display unit 112 while viewing the image displayed on the display unit 112. Friction information is detected according to the rubbing operation, and a composition ratio is set by the composition ratio setting unit 134 according to the friction information. When this combination ratio is input, the combination processing unit 136 combines the images continuously captured by the imaging device 200 with the image displayed on the display unit 112 based on the combination ratio set by the combination ratio setting unit 134. . Further, the composition processing unit 136 displays the composited image on the display unit 112 and presents it to the user again.

(Detailed explanation of image composition processing)
Here, the above-described image composition processing will be described in more detail with reference to FIGS. FIG. 5 is an explanatory diagram showing a matrix corresponding to the number of pixels of an image. FIG. 6 is an explanatory diagram illustrating an example of a brush matrix. FIG. 7 is an explanatory diagram illustrating an example of an input operation. FIG. 8 is an explanatory diagram illustrating an example of a depth matrix.

  First, referring to FIG. The image composition unit 130 holds the matrix information shown in FIG. This matrix indicates a data storage area having the same size as the number of pixels of the image, and is, for example, matrix data in which depth information is held in each cell. The depth referred to here is an index indicating which image is synthesized (displayed) at what ratio. Initially, a predetermined value is stored in this matrix, and the depth stored in the matrix is updated according to the frictional motion by the user. In accordance with this depth, the shooting time of the combined image combined by the image combining unit 130 is determined, or the ratio of the combined image combined with the display image is set.

  Reference is now made to FIG. FIG. 6 is a brush matrix for updating a matrix value corresponding to a pixel area rubbed by the user. That is, this brush matrix is for setting the depth added to the value of the matrix. In this example, a brush matrix having a size of 5 pixels × 5 pixels is set. Further, this brush matrix has a center value of 1.0, and the depth is set so that the value decreases toward the outside. When the position rubbed by the user is detected, the position is adjusted so that the pixel at the detected position matches the center of the brush matrix, and the value of the brush matrix is added to the value of the matrix.

  First, as shown in FIG. 7, a certain pixel area is rubbed by the user. In FIG. 7, both are schematically drawn so that the correspondence between the position rubbed by the user's finger and the position of the matrix can be understood. Actually, since the matrix is matrix data, the matrix is not directly pressed as shown in FIG. However, the image composition unit 130 recognizes the position on the matrix by acquiring information on the position pressed by the user, as in the image shown in FIG. When the user performs a rubbing operation in this manner, the image synthesis unit 130 adds the depth stored in the brush matrix to the matrix value. This is shown in FIG.

  The example of FIG. 8 shows a matrix in which the depth 10.0 is stored in each cell of the matrix as an initial value, and the value of the brush matrix is added to the position rubbed by the user. As shown in FIG. 7, when a certain pixel area is rubbed by the user, the image composition unit 130 adjusts the position of the brush matrix so as to fit the pixel area. Then, the image composition unit 130 adds the depth value stored in each cell of the brush matrix to the depth value stored in each cell of the matrix corresponding to each cell. When the brush matrix is set as shown in FIG. 6, the matrix after the addition is as shown in FIG.

  If the same pixel region is rubbed repeatedly by the user, the matrix values of the rubbed portion are repeatedly added. Therefore, the depth value stored in the matrix increases as the number of frictions increases. Further, the brush matrix value may be added a plurality of times according to the time during which the user keeps touching the same pixel region or the pressure with which the pixel region is pressed. Furthermore, when the user rubs a certain pixel area, the brush matrix may be added more frequently as the friction speed is smaller or larger. With this configuration, the image composition unit 130 can control the matrix values in accordance with various types of friction information.

  The depth for each pixel obtained in this way is used to determine the displayed image and the composition ratio. For these determinations, for example, a conversion table as shown in FIG. 9 is used. The conversion table in FIG. 9 is for determining an image to be displayed. This conversion table is composed of a column indicating a numerical range in which values of the matrix are included, and a column indicating an image classification corresponding to the numerical range.

  For example, when the value (depth) of the matrix corresponding to a certain pixel is 11.0, this pixel is included in the numerical value range 10.0 to 20.0 corresponding to “image 1, image 2”. Therefore, the pixel value of this pixel is calculated using the pixel values of image 1 and image 2 corresponding to each other. Further, the composition ratio when the pixels of the image 1 and the image 2 are combined is determined by the composition ratio setting unit 134 according to the value of the matrix. When the depth is 11.0, the composition ratio between the pixel value D1 of the image 1 and the pixel value D2 of the image 2 is determined to be, for example, D1: D2 = 9: 1. Therefore, the pixel value D ′ of the composite image is D ′ = 0.9 * D1 + 0.1 * D2.

  When the pixel value D has R, G, and B values corresponding to the three primary colors, the combined pixel value obtained by combining the pixel value D1 (R1, G1, B1) and the pixel value D2 (R2, G2, B2) D ′ (R ′, G ′, B ′) is expressed as in the following formula (1). As a result, the pixel areas rubbed by the user in the image 1 and the image 2 are alpha blended and displayed in this way. Note that the pixel values of the combined image are calculated by similarly combining the pixel values for other color space representations such as YCbCr and CMYK.

  Attention should be paid again to the conversion table shown in FIG. In the conversion table shown in FIG. 9, when the value of the matrix is a numerical value range of 0.0 to 10.0, or when the numerical value range is 20.0 to 30.0, each displayed image is an image 1 Alternatively, it is set so that only the image 2 is obtained. In other words, even if the rubbing operation by the user is repeated and the matrix value gradually increases, the pixel value of the rubbed pixel area remains in the synthesized image while the added depth is included in these numerical ranges. It becomes fixed with a thing. This state represents a so-called “reservoir”.

  For example, in the case 2 shown in FIG. 2, the user wants to replace only a part of the photograph P (t2) (around the eye of the subject A) taken later in time with the photograph P (t1). Would think. If there is no “reservoir” described above, the user carefully searches for a moment when the photograph P (t2) is displayed alone while repeating the rubbing operation while checking the displayed composite image. Further, if the moment is removed, the pixel values are synthesized and the image quality is degraded. However, if the “reservoir” is set, the pixel value does not change even if the rubbing operation is performed to some extent after the photograph P (t2) is displayed alone. Therefore, the user can easily and safely find the time point when the photograph P (t2) is displayed alone by repeating the rubbing operation while referring to the displayed image.

  The details of the image composition processing have been described above. As described above, a part of the images continuously captured by the imaging apparatus 200 is combined by the combination ratio setting unit 134 and the combination processing unit 136 of the image combining unit 130 according to the user's rubbing operation. The synthesized image is input to the display unit 112 of the display input unit 110 and presented to the user. However, the combined image is displayed on the display unit 112 every time the image is combined by the combining processing unit 136.

  As described above, the image processing apparatus 100 according to the present embodiment synthesizes a plurality of continuously shot images according to the frictional motion by the user. In particular, the image processing apparatus 100 is characterized in that the image to be synthesized is switched according to the friction information and the composition ratio is adjusted. As described above, by applying the configuration of the image processing apparatus 100, an operation system in which only a part of an image is smoothly synthesized according to the frictional motion by the user is realized, and the user matches the intention with a simple operation. Images can be obtained.

[Functional Configuration of Imaging Device 200]
Here, the functional configuration of the imaging apparatus 200 will be briefly described with reference to FIG. However, the function of the imaging apparatus 200 is realized by the hardware configuration shown in FIG.

  Among the functions of the imaging apparatus 200, the function of the parameter changing unit 210 is important in the present embodiment. The parameter changing unit 210 has a function of changing various controllable parameters for each image when the imaging apparatus 200 executes continuous shooting. Examples of parameters controlled by the parameter changing unit 210 include, for example, depth of field, effects, backlight correction, and flash. The parameter changing unit 210 continuously captures the subject image while gradually changing these parameters. However, the parameter change may be automatically executed by the parameter changing unit 210 or manually by the user.

[Description of Operation of Image Processing Apparatus 100]
Next, the operation of the image processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 10 is an explanatory diagram schematically showing the operation of the image processing apparatus 100 according to the present embodiment. However, here, an example of correcting blinking of a human photograph is described.

  In FIG. 10, a picture P (t1) taken at time t1 and a picture P (t2) taken at time t2 are drawn. Times t1 and t2 are continuous shooting times. Further, two subject images (subject A and subject B) are captured in the photos P (t1) and P (t2).

  The subject A included in the photo P (t1) was taken at the moment of blinking. On the other hand, the subject B included in the photograph P (t1) does not blink and is in a state (OK) that matches the photographer's intention. In addition, the subject A included in the photograph P (t2) does not blink, and is in a state (OK) that matches the photographer's intention. On the other hand, the subject B included in the photograph P (t2) was taken at the moment of blinking. In this case, the user will want to replace the eye portion of the subject B in the photo P (t2) with that of the photo P (t1).

  At this time, as shown in FIG. 10, the user simply rubs around the eyes of the subject B in the photo P (t2), and synthesizes the image of the photo P (t1) corresponding to the friction area. Can do. That is, when the pixel area of the photograph P (t2) is rubbed by the user, the image processing apparatus 100 combines the images of the photograph P (t2) and the photograph P (t1) corresponding to the pixel area (S100). By such a composition process, a composite image in which neither subject A nor subject B blinks is generated as desired by the user.

  In this way, the user can advance the time by rubbing around the eyes of the blinking person. In this way, by changing only the rubbed portion, not the entire image, the depth can be changed only by the position and range intended by the user.

(Details of composition processing S100)
Here, the flow of the composition processing S100 performed by the image processing apparatus 100 will be described in more detail with reference to FIG. FIG. 11 is an explanatory diagram showing the flow of the composition process performed by the image processing apparatus 100 according to the present embodiment. The series of processing shown here is realized mainly by the function of the image composition unit 130 in the functional configuration of the image processing apparatus 100.

  As shown in FIG. 11, first, the image composition unit 130 aligns the position of the brush matrix with the pixel area rubbed by the user, and adds the depth value stored in each cell of the brush matrix to the matrix value. (S102). Next, the image composition unit 130 selects which image is to be composed (displayed) by referring to the conversion table and selecting an image corresponding to the matrix value (S104). Next, the image composition unit 130 calculates the pixel value of the pixel region rubbed by the user based on the matrix value, fills the corresponding pixel as the pixel value constituting the composited display image (S106), and a series of Terminate the process. By these processes, a composite image corresponding to the user's friction motion is generated.

[Application Example 1: Fireworks shooting]
Next, an application example of the present embodiment will be described with reference to FIGS. 12A to 12C. FIG. 12A to FIG. 12C are explanatory diagrams illustrating an example when the technique according to the present embodiment is applied to a scene of shooting fireworks. The fireworks shooting scene shown here is an example in the case where the subject image intended by the photographer appears with a time difference. As described above, the image processing apparatus 100 according to the present embodiment is also preferably used for a scene in which the state of the subject cannot be controlled.

  FIG. 12A to FIG. 12C schematically show images continuously shot aiming at the moment when two fireworks burst. In addition, these series of images are images of a state where the left-side fireworks burst first, and the right-side fireworks burst several seconds later.

  First, refer to FIG. 12A. The shooting time point in FIG. 12A is the moment when the left-side fireworks burst. As for the fireworks on the left side, as the photographer intended, it was possible to shoot with the fireworks fully open. However, at this point in time, the right fireworks are before the explosion, and the fireworks diameter is smaller than what the photographer intended.

  Reference is now made to FIG. The shooting time point in FIG. 12B is the moment when the right fireworks burst. As for the fireworks on the right side, as the photographer intended, it was possible to shoot with the fireworks fully open. However, at the time of this shooting, the fireworks on the left side have already disappeared. Therefore, the photographer will want to replace the right fireworks shown in the image of FIG. 12A with the right fireworks shown in FIG. 12B. Therefore, the technique of the present embodiment that can advance the time only by the rubbed portion is used.

  First, the photographer performs an operation of rubbing within the frame surrounding the right fireworks in the image shown in FIG. 12A. Then, since the time advances only within the rubbed frame, the image of the right fireworks in FIG. 12B is gradually synthesized and displayed. When further rubbing, as shown in FIG. 12C, the image of the right fireworks shown in FIG. 12B is completely synthesized, and a composite image in a state where the left fireworks and the right fireworks burst together is generated. Originally, it is not possible to capture the moment when two fireworks bursting at a time difference burst at the same time. However, using the technique of this embodiment, such a composite image can be easily generated.

[Application example 2: landscape photography]
Next, an application example of this embodiment will be described with reference to FIG. FIG. 13 is an explanatory diagram showing an example in which the technology according to the present embodiment is applied to a scene shooting scene. The scene shooting scene shown here is an example in which there are subjects in front and back, and the front and back cannot be focused simultaneously. As described above, the image processing apparatus 100 according to the present embodiment is preferably used even for scenes that are difficult to shoot.

  FIG. 13 shows a photograph P (t1) and a photograph P (t2). The photograph P (t1) is a photograph taken so that the focus is in the back. On the other hand, the photograph P (t2) is a photograph taken so that the front is in focus. When shooting a landscape, focusing on a building or mountain in the back makes it impossible to focus on the leaves of the trees in the foreground. On the other hand, when focusing on the leaves of the trees in the foreground, the building and mountains in the back will not be focused. Even in such a scene, the photographer will want to shoot an image in which the back and front are both in focus. Therefore, the technique of the present embodiment that can move the parameter (focus) only by the rubbed portion is used.

  First, the photographer performs continuous shooting while changing the field seismic intensity. Then, the photographer performs an operation of rubbing within the frame surrounding the front subject in the photograph P (t1). Then, since the field seismic intensity advances only within the rubbed frame, the image of the photograph P (t2) is gradually synthesized and displayed. Further rubbing, as shown as a composite image in FIG. 13, the foreground image included in the photograph P (t2) is completely combined, and a composite image in which both the back subject and the foreground subject are in focus is obtained. Generated. Originally, it is not possible to obtain an image in which the back and front are simultaneously focused, but by using the technique of this embodiment, such a composite image can be easily generated.

  The two application examples according to the present embodiment have been described above. Of course, the application range of the present embodiment is not limited to this, and various applications are possible according to the type of friction information and the type of parameter.

[Hardware Configuration Example of Image Processing Apparatus 100]
The function of each component included in the image processing apparatus 100 can be realized by, for example, an information processing apparatus having a hardware configuration illustrated in FIG. 14 using a computer program for realizing the above function. is there. FIG. 14 is an explanatory diagram illustrating a hardware configuration of an information processing apparatus capable of realizing the functions of the respective components of the image processing apparatus 100 described above. The form of this information processing apparatus is arbitrary. For example, a personal computer, a mobile phone, a personal information system such as a personal handy phone system (PHS), a personal digital assistant (PDA), a game machine, or various information appliances This includes forms.

As shown in FIG. 14 , the information processing apparatus mainly includes a CPU (Central Processing Unit) 902, a ROM (Read Only Memory) 904, a RAM (Random Access Memory) 906, a host bus 908, and a bridge. 910, an external bus 912, an interface 914, an input unit 916, an output unit 918, a storage unit 920, a drive 922, a connection port 924, and a communication unit 926.

  The CPU 902 functions as, for example, an arithmetic processing unit or a control unit, and controls the overall operation or a part of each component based on various programs recorded in the ROM 904, the RAM 906, the storage unit 920, or the removable recording medium 928. . The ROM 904 stores, for example, a program read by the CPU 902 and data used for calculation. The RAM 906 temporarily or permanently stores, for example, a program that is read into the CPU 902 and various parameters that change as appropriate when the program is executed. These components are connected to each other by, for example, a host bus 908 capable of high-speed data transmission. The host bus 908 is connected to an external bus 912 having a relatively low data transmission speed via a bridge 910, for example.

  The input unit 916 is an operation unit such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. The input unit 916 may be remote control means (so-called remote controller) capable of transmitting a control signal using infrared rays or other radio waves. Note that the input unit 916 includes an input control circuit for transmitting information input using the above-described operation means to the CPU 902 as an input signal.

  The output unit 918 is, for example, a display device such as a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), or an ELD (Electro-Luminescence Display), an audio device such as a speaker, a headphone, or the like. It is a device capable of visually or audibly notifying acquired information to a user, such as a mobile phone or a facsimile.

  The storage unit 920 is a device for storing various data, and includes, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. Is done.

  The drive 922 is a device that reads information recorded on a removable recording medium 928 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable recording medium 928. The removable recording medium 928 is, for example, a DVD medium, a Blu-ray medium, an HD DVD medium, a compact flash (CF) (registered trademark), a memory stick, or an SD memory card (Secure Digital memory card). Of course, the removable recording medium 928 may be, for example, an IC card (Integrated Circuit Card) on which a non-contact IC chip is mounted, an electronic device, or the like.

  The connection port 924 is a port for connecting an external connection device 930 such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal. is there. The external connection device 930 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

  The communication unit 926 is a communication device for connecting to the network 932. For example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB), optical communication A router, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The network 932 connected to the communication unit 926 is configured by a wired or wireless network, such as the Internet, home LAN, infrared communication, visible light communication, broadcast, or satellite communication.

[Hardware Configuration Example of Imaging Device 200]
Next, a hardware configuration example of the imaging apparatus 200 will be described. The function of each component included in the imaging apparatus 200 can be realized by, for example, a hardware configuration illustrated in FIG. FIG. 15 is an explanatory diagram illustrating an example of a hardware configuration capable of realizing the function of each component of the imaging apparatus 200 described above.

  As illustrated in FIG. 15, the imaging apparatus 200 mainly includes a control unit (for example, a CPU; Central Processing Unit) 802, a position acquisition unit 804, a RAM (Random Access Memory) 806, and a ROM (Read Only Memory). 808, a bus 810, an imaging unit 812, an image processing unit 814, a display unit 816, and a storage medium (for example, Flash Memory) 818.

  Although not clearly shown in the figure, an input unit that receives an operation by the user may be further provided. The function of this input part is implement | achieved by operation means, such as a mouse | mouth, a keyboard, a touch panel, a button, a switch, and a lever, for example. The input unit may be remote control means (so-called remote controller) capable of transmitting a control signal using infrared rays, other radio waves, or light waves.

  The control unit 802 is realized by, for example, a CPU or the like, and controls the overall operation of each component or a part thereof based on various programs recorded in the RAM 806, the ROM 808, the recording medium 818, or the like. The ROM 808 stores, for example, a program read by the control unit 802 and data used for calculation. The RAM 806 temporarily or permanently stores, for example, a program that is read into the control unit 802 and various parameters that change as appropriate when the program is executed. These components are connected to each other by, for example, a bus 810 capable of high-speed data transmission.

The position acquisition unit 804 is means for acquiring position information of the imaging device. For example, the position acquisition unit 804 may access a communicable base station and acquire information on an area covered by the base station as position information, or a positioning satellite system (GPS; Global Positioning System). ) Or the like. The position information is, for example, information that can specify the latitude, longitude, or area where the imaging device 200 is located.

  The imaging unit 812 is a unit that acquires an image of a subject via an optical system. The imaging unit 812 is mainly composed of a lens 822, a CCD (Charge Coupled Device) 824, an AF control unit 826, and an AE control unit 828.

  The lens 822 constitutes an optical system and condenses the light reflected by the subject on the CCD 824. The CCD 824 is a solid-state imaging device that is composed of a semiconductor element, receives light collected by the lens 822 by each element, photoelectrically converts the received light, and outputs it. Since the CCD 824 outputs an electrical signal according to the intensity of light received by each element, an image signal can be formed based on the electrical signal. Note that the imaging apparatus 200 may use an image sensor formed of another semiconductor element instead of the CCD 824. The AF control unit 826 is an autofocus mechanism for automatically adjusting the focus. The AE control unit 828 is an automatic exposure (AE) control mechanism for automatically adjusting the shutter speed and the aperture value.

  The image processing unit 814 can convert the image signal output from the CCD 824 into a format of image data that can be displayed on the display unit 816. For example, the image processing unit 814 can convert the image signal into image data corresponding to an encoding format such as BMP, GIF, JPEG, PICT, or PING. Further, the image processing unit 814 changes the size of image data, adjusts brightness and color tone, or adjusts gamma correction and sharpness. Furthermore, the image processing unit 814 can also generate moving image data from a plurality of image signals continuously captured by the imaging unit 812.

  The display unit 816 is, for example, a display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), or an ELD (Electro-Luminescence Display), and visually notifies the acquired information to the user. It is a device that can.

  The recording medium 818 is configured by a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like. For example, the recording medium 818 includes a DVD medium, a Blu-ray medium, an HD DVD medium, a Compact Flash (registered trademark) (CF; CompactFlash), a memory stick, an SD memory card (Secure Digital memory card), or an HDD (Hard Disk Drive). Etc. Further, the imaging apparatus 200 may be configured to use, for example, an IC card (Integrated Circuit Card) mounted with a non-contact IC chip, an electronic device, or the like as the recording medium 818.

[Summary]
Finally, the functional configuration of the image processing apparatus according to the present embodiment and the operational effects achieved by the functional configuration will be briefly summarized.

  First, the functional configuration of the image processing apparatus according to the present embodiment can be expressed as follows.

  The image processing apparatus includes a display screen on which a first image included in continuously captured images is displayed, a friction position detection unit that detects a friction position on the display screen rubbed by a user, An image combining unit configured to combine the pixel region of the first image corresponding to the friction position and the pixel region of the second image included in the image group; However, the image synthesizing unit synthesizes the first image so that a photographing time difference between the first image and the second image increases as the friction amount rubbed by the user increases. A function of switching the second image;

  As described above, the image processing apparatus has a function of displaying the first image included in the continuously photographed image group on the display screen. As the first image, for example, an image having the newest shooting time is selected from among images included in the image group. Further, the image processing apparatus has a function of detecting a friction position on the display screen rubbed by a user by a friction position detection unit. When the friction position is detected by the friction position detection unit, the pixel region designated by the user is specified.

  Furthermore, the image processing apparatus combines the pixel region of the first image corresponding to the friction position and the pixel region of the second image included in the image group by an image combining unit. By being configured in this way, a part of the second image with different shooting time points is synthesized with the first image. At this time, since only the pixel region designated by the user is combined with the first image, it is possible to generate an image that is actually difficult to shoot. Furthermore, since the compositing process executed by the image compositing unit is realized by a friction operation by the user, a simple operation system with high convenience is provided. In particular, since the user can specify the pixel area, the composition ratio setting operation, and the composite image selection operation by only a series of frictional operations, exceptionally high convenience is provided.

  Further, the image composition unit composes the first image so that a photographing time difference between the first image and the second image increases as the friction amount rubbed by the user increases. Switch the second image. With this configuration, the user can sequentially switch the synthesized images by repeating the rubbing operation. As a result, the composite image switching process is also incorporated into a series of friction operations, and a more convenient operation system is realized.

  The image composition unit increases the composition ratio of the second image to the first image as the friction amount increases, and the second image is displayed when the friction amount exceeds a predetermined threshold. It may be configured to switch to an image having a larger photographing time difference from the first image. With this configuration, the process for switching the combined image and the process for adjusting the composite ratio between the display image and the combined image are realized according to a series of friction operations. In other words, the composite image seamlessly changes from the display image to the composite image by the continuous friction operation. As a result, the user can generate a composite image by a more intuitive process.

  Further, the image composition unit displays the second image before switching in the pixel area until the friction amount exceeds a predetermined threshold by a predetermined amount, and when the predetermined amount is exceeded, the second image is displayed. It may be configured to switch. With this configuration, after a part of the first image is completely switched to the second image, the second image is not switched for a predetermined time even if the rubbing operation is continued. You will be able to find the timing easily. As a result, when an image in which a part of the first image is completely replaced with the second image is intended, the timing can be accurately captured by the amount of “reservoir”, and the user can rub with confidence. Will be able to proceed.

  The friction amount is an amount determined by one amount or a combination of plural selected from the number of times the pixel area is rubbed, the pressing force during friction, the friction direction, and the friction speed. Also good. Furthermore, the image group may be a plurality of images that are continuously captured while changing arbitrary parameters that can be changed at the time of capturing. The parameter may be a control parameter for controlling a depth of field, an image effect, or a flash. However, the configuration of the image processing apparatus is not limited to these examples, and various modifications can be made according to the function and configuration of the imaging apparatus that captures an image group.

[effect]
Here, the effects obtained by applying this embodiment will be briefly summarized. However, specific effects obtained by each configuration of the present embodiment have already been described in the above description. Therefore, what is described here is a summary of these specific effects. First, when the technique according to the above-described embodiment is used, an effect that it is possible to synthesize continuously captured images with an easy operation system is obtained. Further, it is possible to obtain an effect that an image can be synthesized by changing the position or range intended by the user or the synthesis degree by a series of frictional operations. Furthermore, it is possible to easily generate an image that is actually difficult to shoot.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to the example which concerns. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

It is explanatory drawing which shows an example of the to-be-photographed object image. It is explanatory drawing which shows an example of the to-be-photographed object image. It is a principle explanatory view of image processing technology concerning one embodiment of the present invention. It is explanatory drawing which shows the function structure of the image processing apparatus which concerns on this embodiment. It is explanatory drawing which shows an example of the matrix corresponding to the pixel of an image pick-up element. It is explanatory drawing which shows an example of the brush matrix which concerns on this embodiment. It is explanatory drawing which shows an example of input operation which concerns on this embodiment. It is an example of the depth matrix which concerns on this embodiment. It is explanatory drawing which shows an example of the conversion table which concerns on this embodiment. It is operation | movement explanatory drawing of the image processing apparatus which concerns on this embodiment. It is explanatory drawing which shows the flow of the depth calculation process which concerns on this embodiment. It is explanatory drawing which shows the example of the one subject image image | photographed by continuous imaging | photography. It is explanatory drawing which shows the example of the one subject image image | photographed by continuous imaging | photography. It is explanatory drawing which shows an example of the synthesized image which concerns on this embodiment. It is explanatory drawing which shows one application example of the image processing technique which concerns on this embodiment. It is explanatory drawing which shows the hardware structural example of the image processing apparatus (information processing apparatus) which concerns on this embodiment. It is explanatory drawing which shows the hardware structural example of the imaging device which concerns on this embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 110 Display input part 112 Display part 114 Friction position detection part 130 Image composition part 132 Friction information detection part 134 Composition ratio setting part 136 Composition processing part 200 Imaging device 210 Parameter change part

Claims (7)

A display screen on which a first image included in a group of continuously shot images is displayed;
A friction position detector for detecting a friction position on the display screen rubbed by the user;
An image synthesis unit that synthesizes the pixel area of the first image corresponding to the friction position and the pixel area of the second image included in the image group;
With
The image combining unit combines the first image with the first image so that a difference in photographing time between the first image and the second image increases as the friction amount rubbed by the user increases. An image processing apparatus that switches images.   The image composition unit increases the composition ratio of the second image to the first image as the friction amount increases, and the second image is displayed when the friction amount exceeds a predetermined threshold. The image processing apparatus according to claim 1, wherein the image processing apparatus switches to an image having a larger photographing time difference from one image.   The image composition unit displays the second image before switching in the pixel area until the friction amount exceeds a predetermined threshold by a predetermined amount, and switches the second image when the predetermined amount is exceeded. The image processing apparatus according to claim 2.   The amount of friction is an amount determined by one amount or a combination of a plurality of times selected from the number of times the pixel area is rubbed, a pressing force during friction, a friction direction, and a friction speed. The image processing apparatus according to 3.   The image processing apparatus according to claim 1, wherein the image group is a plurality of images that are continuously captured while changing arbitrary parameters that can be changed at the time of capturing.   The image processing apparatus according to claim 5, wherein the parameter is a control parameter for controlling a depth of field, an image effect, or a flash. A friction position detecting step of detecting a friction position on the display screen rubbed by a user when a first image included in a group of continuously photographed images is displayed on the display screen;
An image synthesis step in which a pixel area of the first image corresponding to the friction position and a pixel area of a second image included in the image group are synthesized;
Including
In the image synthesizing step, the second image is synthesized with the first image such that a difference in photographing time between the first image and the second image increases as the friction amount rubbed by the user increases. An image processing method in which images are switched.

Images