JP6610713B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

Conventionally, a composite image technique capable of simultaneously displaying a plurality of images obtained from a plurality of imaging devices is known.
For example, Patent Document 1 describes a technique in which a part of a removed area in one image is made transparent and combined with the other image.

JP 2009-253554 A

  However, since the technique described in Patent Document 1 is simply a technique for synthesizing a plurality of images, there is a possibility that the synthesis result does not match the user's preference.

  An object of the present invention is to be able to generate a composite image that is more suited to user preferences when combining a plurality of images.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention includes:
Image acquisition means for sequentially acquiring a first image captured in the first direction and a second image captured in the second direction simultaneously;
First display control means for simultaneously displaying the first image and the second image acquired by the image acquisition means on the display means simultaneously;
First input means for inputting a first predetermined instruction during display of the first image and the second image by the first display control means;
When the first predetermined instruction is input by the first input means, the display of the first image and the second image is fixed while the display of one of the first image and the second image is fixed. Second display control means for continuously displaying the other of the two images;
Second input means for inputting a second predetermined instruction during display of the first image and the second image displayed on the display means by the second display control means;
The first image corresponding to the time when the first predetermined instruction is input by the first input means and the second image corresponding to the time when the second predetermined instruction is input by the second input means. A synthesis means for synthesizing the image;
It is characterized by providing .

  According to the present invention, when combining a plurality of images, it is possible to generate a combined image that suits the user's preference.

1A and 1B are schematic diagrams illustrating an external configuration of an image processing apparatus according to an embodiment of the present invention, in which FIG. 1A is a front view and FIG. 1B is a rear view. 1 is a block diagram illustrating a hardware configuration of an image processing apparatus according to an embodiment of the present invention. It is a functional block diagram which shows the functional structure for performing a two-way imaging | photography process among the functional structures of the image processing apparatus of FIG. It is a schematic diagram which shows an example of the front image imaged by 16 A of 1st imaging parts. It is a schematic diagram which shows an example of the back image imaged by the 2nd imaging part 16B. It is a schematic diagram which shows the state by which the empty area was specified in the front image. It is a schematic diagram which shows the state by which the candidate of the synthesis position was specified in the front image. It is a schematic diagram which shows the state by which the background image was synthesize | combined with the front image. 4 is a flowchart for explaining the flow of a two-way imaging process executed by the image processing apparatus of FIG. 2 having the functional configuration of FIG. 3.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1A and 1B are schematic views showing an external configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 1A is a front view and FIG. 1B is a rear view.
FIG. 2 is a block diagram showing a hardware configuration of the image processing apparatus according to the embodiment of the present invention.
The image processing apparatus 1 is configured as a digital camera, for example.

  The image processing apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, a first imaging unit 16A, A second imaging unit 16B, an input unit 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21 are provided.

  The CPU 11 executes various processes according to a program recorded in the ROM 12 such as a program for two-way imaging processing or a program loaded from the storage unit 19 to the RAM 13.

  The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

The CPU 11, ROM 12, and RAM 13 are connected to each other via a bus 14.
An input / output interface 15 is also connected to the bus 14. Connected to the input / output interface 15 are a first imaging unit 16A, a second imaging unit 16B, an input unit 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21.

The first imaging unit 16A is installed on the front side of the image processing device 1 (the surface on the opposite side to the display screen of the output unit 18), and images the subject on the front side of the image processing device 1. Hereinafter, a captured image captured by the first imaging unit 16A is referred to as a “front image”.
The second imaging unit 16B is installed on the rear side of the image processing apparatus 1 (the same side as the display screen of the output unit 18), and images the subject on the rear side of the image processing apparatus 1. Since it is assumed that the second imaging unit 16B mainly captures the face of the photographer, the entire face of the photographer has an angle of view with the photographer holding the image processing apparatus 1 for photographing. A lens with a focal length is provided. Hereinafter, a captured image captured by the second imaging unit 16B is referred to as a “rear image”.

Although not shown, the first imaging unit 16A and the second imaging unit 16B include an optical lens unit and an image sensor.
The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. Through various signal processing, a digital signal is generated and output as an output signal of the first imaging unit 16A or the second imaging unit 16B.
Such an output signal of the first imaging unit 16A or the second imaging unit 16B is hereinafter referred to as “captured image data”. Data of the captured image is appropriately supplied to the CPU 11 or an image processing unit (not shown).

The input unit 17 includes various buttons and the like, and inputs various types of information according to user instruction operations.
The output unit 18 includes a display, a speaker, and the like, and outputs images and sounds.
The storage unit 19 is configured by a hard disk or a DRAM (Dynamic Random Access Memory) or the like, and stores facial feature data, various image data, and the like, which will be described later.
The communication unit 20 controls communication performed with other devices (not shown) via a network including the Internet.

A removable medium 31 composed of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 21. The program read from the removable medium 31 by the drive 21 is installed in the storage unit 19 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 19 in the same manner as the storage unit 19.
Although not shown, the image processing apparatus 1 can appropriately include hardware for assisting photographing such as a strobe light emitting device.

FIG. 3 is a functional block diagram showing a functional configuration for executing the two-way imaging process among the functional configurations of such an image processing apparatus 1.
In the two-way shooting process, when the front side subject is shot by the first imaging unit 16A, the rear side subject is shot together by the second imaging unit 16B, and the captured image of the rear side subject is captured on the front side. A series of processes to be combined with a captured image of a subject.

When the two-way imaging process is executed, as shown in FIG. 3, in the CPU 11, the imaging control unit 51, the face recognition unit 52, the free area analysis unit 53, the synthesis position analysis unit 54 and the image synthesis unit 55 And the display control part 56 functions.
Further, a face recognition information storage unit 71 and an image storage unit 72 are set in one area of the storage unit 19.
The face recognition information storage unit 71 stores a plurality of facial feature data that are related to each other. For example, the face recognition information storage unit 71 stores facial feature data of all family members who use the image processing apparatus 1.
The image storage unit 72 appropriately stores image data captured by the first imaging unit 16A, image data captured by the second imaging unit 16B, and composite image data combined by the image combining unit 55. .

The imaging control unit 51 causes the first imaging unit 16A and the second imaging unit 16B to acquire live view images of the front image and the rear image.
Further, when the shutter button is half-pressed, the imaging control unit 51 fixes parameters such as a focus position, an aperture, and an exposure to values that are assumed at the time of shooting, and is predicted to be acquired as a captured image. (Hereinafter referred to as “front image during half shutter”) is caused to be imaged by the first imaging unit 16A.
In addition, the imaging control unit 51 causes the first imaging unit 16A to capture a front image for recording when the shutter button is fully pressed. Furthermore, the imaging control unit 51 causes the second imaging unit 16B to capture a recording back image to be combined with the front image when an operation for instructing imaging of the recording back image is performed. The imaging control unit 51 causes the first imaging unit 16A to capture a front image when the shutter button is fully pressed, and then captures a back image for recording when the shutter button is fully pressed again. When an instruction operation is performed, the second imaging unit 16B is caused to capture a back image for recording.

FIG. 4 is a schematic diagram illustrating an example of a front image captured by the first imaging unit 16A. FIG. 5 is a schematic diagram illustrating an example of a back image captured by the second imaging unit 16B.
4 and 5 show an example in which a group photo of a plurality of persons included in the front image of FIG. 4 is taken by a person included in the back image of FIG. Note that the person (subject F7) included in the rear image in FIG. 5 is related to a part of the plurality of persons (subjects F1 to F6) included in the front image in FIG. 4 (for example, a relationship such as “family”). The facial feature data is stored in the face recognition information storage unit 71.

The face recognition unit 52 recognizes the face of the subject included in the front image and the back image when the shutter button is pressed halfway. In addition, the face recognition unit 52 refers to the facial feature data stored in the face recognition information storage unit 71 to detect a face having relevance included in the front image and the back image.
The vacant area analysis unit 53 analyzes the arrangement of the subject in the front image and identifies an area where the main subject is not captured as an vacant area. For example, the free area analysis unit 53 detects the main subject and the background based on the in-focus state and the like, and specifies the area where the main subject is not copied (that is, the background area) as the free area.

FIG. 6 is a schematic diagram showing a state where an empty area is specified in the front image.
As shown in FIG. 6, in the front image of FIG. 4, the central area where a plurality of persons gather is specified as the area where the main subject is photographed, and the surrounding area is an empty area (the hatched line in FIG. 6). Part).

The composite position analyzing unit 54 analyzes the front image based on the face area having the relationship recognized by the face recognizing unit 52 and the free area specified by the free area analyzing unit 53, The position (synthesizing position) for synthesizing images is specified. Specifically, the composite position analysis unit 54 specifies a face area of the subject having relevance to the face of the subject included in the back image in the front image, and selects a free area close to the specified face area. . If the free area is specified over a wide range, it is possible to select a free area by partially specifying a set size range.
At this time, when a plurality of face areas of the subject having relevance to the face of the subject included in the background image are specified in the front image, the synthesis position analysis unit 54 creates a background area that is close to each face area. It is specified as a plurality of candidates for the image synthesis position. The composite position analysis unit 54 sets priorities for the plurality of composite position candidates, and selects a composite position according to the priorities when the two-way imaging process is executed. As a method for setting the priority order, for example, the order of face areas having a high degree of coincidence with the facial feature data stored in the face recognition information storage unit 71, the order of increasing free areas, and the like can be employed.

FIG. 7 is a schematic diagram illustrating a state in which the candidate for the synthesis position is specified in the front image.
As illustrated in FIG. 7, the subjects F1 to F6 have a relationship with the subject F7, and thus are detected as faces having a relationship by the face recognition unit 52. In the example illustrated in FIG. 7, three locations (composite positions C <b> 1 to C <b> 3) among the empty areas close to the subjects F <b> 1 to F <b> 6 are specified as candidate composite positions.

When the shutter button is half-pressed, the image composition unit 55 displays a live view image of the rear image at a position that is a candidate for the composition position specified by the composition position analysis unit 54 in the front image at the time of half shutter. Is synthesized. At this time, instead of the live view image of the back image, the back image at the timing when the shutter button is half-pressed may be fixedly combined.
In addition, when an operation for instructing to capture a back image for recording is performed, the image composition unit 55 positions the candidate for the composition position specified by the composition position analysis unit 54 in the front image for recording. The recording back image captured by the second imaging unit 16B is synthesized.
Here, when the image composition unit 55 synthesizes the live view image of the back image or the back image for recording at a position that is a candidate for the composition position in the front image, the size is equal to the size of the face detected in the front image. Then, the size of the face detected in the back image is resized and combined.
Then, the image composition unit 55 stores a composite image of the recording front image and the recording background image in the image storage unit 72.

FIG. 8 is a schematic diagram illustrating a state in which a background image is combined with a front image.
In the example shown in FIG. 8, the case where the back image is synthesized at the synthesis position C1 is shown.
When the live view image of the rear image is combined, the rear image captured in real time is sequentially displayed at the combining position C1.

  The display control unit 56 displays the live view image acquired by the first imaging unit 16A and the second imaging unit 16B on the display of the output unit 18. Further, the display control unit 56 displays the composite image synthesized by the image synthesis unit 55 on the display of the output unit 18. For example, the display control unit 56 outputs a composite image of the front image and the live view image of the rear image at the time of half shutter of the front image, or a composite image of the recording front image and the recording rear image. Display on the display.

Next, the operation will be described.
FIG. 9 is a flowchart for explaining the flow of two-way imaging processing executed by the image processing apparatus 1 of FIG. 2 having the functional configuration of FIG.
The two-way shooting process is started by an operation for starting the two-way shooting process on the input unit 17 by the user.

In step S1, the imaging control unit 51 receives an operation on the input unit 17 by the user.
In step S <b> 2, the imaging control unit 51 determines whether or not the user's operation on the input unit 17 is a half-press of the shutter button.
If the operation of the input unit 17 by the user is not half-pressing the shutter button, NO is determined in step S2, and the process proceeds to step S1.
On the other hand, when the operation on the input unit 17 by the user is half pressing of the shutter button, YES is determined in step S2, and the process proceeds to step S3.

In step S <b> 3, the face recognition unit 52 refers to the facial feature data stored in the face recognition information storage unit 71, and has a relevant face among the faces of the subject included in the front image and the back image. Is detected.
In step S <b> 4, the face recognition unit 52 determines whether or not a related face has been detected among the faces of the subject included in the front image and the back image.
If no relevant face is detected among the faces of the subject included in the front image and the back image, NO is determined in step S4, and the process proceeds to step S9.
On the other hand, if a related face is detected among the faces of the subject included in the front image and the back image, YES is determined in step S4, and the process proceeds to step S5.

In step S5, the free space analysis unit 53 analyzes the arrangement of the subject in the front image, and specifies a region where the main subject is not copied as a free space.
In step S <b> 6, the composite position analysis unit 54 analyzes the front image based on the face area having the relationship recognized by the face recognition unit 52 and the empty area specified by the empty area analysis unit 53. The position where the rear image is synthesized in the front image is specified.
In step S7, the image composition unit 55 resizes the face size detected in the back image to a size equivalent to the face size detected in the front image, and the composite position in the front image at the time of the half shutter. The live view image of the back image is synthesized at a position that is a candidate for the synthesis position specified by the analysis unit 54.

In step S8, the display control unit 56 displays a composite image in which the live view image of the back image is combined with the front image at the time of half shutter.
In step S9, the imaging control unit 51 receives an operation on the input unit 17 by the user.
In step S10, the imaging control unit 51 determines whether or not the user's operation on the input unit 17 is a full press of the shutter button.
If the operation on the input unit 17 by the user is not a full press of the shutter button, NO is determined in step S10, and the process proceeds to step S9.
On the other hand, when the operation of the input unit 17 by the user is a full press of the shutter button, YES is determined in step S10, and the process proceeds to step S11.

In step S11, the imaging control unit 51 causes the first imaging unit 16A to capture a front image for recording.
In step S12, the imaging control unit 51 determines whether or not an operation for instructing imaging of a recording back image has been performed. For example, a full-pressing operation on the shutter button again may be an operation for instructing to capture a back image for recording.
If an operation for instructing to capture a back image for recording is performed, YES is determined in step S12, and the process proceeds to step S15.
On the other hand, when an operation for instructing to capture a back image for recording is not performed, NO is determined in step S12, and the process proceeds to step S13.

In step S13, the image composition unit 55 resizes the size of the face detected in the back image to a size equivalent to the size of the face detected in the front image, and analyzes the composite position in the front image for recording. The live view image of the rear image is synthesized at a position that is a candidate for the synthesis position specified by the unit 54.
In step S14, the display control unit 56 displays a composite image obtained by combining the live view image of the back image with the front image for recording.
After step S14, the process proceeds to step S12.
In step S15, the imaging control unit 51 causes the second imaging unit 16B to capture a recording back image to be combined with the recording front image.

In step S <b> 16, the image composition unit 55 resizes the face size detected in the back image to a size equivalent to the face size detected in the front image, and performs a composite position analysis in the recording front image. The back image for recording is synthesized at a position that is a candidate for the synthesis position specified by the unit 54.
In step S <b> 17, the display control unit 56 displays a composite image in which the recording back image is combined with the recording front image. At this time, if there are a plurality of candidate positions for the synthesis position specified by the synthesis position analysis unit 54 in the recording front image, the display control unit 56 synthesizes the back image at each synthesis position by the image synthesis unit 55. The synthesized images thus displayed are sequentially displayed on the display of the output unit 18. However, the composite image in which the back image is combined at each combining position by the image combining unit 55 may be displayed side by side on the display of the output unit 18.

In step S18, the image composition unit 55 determines whether or not an operation for determining a composite image of the recording front image and the recording back image has been performed.
That is, when the composite image in which the back image is combined at each composite position is sequentially displayed on the display of the output unit 18, the user's view of the composite image currently displayed on the display among the composite images displayed in that order is displayed. It is determined whether or not an operation (operation for determining a composite image) has been performed.
In addition, when a composite image in which a back image is combined at each composite position is displayed side by side on the display of the output unit 18, an operation for selecting a desired composite image (an operation for determining a composite image) is performed by a user operation on the display. ) Is determined.
If the operation for determining the composition position for compositing the back image has not been performed, NO is determined in step S18, and the process proceeds to step S16.
On the other hand, when the operation for determining the composition position for compositing the back image is performed, YES is determined in step S18, and the process proceeds to step S19.

In step S <b> 19, the image composition unit 55 stores the composite image of the determined recording front image and recording back image in the image storage unit 72.
That is, in step S18, composite images obtained by combining the back images at the respective composite positions are sequentially displayed on the display of the output unit 18, and among the composite images displayed in that order, the user for the composite image currently displayed on the display is displayed. When the above operation is performed, the composite image displayed on the display is stored in the image storage unit 72.
In step S18, when a composite image obtained by combining the back image at each composite position is displayed side by side on the display of the output unit 18, and an operation for selecting a desired composite image is performed by an operation on the display of the user. The selected composite image is stored in the image storage unit 72.
After step S19, the two-way shooting process ends.
By such processing, the rear image can be synthesized and displayed at a position having a relationship with the subject in the rear image in the front image.
Therefore, when combining a plurality of images, it is possible to generate a combined image that suits the user's preference.

[Modification 1]
In the above embodiment, the image composition unit 55 fixes the front image at the time of half shutter when the shutter button is half-pressed, and is specified by the composition position analysis unit 54 in the front image at the time of half shutter. The description has been made assuming that the live view image of the back image is synthesized at the position as the candidate for the synthesis position. That is, the example in which the front image is fixed first among the front image and the back image has been described.
On the other hand, it is possible to fix the rear image first, and then photograph the subject of the front image at an arbitrary timing.
Specifically, a function for performing a half-shutter operation on the rear image (an operation for fixing the rear image) is assigned to any one of the buttons on the input unit 17, and the imaging control unit 51 performs the half-shutter operation on the rear image. Correspondingly, the back image at the time of half shutter is acquired. Thereafter, when the photographer instructs imaging of the front image at an arbitrary timing while viewing the state of the subject of the front image in the live view image of the front image, the imaging control unit 51 acquires the front image. At this time, when the live view image of the front image is displayed, the back image at the time of half shutter is displayed by the image composition unit 55 at a position that is a candidate for the composition position of the front image specified by the composition position analysis unit 54. It is good also as combining and displaying.
As a result, it is possible to preferentially take an image of the front image and the back image in which the subject is in a state suitable for photographing, and image the other image at a more appropriate timing.

[Modification 2]
In the above-described embodiment, when the front image and the back image are captured, it is possible to combine the other imaging condition with one imaging condition (shutter speed, white balance, color, image brightness, or the like). In other words, the imaging control unit 51 may adjust the imaging condition of the other image on the basis of one image so that the brightness, color, white balance, etc. of the captured front image and rear image are the same. Is possible.
Thereby, when a back image is combined with a front image, it can control that a picture quality differs mutually and it can be set as a composite image with little sense of incongruity.

[Modification 3]
In the above embodiment, when a related face is not detected in step S4, the process proceeds to step S9. However, the present invention is not limited to this.
That is, if no relevant face is detected in step S4, the front image and the back image may be stored separately without being combined.
In this way, it is possible to prevent unnecessary generation of a composite image in which the front image and the back image are not related to each other.

The image processing apparatus 1 configured as described above includes an imaging control unit 51, a face recognition unit 52, a synthesis position analysis unit 54, and an image synthesis unit 55.
The imaging control unit 51 acquires a first image and a second image.
The face recognition unit 52 determines the relationship between the subject in the first image and the subject in the second image.
The composite position analysis unit 54 determines the composite position of the second image in the first image based on the relevance of the subject determined by the face recognition unit 52.
The image composition unit 55 synthesizes the first image and the second image at the composition position determined by the composition position analysis unit 54.
Thereby, in the first image, the second image can be synthesized at a position having relevance to the subject in the second image.
Therefore, when combining a plurality of images, it is possible to generate a combined image that suits the user's preference.

In addition, the face recognition unit 52 specifies the synthesis position of the second image in the first image at a predetermined timing related to shooting.
Accordingly, it is possible to present the photographer with the composite position of the second image in the first image at a predetermined timing related to photographing (for example, when a half shutter is operated).

The image processing apparatus 1 also includes a face recognition information storage unit 71.
The face recognition information storage unit 71 stores a predetermined relationship between the subject in the first image and the subject in the second image.
The face recognition unit 52 determines whether or not the subject in the first image and the subject in the second image have a predetermined relationship stored in the face recognition information storage unit 71.
When the face recognition unit 52 determines that the subject in the first image and the subject in the second image have the predetermined relationship stored in the face recognition information storage unit 71, the composite position analysis unit 54 Based on the relevance, the synthesis position of the second image in the first image is determined.
Thereby, the composite position of the second image can be determined by reflecting the relationship between the subject of the first image and the subject of the second image.
Therefore, when combining a plurality of images, it is possible to generate a combined image that suits the user's preference.

In addition, the image processing apparatus 1 includes a free space analysis unit 53.
The free space analysis unit 53 identifies a region in the first image where the second image should be synthesized.
The synthesis position analysis unit 54 determines a synthesis position within the area to be synthesized specified by the empty area analysis unit 53.
Thereby, the synthesis position of the second image can be determined within an area suitable for synthesizing the second image.

Further, the image processing apparatus 1 includes an imaging control unit 51, an empty area analysis unit 53, a synthesis position analysis unit 54, and an image synthesis unit 55.
The imaging control unit 51 includes a first image captured in the first direction by the first imaging unit 16A, and a second image captured in the second direction by the second imaging unit 16B different from the first imaging unit 16A. Get the image and.
The vacant area analysis unit 53 specifies an area where the second image in the first image acquired by the imaging control unit 51 is to be synthesized.
The synthesis position analysis unit 54 determines the synthesis position of the second image in the first image within the area specified by the empty area analysis unit 53.
The image composition unit 55 synthesizes the first image and the second image at the composition position determined by the composition position analysis unit 54.
Accordingly, in the first image in which the first direction is captured, the second image is located at a position having relevance to the subject in the second image in which the second direction different from the first direction is captured. Can be synthesized.
Therefore, when combining a plurality of images, it is possible to generate a combined image that suits the user's preference.

Further, the image processing apparatus 1 includes an imaging control unit 51, a display control unit 56, an input unit 17, and an image composition unit 55.
The imaging control unit 51 sequentially acquires a first image captured in the first direction and a second image captured in the second direction simultaneously.
The display control unit 56 sequentially displays the first image and the second image acquired by the imaging control unit 51 on the display.
The input unit 17 inputs a first predetermined instruction during display of the first image and the second image by the display control unit 56.
Further, when the first predetermined instruction is input from the input unit 17, the display control unit 56 fixes the first image and the first image and the second image in a state where the display of one of the first image and the second image is fixed. The other image of the second image is continuously displayed.
The input unit 17 inputs a second predetermined instruction during display of the first image and the second image displayed on the display by the display control unit 56.
The image compositing unit 55 includes a first image corresponding to the time when the first predetermined instruction is input by the input unit 17 and a second image corresponding to the time when the second predetermined instruction is input by the input unit 17. Composite with the image.
This makes it possible to synthesize the first image and the second image at the timing when a predetermined instruction is input to each of the first image and the second image that are sequentially captured at the same time.

Further, the image processing apparatus 1 includes an imaging control unit 51, an image composition unit 55, a display control unit 56, and an input unit 17.
The imaging control unit 51 acquires a first image obtained by imaging the first direction and a second image obtained by imaging the second direction in association with the imaging of the first image.
The image composition unit 55 generates a plurality of candidate images obtained by compositing the second image at any of a plurality of positions in the first image.
The display control unit 56 displays a plurality of candidate images generated by the image composition unit 55 on the display.
The input unit 17 selects a specific candidate image from among a plurality of candidate images displayed on the display by the display control unit 56.
Further, the image composition unit 55 causes the image storage unit 72 to record the specific candidate image selected by the input unit 17.
As a result, it is possible to easily select a composite image that suits the user's preference from among candidate images obtained by combining the second image at any of a plurality of positions in the first image.

Further, the image composition unit 55 adjusts at least one of the imaging conditions of the first image and the second image to match the other.
Thereby, when a 2nd image is synthesize | combined with a 1st image, it can suppress that a mutually different image quality differs greatly, and it can be set as a synthesized image with little discomfort.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

In the above-described embodiment, the case where the subject on the front side and the back side of the image processing apparatus 1 is imaged is described as an example. However, the present invention is not limited to this. That is, the present invention can be applied to capturing images in different directions such as the front side and the side of the image processing apparatus 1.
In the above-described embodiment, the case where the image processing apparatus 1 captures the front image and the back image has been described as an example. However, the present invention is not limited to this. That is, one image such as a front image or a rear image is captured by another device, and the present invention can be applied to these images.

In the above-described embodiment, the image processing apparatus 1 to which the present invention is applied has been described using a digital camera as an example, but is not particularly limited thereto.
For example, the present invention can be applied to general electronic devices having an image processing function. Specifically, for example, the present invention can be applied to a notebook personal computer, a video camera, a portable navigation device, a mobile phone, a smartphone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 3 is merely an example, and is not particularly limited. That is, it is sufficient that the image processing apparatus 1 has a function capable of executing the above-described series of processes as a whole, and what functional blocks are used to realize this function is not particularly limited to the example of FIG. .
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 2 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disc is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disc), a Blu-ray (registered trademark) Disc (Blu-ray Disc), and the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preliminarily incorporated in the apparatus main body includes, for example, the ROM 12 in FIG. 2 in which the program is recorded, the hard disk included in the storage unit 19 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalents thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
Image acquisition means for acquiring a first image and a second image;
Relevance determining means for determining relevance between the subject in the first image and the subject in the second image;
Determining means for determining a composite position of the second image in the first image based on the relevance of the subject determined by the relevancy determining means;
Image combining means for combining the first image and the second image at the combining position determined by the determining means;
An image processing apparatus comprising:
[Appendix 2]
The image processing apparatus according to appendix 1, wherein the determination unit specifies a synthesis position of the second image in the first image at a predetermined timing related to photographing.
[Appendix 3]
Storage means for storing a predetermined relationship between the subject in the first image and the subject in the second image;
The relevance determining means determines whether or not the subject in the first image and the subject in the second image have a predetermined relevance stored in the storage means,
The determining means determines that the relevancy determining means determines that the subject in the first image and the subject in the second image have a predetermined relevance stored in the storage means. 3. The image processing apparatus according to appendix 1 or 2, wherein a composite position of the second image in the first image is determined based on
[Appendix 4]
Area specifying means for specifying an area in which the second image in the first image is to be synthesized;
The image processing apparatus according to any one of appendices 1 to 3, wherein the determining unit determines the combining position within the region to be combined specified by the region specifying unit.
[Appendix 5]
A first image captured in the first direction by the first image capturing unit and a second image captured in the second direction by the second image capturing unit different from the first image capturing unit are acquired. Image acquisition means;
Area specifying means for specifying an area to be combined with the second image in the first image acquired by the image acquiring means;
Determining means for determining a synthesis position of the second image in the first image within the area specified by the area specifying means;
Image combining means for combining the first image and the second image at the combining position determined by the determining means;
An image processing apparatus comprising:
[Appendix 6]
Image acquisition means for sequentially acquiring a first image captured in the first direction and a second image captured in the second direction simultaneously;
First display control means for sequentially displaying the first image and the second image acquired by the image acquisition means on a display means;
First input means for inputting a first predetermined instruction during display of the first image and the second image by the first display control means;
When the first predetermined instruction is input by the first input means, the display of the first image and the second image is fixed while the display of one of the first image and the second image is fixed. Second display control means for continuously displaying the other of the two images;
Second input means for inputting a second predetermined instruction during display of the first image and the second image displayed on the display means by the second display control means;
The first image corresponding to the time when the first predetermined instruction is input by the first input means and the second image corresponding to the time when the second predetermined instruction is input by the second input means. A synthesis means for synthesizing the image;
An image processing apparatus comprising:
[Appendix 7]
An image acquisition means for acquiring a first image obtained by imaging the first direction and a second image obtained by imaging the second direction in association with imaging of the first image;
Generating means for generating a plurality of candidate images obtained by combining the second image with any one of a plurality of positions in the first image;
Display control means for displaying on the display means a plurality of candidate images generated by the generating means;
Selecting means for selecting a specific candidate image from among the plurality of candidate images displayed on the display means by the display control means;
Recording control means for causing the recording means to record the specific candidate image selected by the selection means;
An image processing apparatus comprising:
[Appendix 8]
The image processing according to any one of appendices 1 to 7, further comprising an adjustment unit that adjusts at least one of the imaging conditions of the first image and the second image to match the other. apparatus.
[Appendix 9]
An image acquisition step of acquiring a first image and a second image;
A relevance determining step for determining relevance between the subject in the first image and the subject in the second image;
A determination step of determining a composite position of the second image in the first image based on the relationship of the subject determined in the relationship determination step;
An image combining step of combining the first image and the second image at the combining position determined in the determining step;
An image processing method comprising:
[Appendix 10]
On the computer,
An image acquisition function for acquiring a first image and a second image;
A relevance determination function for determining relevance between the subject in the first image and the subject in the second image;
A determination function for determining a composite position of the second image in the first image based on the relationship of the subject determined by the relationship determination function;
An image composition function for compositing the first image and the second image at the composition position determined by the determination function;
A program characterized by realizing.
[Appendix 11]
An image processing method used in an image processing apparatus,
A first image captured in the first direction by the first image capturing unit and a second image captured in the second direction by the second image capturing unit different from the first image capturing unit are acquired. An image acquisition step;
An area specifying step for specifying an area to be combined with the second image in the first image acquired in the image acquiring step;
A determining step of determining a composite position of the second image in the first image within the region specified in the region specifying step;
An image combining step of combining the first image and the second image at the combining position determined in the determining step;
An image processing method comprising:
[Appendix 12]
On the computer,
A first image captured in the first direction by the first image capturing unit and a second image captured in the second direction by the second image capturing unit different from the first image capturing unit are acquired. Image acquisition function,
An area specifying function for specifying an area to be combined with the second image in the first image acquired by the image acquisition function;
A determination function for determining a composite position of the second image in the first image within the region specified by the region specification function;
An image synthesis function for synthesizing the first image and the second image at a synthesis position determined by the specific function;
A program characterized by realizing.
[Appendix 13]
An image processing method used in an image processing apparatus,
An image acquisition step of sequentially acquiring a first image captured in the first direction and a second image captured in the second direction simultaneously;
A first display control step for sequentially displaying on the display means the first image and the second image acquired in the image acquisition step;
A first input step of inputting a first predetermined instruction during display of the first image and the second image in the first display control step;
When the first predetermined instruction is input in the first input step, the display of the first image and the second image is fixed while the display of one of the first image and the second image is fixed. A second display control step of continuously displaying the other of the two images;
A second input step of inputting a second predetermined instruction during the display of the first image and the second image displayed on the display means in the second display control step;
The first image corresponding to the time point when the first predetermined instruction is input in the first input step and the second image corresponding to the time point when the second predetermined instruction is input in the second input step. A compositing step for compositing the image;
An image processing method comprising:
[Appendix 14]
On the computer,
An image acquisition function for sequentially acquiring a first image captured in the first direction and a second image captured in the second direction simultaneously;
A first display control function for sequentially displaying the first image and the second image acquired by the image acquisition function on a display unit;
A first input function for inputting a first predetermined instruction during display of the first image and the second image by the first display control function;
When the first predetermined instruction is input by the first input function, the display of the first image and the second image is fixed while the display of one of the first image and the second image is fixed. A second display control function for continuously displaying the other image of the two images;
A second input function for inputting a second predetermined instruction during display of the first image and the second image displayed on the display means by the second display control function;
The first image corresponding to the time when the first predetermined instruction is input by the first input function and the second image corresponding to the time when the second predetermined instruction is input by the second input function A compositing function that combines images,
A program characterized by realizing.
[Appendix 15]
An image processing method used in an image processing apparatus,
An image acquisition step of acquiring a first image obtained by imaging the first direction and a second image obtained by imaging the second direction in association with imaging of the first image;
Generating a plurality of candidate images obtained by synthesizing the second image in any of a plurality of positions in the first image;
A display control step for displaying a plurality of candidate images generated in the generation step on a display means;
A selection step of selecting a specific candidate image from among the plurality of candidate images displayed on the display means in the display control step;
A recording control step for causing the recording means to record the specific candidate image selected in the selection step;
An image processing method comprising:
[Appendix 16]
On the computer,
An image acquisition function for acquiring a first image obtained by imaging the first direction and a second image obtained by imaging the second direction in association with the imaging of the first image;
A generating function for generating a plurality of candidate images obtained by combining the second image with any one of a plurality of positions in the first image;
A display control function for causing the display means to display a plurality of candidate images generated by the generation function;
A selection function for selecting a specific candidate image from among a plurality of candidate images displayed on the display means by the display control function;
A recording control function for causing the recording means to record the specific candidate image selected by the selection function;
A program characterized by realizing.

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Bus, 15 ... Input / output interface, 16A ... 1st imaging part, 16B 2nd imaging unit, 17 ... input unit, 18 ... output unit, 19 ... storage unit, 20 ... communication unit, 21 ... drive, 31 ... removable media, 51 ... Imaging control unit, 52 ... Face recognition unit, 53 ... Empty area analysis unit, 54 ... Composition position analysis unit, 55 ... Image composition unit, 56 ... Display control unit, 71 ... Face recognition information storage unit, 72 ... Image storage unit

Claims (4)

Image acquisition means for sequentially acquiring a first image captured in the first direction and a second image captured in the second direction simultaneously;
First display control means for simultaneously displaying the first image and the second image acquired by the image acquisition means on the display means simultaneously;
First input means for inputting a first predetermined instruction during display of the first image and the second image by the first display control means;
When the first predetermined instruction is input by the first input means, the display of the first image and the second image is fixed while the display of one of the first image and the second image is fixed. Second display control means for continuously displaying the other of the two images;
Second input means for inputting a second predetermined instruction during display of the first image and the second image displayed on the display means by the second display control means;
The first image corresponding to the time when the first predetermined instruction is input by the first input means and the second image corresponding to the time when the second predetermined instruction is input by the second input means. A synthesis means for synthesizing the image;
An image processing apparatus comprising: The image processing apparatus according to claim 1, further comprising an adjusting unit configured to adjust at least one of the imaging conditions of the first image and the second image to match the other. An image processing method used in an image processing apparatus,
An image acquisition step of sequentially acquiring a first image captured in the first direction and a second image captured in the second direction simultaneously;
A first display control step of simultaneously displaying the first image and the second image acquired in the image acquisition step simultaneously on a display unit;
A first input step of inputting a first predetermined instruction during display of the first image and the second image in the first display control step;
When the first predetermined instruction is input in the first input step, the display of the first image and the second image is fixed while the display of one of the first image and the second image is fixed. A second display control step of continuously displaying the other of the two images;
A second input step of inputting a second predetermined instruction during the display of the first image and the second image displayed on the display means in the second display control step;
A first image corresponding to the time point when the first predetermined instruction is input in the first input step and a second image corresponding to the time point when the second predetermined instruction is input in the second input step. A compositing step for compositing the image;
An image processing method comprising: On the computer,
An image acquisition function for sequentially acquiring a first image captured in the first direction and a second image captured in the second direction simultaneously;
A first display control function for simultaneously displaying the first image and the second image acquired by the image acquisition function simultaneously on a display unit;
A first input function for inputting a first predetermined instruction during display of the first image and the second image by the first display control function;
When the first predetermined instruction is input by the first input function, the display of the first image and the second image is fixed while the display of one of the first image and the second image is fixed. A second display control function for continuously displaying the other image of the two images;
A second input function for inputting a second predetermined instruction during display of the first image and the second image displayed on the display means by the second display control function;
Second corresponding to the time when the second predetermined instruction is input by the first image and the previous SL second input function corresponding to the time when the first predetermined instruction is input by said first input function A compositing function for compositing images with
A program characterized by realizing.

Images