JP2021170810A - Photographing device - Google Patents

Abstract

To provide a photographing device that easily takes a picture in which all of a plurality of persons appear.SOLUTION: In a photographing device, a face recognition processing part executes face recognition processing to first image data to recognize a first person. A synthesis processing part, when second image data, which is photographed at an arbitrary photographing timing different from a photographing timing of the first image data and which has a background same as a background of the first image data and has a second person different from the first person photographed, is recorded in a storage part, creates synthetic data in which the first person and the second person are superimposed on the same background using the first image data and the second image data.SELECTED DRAWING: Figure 1B

Description

The present invention relates to an image pickup apparatus and an image processing method, and particularly relates to a technique for synthesizing image data captured by a plurality of people.

As a background technology in the present technical field, Patent Document 1 has an object of "making it possible to easily take a group photograph by a plurality of people including a photographer", and as a solution to the problem, "a composite photography mode is provided in a digital camera. Immediately after being instructed, one photographer takes a picture a of another photographed person (plurality). Then, it is handed over to a digital camera by one of the other photographed persons, and one of the previous photographers The captured image b is captured. Then, there are a plurality of faces of a predetermined size or larger in a predetermined composite condition, for example, one captured image, and one face of a predetermined size or larger in the other captured image. In some cases, or when the person composition of the face detected in each of the two captured images a and b is different, the face image included in either of the two captured images a and b captured in succession is displayed. On the other hand, a new composite image c combined is created. By doing so, a group photograph taken by a plurality of people can be easily combined. "

U.S. Patent Application Publication No. 2012/120273 (excerpt)

In Patent Document 1, when the captured images a and b are continuously photographed (hereinafter referred to as "series of imaging") immediately after the composite imaging mode is instructed, the composite image c is created based on these images. However,
For example, if another photo opportunity comes during a series of shots, you may miss a photo opportunity because you cannot take another image until the series of shots is complete, and if you don't like one photo, Since it is necessary to take a picture again from the beginning, there is room for further improvement in operability when generating a composite photograph.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image pickup apparatus and an image processing method having improved operability when easily generating a photograph including all of a plurality of people. ..

As a means for solving the above-mentioned problems, the technique described in the claims is used. For example, in an imaging device, an imaging processing unit that captures a subject and generates image data, a face recognition processing unit that executes face recognition processing of the image data, and an image recording unit that records the image data. The face recognition processing unit includes a composition processing unit that executes a composition process so that a person captured by each of the plurality of image data is included in one image data to generate the composition data. , The face recognition process is executed on the first image data to recognize the face of the first person, and the first image data is shot at an arbitrary shooting timing different from the shooting timing of the first image data. When the second image data which has the same background as the background of the first image data and is captured by a second person different from the first person is recorded in the image recording unit. The synthesis processing unit uses the first image data and the second image data to generate the composite data in which the first person and the second person are superimposed on the same background.

According to the present invention, it is possible to provide an imaging device and an image processing method with improved operability when easily generating a photograph including a plurality of people. Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

Block diagram showing a configuration example of the image pickup apparatus according to the first embodiment The figure which shows the software block diagram of the image pickup apparatus which concerns on 1st Embodiment Front view of the image pickup apparatus according to the first embodiment Rear view of the image pickup apparatus according to the first embodiment A flowchart showing an overall operation example of the image pickup apparatus according to the first embodiment. A flowchart showing another operation example (automatic composition mode) of the image pickup apparatus according to the first embodiment. A flowchart showing an operation example of the synthesis processing unit according to the first embodiment. The figure which shows the display example of the display part immediately after shooting by the image pickup apparatus which concerns on 1st Embodiment. The figure which shows the display example of the image data synthesized by the image pickup apparatus which concerns on 1st Embodiment The figure which shows the display example of the selection screen whether or not to record the image data synthesized by the image pickup apparatus which concerns on 1st Embodiment. The figure which shows the display example when the image could not be combined by the image pickup apparatus which concerns on 1st Embodiment The figure which shows the display example of the selection screen whether or not to display the image data combined with another image by the image pickup apparatus which concerns on 1st Embodiment. Explanatory drawing which shows an example of search and synthesis of image data which concerns on 1st Embodiment Explanatory drawing which shows an example of search and synthesis of image data which concerns on 1st Embodiment A flowchart showing an overall operation example of the image pickup apparatus according to the second embodiment. A flowchart showing an operation example of the synthesis processing unit according to the second embodiment. The figure which shows the display example of the display part when the play button is pressed in the image pickup apparatus which concerns on 2nd Embodiment. The figure which shows the display example of the display part at the time of changing to another image data in the image pickup apparatus which concerns on 2nd Embodiment. The figure which shows the display example of the display part when the composition button is pressed in the image pickup apparatus which concerns on 2nd Embodiment. The figure which shows the display example of the display part at the time of changing to another image data in the image pickup apparatus which concerns on 2nd Embodiment. Explanatory drawing which shows an example of search and synthesis of image data which concerns on 2nd Embodiment A flowchart showing an overall operation example of the image pickup apparatus according to the third embodiment. A flowchart showing an operation example of the synthesis processing unit according to the third embodiment. The figure which shows the display example of the display part immediately after shooting in the image pickup apparatus which concerns on 3rd Embodiment, or when the play button is pressed. The figure which shows the display example of the image data synthesized by the image pickup apparatus which concerns on 3rd Embodiment Explanatory drawing which shows an example of search and synthesis of image data which concerns on 3rd Embodiment A block diagram showing a configuration example of an imaging device according to a fourth embodiment. The figure which shows the software block diagram of the image pickup apparatus which concerns on 4th Embodiment A flowchart showing an operation example of the synthesis processing unit according to the fourth embodiment. Explanatory drawing which shows an example of search and synthesis of image data which concerns on 4th Embodiment

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and duplicate description thereof will be omitted. In the following embodiments, the image pickup apparatus according to the present invention will be described as an example.

<First Embodiment>
In the first embodiment, the image data being displayed and the image data selected by the imaging device are combined in a manual composition mode (operation mode in which the image data is combined after the user gives a composition instruction) or an automatic composition mode (operation in which the image pickup device automatically synthesizes the image data). It is an embodiment which synthesizes by mode).

[Hardware configuration of imaging device]
FIG. 1A is a block diagram showing a configuration example of an imaging device according to the first embodiment of the present invention.

In FIG. 1A, the main control unit 101 is composed of a CPU (Central Processing Unit) or the like, and has a ROM (Read Only Memory) 102 or a RAM (Random Access Memory) 103.
The entire image pickup apparatus 1 is controlled according to various operation programs and data stored in the image pickup apparatus 1.
The system bus 100 is a data communication path for transmitting and receiving data between the control unit 101 and each unit in the imaging device 1.

The ROM 102 is a memory in which various programs for controlling the image pickup apparatus 1 are stored, for example, an EEPROM (Electrically Erasable Programmable ROM) or a flash RO.
A rewritable ROM such as M is used.

The RAM 103 is used as a temporary storage area for executing a program stored in the ROM 102 or as a temporary storage for captured images.

The storage unit 104 (corresponding to the image recording unit) stores information such as operation setting values of the image pickup apparatus 1, and is a non-volatile and rewritable device such as a flash ROM or SSD (Solid State Drive). Is used.

The ROM 102 and the RAM 103 may be integrated with the main control unit 101. again,
The ROM 102 does not have an independent configuration as shown in FIG. 1A, and may use a partial storage area in the storage unit 104.

The external storage medium interface (I / F) 105 is an interface for recording / reading information on an external recording medium 106 that can be stored and taken out in an image pickup device 1 such as an SD card.

The external interface 107 is a group of interfaces for expanding the functions of the image pickup apparatus 1, and in the present embodiment, the USB (Universal Serial Bus) interface (I / F).
) 107a, a video output interface (I / F) 107b, and an audio output interface (I / F) 107c. The USB interface 107a is connected to the USB interface of an external device such as a personal computer or a television receiving device, and the image data or the like stored in the storage unit 104 or the external recording medium 106 can be read and displayed by the external device. The video output interface 107b and the audio output interface 107c output a video signal / audio signal to an external video / audio output device. Here, the video output interface 107b and the audio output interface 107c are HDMI (High-Definition).
The video and audio may be output together using the Multimedia Interface (registered trademark).

The voice input unit 108 includes a microphone that converts the voice around the image pickup device 1 into an electric signal, an A / D converter that converts the voice converted into the electric signal into voice data of a digital signal, and the like.

The audio signal processing unit 109 performs filtering processing on the audio data input from the audio input unit, processing for converting the audio data according to the format of the moving image recorded on the external recording medium 106, and the like.

The imaging unit 110 includes an optical system including a lens for zooming and focusing operations, a mechanism system for driving the lens for zooming and focusing operations, and a CCD or a CCD that converts an optical image of a subject input from the lens into an electric signal. It is composed of an image sensor such as CMOS and an electrical device such as an A / D converter that converts an image converted into an electric signal into image data of a digital signal.

The image signal processing unit 111 performs image processing such as white balance adjustment, exposure adjustment, and gamma correction on the image data input from the image pickup unit 110. Further, according to the format of recording on the external recording medium 106, for example, in the case of a moving image, MPEG2 or H.M. It is converted into image data such as 264, and in the case of a still image, it is converted into image data such as JPEG and TIFF. In addition, image data storage processing, face recognition processing, pattern matching processing, and the like performed in the storage unit 104 or the external recording medium 106 are performed.

The display unit 112 displays an image of the subject captured by the image sensor of the image pickup unit 110, an image recorded on the storage unit 104 or the external recording medium 106, a screen for making various settings of the image pickup device 1, and the like. ..

The operation unit 113 is an instruction input unit for inputting an operation instruction to the image pickup apparatus 1, and has a power on / off button, a shutter button, a playback button, a composition button, a button for performing various settings, and the like. Further, a touch panel may be provided on the surface of the display unit 112, and an operation instruction may be input to the image pickup apparatus 1 by detecting the position of a button or the like displayed on the display unit 112 and the position where the touch panel is touched.

The timekeeping unit 114 sets the elapsed time from the date and time set by the user, for example, to RTC (Real T).
ime Clock) Measures using a circuit and outputs date / time / time information.
Based on the date / time / time information acquired from 14, the date / time / time information is added to the image data for recording.

The GPS receiving unit 115 receives radio waves from a plurality of GPS satellites, and can acquire position (latitude, longitude, etc.) information of the imaging device 1 based on the received signals. This position information is also recorded in addition to the image data.

The power supply unit 116 includes a battery (not shown), and is directed to each part constituting the image pickup apparatus 1 according to a power-on state, a power-off state, a power-off standby state, and the like according to an instruction from the main control unit 101. Supply power.

The image pickup device 1 may be a digital camera, a mobile phone, a smartphone, a tablet terminal, a navigation device, or the like as long as it includes an image pickup unit 110, or a PDA (Per).
Sonal Digital Assistants) and notebook PCs (Persona)
l Computer) may be used. It may be a music player having a communication function, a portable game machine, or other portable digital device. Further, a wearable smart watch, smart glasses, or the like may be used.

[Software configuration of imaging device]
FIG. 1B is a diagram showing a software configuration diagram of the image pickup apparatus according to the first embodiment, and is a ROM.
The software configuration in 102, the RAM 103, and the storage unit 104 is shown.

In FIG. 1B, the main control unit 101 executes a program stored in the ROM 102 to execute the basic operation execution unit 1021, the position information acquisition unit 1022, the image pickup processing unit 1023, the image recording unit 1024, the image reproduction unit 1025, and the power supply management. A unit 1026, a synthesis processing unit 1027, and a face recognition processing unit 1028 are configured. Further, the RAM 103 temporarily holds a temporary storage area 1031 that temporarily holds data as needed when executing a program stored in the ROM 102, and temporarily captures an image taken by the imaging processing unit 1023 via the imaging unit 110. It is assumed that the image temporary storage area 1032 to be stored is provided.

The storage unit 104 includes a setting information storage area 1041 for storing setting information when executing a program stored in the ROM 102, an image storage area 1042 for storing still image data and moving image data captured by the imaging device 1, and the like. Also has.

In the following, for the sake of simplicity, it is assumed that the main control unit 101 controls each operation block by executing the program stored in the ROM 102.

The basic operation execution unit 1021 controls various settings of the image pickup apparatus 1 and overall operations.

The position information acquisition unit 1022 performs a process of acquiring position information such as latitude and longitude of the image pickup apparatus 1 based on the signal from the GPS satellite received by the GPS reception unit 115. Also, Wi-Fi (
It has a reception function (registered trademark), and it is possible to improve the accuracy by correcting the position information using the information of the radio wave reception level of Wi-Fi, or the position information using the information of the radio wave reception level of Wi-Fi. May be obtained.

The image pickup processing unit 1023 receives when the shutter button included in the operation unit 113 is pressed.
The image data of the subject captured by the image pickup unit 110 is stored in the image temporary storage area 1032 of the RAM 103.
Performs the process of importing into.

The image recording unit 1024 performs a process of recording the image data captured in the image temporary storage area 1032 in the image storage area 1042 of the storage unit 104, or recording the image data in the external recording medium 106 via the external recording medium interface 105.

The image reproduction unit 1025 RAs the image of the subject captured by the image sensor of the image pickup unit 110.
A process of displaying on the display unit 112 is performed via the image temporary storage area 1032 of the M103. Further, the image data recorded in the storage unit 104 or the external recording medium 106 is read out and displayed on the display unit 112.

The power management unit 1026 performs a process of controlling the power supply to each part constituting the image pickup apparatus 1 by the power button included in the operation unit 113.

The compositing processing unit 1027 may be the storage unit 104 or the external recording medium 106 when the compositing button included in the operation unit 113 is pressed or when the display of the compositing image data is selected.
Performs a process of synthesizing the image data stored in.

The face recognition processing unit 1028 performs face recognition processing of image data stored in the RAM 103, the storage unit 104, and the external recording medium 106.

The program stored in the ROM 102 may be stored in advance at the time of product shipment. Further, after the product is shipped, a program acquired from a server device or the like on the Internet by a personal computer may be stored via the USB interface unit 107a.

Instead of providing the image temporary storage area 1032 in the RAM 103, the image pickup unit 110 or the image signal processing unit 111 may be provided with a buffer memory for temporarily storing image data.

[Outline view of imaging device]
The appearance of the image pickup apparatus according to the first embodiment will be described with reference to FIGS. 2A and 2B.

FIG. 2A is a front view of the image pickup apparatus 1. A lens constituting the image pickup unit 110 is arranged in front of the image pickup apparatus 1. Further, a power button 113-1 and a shutter button 113-2 constituting the operation unit 113 are arranged on the upper part of the image pickup apparatus 1.

FIG. 2B is a rear view of the image pickup apparatus 1. A display unit 112 and an operation unit 1 are on the back surface of the image pickup apparatus 1.
Selection / decision operation button 113-3 constituting 13, menu button 113-4 for selecting various settings / file operation processes, storage unit 104 or external recording medium 10
A reproduction button 113-5 for reproducing the image data recorded in 6 and a composition button 113-6 (corresponding to a composition instruction operation unit) for synthesizing the image data are arranged. The display unit 112 is provided with a touch sensor, and the shutter button 113-2 and the selection / decision operation button 11
It may be possible to operate instead of 3-3, the menu button 113-4, the play button 113-5, and the composite button 113-6.

[Overall operation of the imaging device]
FIG. 3 is a flowchart showing an overall operation example of the image pickup apparatus according to the first embodiment. Figure 3
Then, the manual mode, that is, the user inputs an instruction for image composition, and more specifically, the image composition process (S313) is executed triggered by "pressing down the image composition button (S312; Yes)" in the following example. The flow of operation in the case of

First, the power management unit 1026 confirms that the power button 113-1 has been pressed. If it is not pressed down (S301; No), it waits in this step without transitioning to the next step.

When the power button 113-1 is pressed (S301; Yes), the imaging device 1 is activated (S302). When activated, the image of the subject captured by the image sensor of the imaging unit 110 is displayed on the display unit 112.

The power management unit 1026 confirms that the power button 113-1 has been pressed. Power button 11
When 3-1 is pressed (S303; Yes), the termination process of the image pickup apparatus 1 is performed (S304).
).

When the power button 113-1 is not pressed (S303; No), the image processing unit 102
3 confirms that the shutter button 113-2 is pressed. Shutter button 113-
When 2 is pressed (S305; Yes), the image pickup unit 110 captures and acquires the image data of the subject (S306), and stores the image data in the image temporary storage area 1032 of the RAM 103. In addition, additional information such as the captured model, aperture value, number of pixels, ISO sensitivity, shooting date and time, and position information is also stored in the image temporary storage area 1032. Then, the image data and additional information stored in the image temporary storage area 1032 are recorded by the image recording unit 1024 in the storage unit 104 or the external recording medium 106 (S307). Then, the image pickup unit 110 takes an image, and the image temporary storage area 10
The image data stored in 32 is displayed on the display unit 112 (S309).

When the shutter button 113-2 is not pressed (S305; No), the image reproduction unit 1025 confirms that the reproduction button 113-5 is pressed. When the play button 113-5 is pressed (S308; Yes), the image data recorded in the storage unit 104 or the external recording medium 106 is reproduced by the image reproduction unit 1025 and displayed on the display unit 112 (S309).
). The image to be reproduced / displayed can be displayed on the display unit 112 by pressing the reproduction button 113-5. The image data displayed on the display unit 112 and the image data (S306) photographed and generated by the user and displayed on the display unit 112 correspond to the first image data.

FIG. 6A is a diagram showing a display example of the display unit 112 immediately after being photographed by the image pickup apparatus 1. Person A is photographed against the background of the mountain. The display time of the captured image data is, for example, about 3 seconds, and may be changed by the user. The display of the image data disappears after a lapse of a predetermined time, but when the play button 113-5 is pressed, the display is displayed again.

When the play button 113-5 is not pressed (S308; No), the synthesis processing unit 102
Confirm that the composite button 113-6 is pressed by step 7. When the compositing button 113-6 is pressed (S312; Yes), the imaging unit 110 takes an image, and the image data recorded in the storage unit 104 or the external recording medium 106 via the image temporary storage area 1032 is combined with the compositing processing unit. 1027
(S313), and the combined image data is displayed on the display unit 112 (S31).
5).

FIG. 6B is a diagram showing a display example of image data synthesized by the image pickup apparatus 1. A person B taken against the background of the same mountain is combined with an image including a person A taken against the background of a mountain.

When the selection / determination for recording the image data synthesized by the selection / decision operation button 113-3 in the storage unit 104 or the external recording medium 106 is performed (S317: Yes), the image recording unit 1024 is combined by the composition processing unit 1027. The image data is recorded in the storage unit 104 or the external recording medium 106 (S318).

FIG. 7A is a diagram showing a display example of a selection screen for whether or not to record the image data synthesized by the image pickup apparatus 1. The user selects “yes” or “no” with the selection / decision operation button 113-3 and decides.

The image data synthesized by the selection / decision operation button 113-3 is not recorded in the storage unit 104 or the external recording medium 106. When a selection / decision is made (S317: No), or when the composite button is not pressed (S312). No), after the image reproduction process (S315)
Alternatively, after the image reproduction process (S309), the process returns to step S303.

FIG. 7B is a diagram showing a display example when the images cannot be combined. When the images cannot be combined (when the images cannot be combined as a result of the image composition processing by S313), the screen shown in FIG. 7B is displayed on the display unit 112 instead of FIG. 7A.

By performing the above processes S301 to S318, images are recorded, reproduced, and combined.

It should be noted that the composition process may be selected and executed by the menu button 113-4 or the selection / decision operation button 113-3 instead of the independent composition button 113-6. Further, if an important person is registered in advance and this person is photographed, the compositing operation may be automatically performed. Further, when the photograph is taken by touching the subject displayed on the display unit 112 provided with the touch sensor instead of pressing the shutter button 113-2, if the subject is a person, the person may be registered as an important person. .. Further, the compositing operation may be automatically performed when the photographed person deviates from the center of the image by a predetermined value or more or by a predetermined ratio or more, for example, when the person deviates from the center by 20% or more.

Next, the image composition process in the so-called automatic composition mode, in which the composition process is automatically performed when a person is photographed, will be described with reference to FIG. FIG. 4 is a flowchart showing another operation example (automatic composition mode) of the image pickup apparatus 1 according to the first embodiment.

Since S401 to S409 in FIG. 4 are the same as S301 to S309 in FIG. 3, the description thereof will be omitted.

The synthesis processing unit 1027 confirms whether or not it is in the automatic synthesis mode (S412). The automatic composition mode is set in advance with the menu button 113-4 and the selection / decision operation button 113-3. Therefore, the menu button 113-4 and the selection / decision operation button 113-3 correspond to the mode setting unit.

In the case of the automatic composition mode (S412; Yes), the composition processing unit 1027 takes an image with the image pickup unit 110 and synthesizes the image data recorded in the storage unit 104 or the external recording medium 106 via the image temporary storage area 1032 (S413). ). The composition processing unit 1027 is the image temporary storage area 1
The synthesis process is performed at 032.

The display unit 112 displays a screen for selecting whether or not to display the combined image data. The user selects / determines whether or not to display the combined image data by pressing the selection / determination operation button 113-3 (S414; Yes).

FIG. 8 is a diagram showing a display example of a selection screen for whether or not to display image data combined with another image by the image pickup apparatus 1. The user selects “yes” or “no” with the selection / decision operation button 113-3 and decides. If it cannot be combined with another image, it will not be displayed.

S415-418: Since it is the same as S315-S318 in FIG. 3, the description thereof will be omitted.

If the automatic composition mode is not set (S412; No), or if the combined image data is not displayed (S414; No), the process returns to S403.

By performing the above processes S401 to S418, images are recorded, reproduced, and combined in the automatic composition mode.

[Image composition processing]
Next, the image composition processing of S313 and S411 will be described.

FIG. 5 is a flowchart showing an operation example of the synthesis processing unit 1027.

The compositing processing unit 1027 searches for the latest image on the same shooting date as the image displayed on the display unit 112 from the image data stored in the image storage area 1042 of the storage unit 104 or 106 other than the external recording field. (S501). Hereinafter, the searched image is referred to as a “search image”, and the image displayed on the display unit 112 is referred to as a “display image”.

The compositing processing unit 1027 confirms whether or not the shooting location of the search image is the same as the shooting location of the display image based on the position information acquired by the position information acquisition unit 1022 (S502).

When the shooting locations are the same (S502; Yes), it is confirmed whether or not the shooting date and time of the search image and the shooting date and time of the displayed image are within a predetermined time (S503). The predetermined time is, for example, about 3 minutes and may be changed by the user.

When the shooting date and time is within the predetermined time (S503; Yes), it is confirmed by pattern matching whether or not the background of the search image is the same as the background of the display image (S504).

When the background is the same (S504; Yes), the face recognition processing unit 1028 confirms whether or not the same person as the person included in the display image is included in the search image (S505).

When the same person is not included (S505; No), when the search image and the display image are superimposed, the position of the person captured in the display image in the display image and the position of the person captured in the display image are extracted from the search image. Check whether or not the position of the person to be combined with the display image overlaps (S5).
06). The judgment level may be set, such as Yes when the people overlap even a little, or Yes only when the faces overlap.

The same person as the shooting location of the search image and the display image (S502; Yes), the shooting date and time is within the predetermined time (S503; Yes), the background is the same (S504; Yes), and the same person is not included (S505; No). ), And when the positions of the people do not overlap (S506; No), the person in the search image is extracted by contour detection, pasted on the display image, and combined (S507). Alternatively, a person in the display image is extracted by contour detection, pasted on the search image, and combined.

When the shooting location of the search image and the display image are not the same (S502; No), or when the shooting date and time are not within the predetermined time (S503; No), or when the background is not the same (S503; No).
In S504; No), when the same person is included (S505; Yes), or when the people overlap (S506; Yes), it is confirmed whether or not there is another image on the same shooting date (S508).

If there is another image (S508; Yes), then the new image is searched (S509), the process returns to S502, and the process is continued. If there is no other image (S508; No), the process ends. The order of S502 and S503 may be changed, and these steps correspond to the process of searching for the second image data candidate.

By performing the above processes S501 to S509, an image to be pasted on the display image is searched, a person is extracted, and a person is synthesized.

Further, a geomagnetic sensor may be mounted on the image pickup device, and whether or not the orientation of the image pickup device is the same may be added to the above determination.

Further, in the case of an imaging device not equipped with GPS, the synthesis process may be executed by omitting S502.

FIG. 9 is an explanatory diagram showing an example of searching and synthesizing image data according to the first embodiment.

The image data 901 to 905 are image data taken in order from 10:02: 00 to 10:02:55 on 2015.03.15 at latitude 35 ° 24'53 "north and longitude 138 ° 51'31" east. be. The image data 901, 902, 904, and 905 have the same background, and only the image data 903 has a different background.

The date and time information is acquired by the time measuring unit 114, and the position information is acquired by the GSP receiving unit 115.

The image data 901 and 902 are image data in which the person B is photographed by the person A, and the image data 903 is image data in which the bird is photographed by the person A or the person B. The image data 904 and 905 are image data in which the person A is photographed by the person B.

Generally, the user repeats shooting until satisfactory image data is obtained. Image data 9
Both 01 and 902 are image data in which the person B is photographed by the person A, but since the person B in the image data 901 does not have a smile, it indicates that the image data 902 has been photographed again by the person A. Similarly, both the image data 904 and 905 are image data in which the person A is photographed by the person B, but since the person A in the image data 904 does not have a smile, the image data 905 is re-photographed by the person B. Show that. In addition, the image data 903 is a series of shots (
Indicates that a bird that suddenly appeared during (meaning the period of shooting of image data 901 to 905) was shot.

When the shutter button 113-2 of the image pickup device 1 is pressed, the subject is photographed and the display unit 112
Display the captured image in. Then, when the composite button 113-6 is pressed, the images are composited.

The image data 906 is image data synthesized from the image data 902 and 905.

FIG. 6A is a display example of the display unit 112 immediately after the image data 905 is photographed by the image pickup apparatus 1 or after the reproduction button 113-5 is pressed after the image capture. Here, when the composition button 113-6 of the image pickup apparatus 1 is pressed, the image data 903 is searched according to the flowchart of FIG. 5, and the image data 902 and 905 are combined to generate the image data 906. FIG. 6B is a display example of the image data 906 synthesized by the image pickup apparatus 1.

The image data 901 to 904 have the same shooting location as the image data 905 (S502; Yes).
And the shooting date and time is within a predetermined time (S503; Yes). The image data 904 includes the same person as the image data 905 (S505; Yes). The background of the image data 903 is different from that of the image data 905 (S504; No). The image data 901 and 902 have the same background as the image data 905 (S504; Yes) and do not include the same person (S505;
No), and the persons do not overlap (S506; No). Of these, image data 902 is selected as the newest image data and combined with image data 905 to generate composite image data 906. In the example of FIG. 9, the image 905 corresponds to the first image data, and the image 902 corresponds to the second image data. The image data 903 is excluded because the background is different from that of the image data 905. However, when the image pickup apparatus 1 includes a lens having a zoom function, the image data whose lens focal length differs by a predetermined value or more may be excluded.

It should be noted that the deviation of the angle of view of each image data can be absorbed by performing the composition with reference to the background. For example, the person B of the image data 902 is extracted by contour detection and pasted on the image data 905 with reference to the background. Alternatively, the person A of the image data 905 may be extracted by contour detection and combined with the image data 902 with reference to the background.

FIG. 10 is an explanatory diagram showing an example of searching and synthesizing image data according to the first embodiment.

Image data 1001 and 1002 have latitude 35 ° 24'53 "north and longitude 138 ° 51'31 east.
The image data are taken in order from 10:02: 00 to 10:02:50 on 2015.03.16. The image data 1001 and 1002 have the same background.

The image data 1001 is image data in which the person B is photographed by the person A, and the image data 1002 is image data in which the person A is photographed by the person B.

When the composite button 113-6 of the image pickup apparatus 1 is pressed after the image data 1002 is captured, the images are composited. Alternatively, in the automatic composition mode, the images are automatically combined after shooting.

The composite image data 1003 is image data synthesized from the image data 1001 and 1002.

After recording the composite image data 1003 in the storage unit 104 or the external recording medium 106, or without recording, the image data 100 is a new image taken by the person B by the person A.
It is 4. Both the image data 1001 and 1004 are image data in which the person B is photographed by the person A, but since the person B in the image data 1001 does not have a smile, the image data 1004 is photographed again by the person A. show. In addition, the person B can take a picture of the person A again.

When the compositing button 113-6 of the image pickup apparatus 1 is pressed after the image data 1004 is captured, the images are composited again. Alternatively, in the automatic composition mode, the images are automatically combined again after shooting.

The image data 1005 is image data synthesized from the image data 1002 and 1004.

Further, it is possible to take a picture again, and it is possible to obtain a desired composite image by repeating shooting and compositing as many times as necessary.

As described above, in the present embodiment, when the compositing button is pressed after the imaging device has taken an image to be synthesized, the imaging device can search for the optimum image and perform compositing. Alternatively, the image pickup apparatus can automatically search for the optimum image and perform compositing after shooting.

Any number of images to be combined may be taken, and the imaging device searches for the most suitable image for composition. Further, a plurality of images to be combined can be combined regardless of the shooting timing as long as they have the same background. Therefore, even if only one image is retaken or an image that is not the target of composition is taken during the shooting timing of the image to be combined, the image that is not the target of composition is excluded and the composition process is executed. It is possible. again,
A desired composite image can be obtained by repeating shooting and compositing as many times as necessary. As a result, it is possible to improve the operability when a plurality of people take a composite image that fits in one image. In the present embodiment, an example in which a newer image is preferentially selected is shown, but an image including a smiling person or a person facing the front may be preferentially selected.

<Second embodiment>
The second embodiment is an embodiment in which composition is performed using image data selected by the user.

[Overall operation of the imaging device]
FIG. 11 is a flowchart showing an overall operation example of the image pickup apparatus according to the second embodiment.

Since S1101 to S1108 are the same as S301 to S308 in FIG. 3, the description thereof will be omitted.

The image data imaged by the image capturing unit 110 and stored in the image temporary storage area 1032 is displayed in the display unit 11.
It is displayed in 2 (S1109).

When the play button 113-5 is pressed (S1108; Yes), or when the image change is selected by the selection / decision operation button 113-3 (S1111; Yes), the storage unit 104 or the external recording medium 106 is displayed. The recorded image data is reproduced by the image reproduction unit 1025 and displayed on the display unit 112 (S1110). Here, the selection / decision operation button 11
3-3 corresponds to the selection operation unit because it functions as an operation member for the user to select the image stored in the storage unit 104.

FIG. 13A is a diagram showing a display example of the display unit 112 when the play button 113-5 is pressed.

When the user selects whether or not to change the image data displayed on the display unit 112 to another image data by using the selection / decision operation button 113-3, the image data displayed on the display unit 112 is different. Instead of image data (S1111; Yes).

FIG. 13B is a diagram showing a display example of the display unit 112 by the image pickup apparatus 1 when the image data is changed to another image data.

Since S1112 is the same as S312 in FIG. 3, the description thereof will be omitted.

When the composition button 113-6 is pressed (S1112; Yes), or when the image change is selected by the selection / decision operation button 113-3 (S1116; Yes), the storage unit 104 or the external recording medium 106 is displayed. The recorded image data is synthesized by the synthesis processing unit 1027 (S1113: Yes). The composition is performed in the image temporary storage area 1032.

Since S1115 is the same as S315 in FIG. 3, the description thereof will be omitted.

FIG. 14A is a display example of the display unit 112 when the composite button 113-6 is pressed.

The user selects whether or not to change the image data to be combined to another image data by pressing the selection / decision operation button 113-3 (S1116).

FIG. 14B is a display example of the display unit 112 by the image pickup apparatus 1 when the image data is changed to another image data. The image data 1404 selected in S1116 is not changed, and the image data 1401 to 1403 to be combined with the image data 1404 is changed.

Since S1117 to S1118 are the same as S317 to S318 in FIG. 3, the description thereof will be omitted.

By performing the above processes S1101 to S1118, images are recorded, reproduced, and combined.

[Image composition processing]
Next, the image composition process of S1113 will be described. FIG. 12 is a flowchart showing an operation example of the synthesis processing unit according to the second embodiment.

It is confirmed whether or not the user has instructed to change the image (S1200).

When there is no image change instruction by the user (S1200; No), the newest image is searched for on the same shooting date as the image displayed on the display unit (S1201).

Since S1202 to S1209 are the same as S502 to S509 in FIG. 5, the description thereof will be omitted.

When the user gives an image change instruction (S1200; Yes), the shooting location of the search image and the display image is not the same (S1202; No), or the shooting date and time is not within the predetermined time (S1203; No). Or when the backgrounds are not the same (S120)
4; No), or when the same person is included (S1205; Yes), or when the persons to be combined overlap (S1206; Yes), it is confirmed whether or not there is another image on the same shooting date (S1208). ), If there is another image (S1208; Yes), then the new image is searched (S1209), the process returns to S1202, and the process is continued. If there is no other image (
S1208; No) ends the process.

By performing the above processes S1200 to S1209, an image to be pasted on the display image is searched and combined.

FIG. 15 is an explanatory diagram showing an example of searching and synthesizing image data according to the second embodiment.

The image data 1501 to 1505 are at latitude 35 ° 24'53 "north and longitude 138 ° 51'31 east.
The image data are taken in order from 10:02: 00 to 10:02:55 on 2015.03.15. The image data 1501 to 1505 have the same background. The image data 150
Reference numerals 1 to 1503 are image data in which the person B is photographed by the person A, and the image data 150
4 and 1505 are image data obtained by the person A taken by the person B.

When the shutter button 113-2 of the image pickup device 1 is pressed, the subject is photographed and the display unit 112
Display the captured image in. Then, when the composite button 113-6 is pressed, the images are composited.

The image data 1506 is image data synthesized from the image data 1502 and 1504.

As described above, according to the present embodiment, the same effect as that of the first embodiment is obtained, and further,
The user can freely select the image to be combined.

<Third Embodiment>
The third embodiment is an embodiment in which only the registered person is added to the combined image and the unregistered person is deleted from the combined image.

[Overall operation of the imaging device]
FIG. 16 is a flowchart showing an overall operation example of the image pickup apparatus according to the third embodiment.

Since S1601 to S1612 are the same as S301 to S312 in FIG. 3, the description thereof will be omitted.

FIG. 18A is a diagram showing a display example of the display unit 112 immediately after shooting by the image pickup apparatus 1 or when the playback button 113-5 is pressed. Person A is photographed against the background of the mountain, but unregistered persons C and D are also photographed. Persons A and B are registered as important persons in advance. again,
When a photograph is taken by touching a person displayed on the display unit 112 provided with a touch sensor instead of the shutter button 113-2, the person may be registered as an important person.

When the composite button 113-6 is pressed (S1612; Yes), the image data captured by the imaging unit 110 and stored in the image temporary storage area 1032 and the image data recorded in the storage unit 104 or the external recording medium 106. Is synthesized by the synthesis processing unit 1027 (S1613).
.. The composition is performed in the image temporary storage area 1032, and the registered person is added by the composition and the unregistered person is deleted.

FIG. 18B is a diagram showing a display example of image data synthesized by the image pickup apparatus 1. A person B taken against the background of the same mountain is combined with an image including a person A taken against the background of a mountain.
And the unregistered persons C and D have been deleted.

Since S1615 to S1618 are the same as S315 to S318 in FIG. 3, the description thereof will be omitted.

By performing the above processes S1601 to S1618, images are recorded, reproduced, and combined.

[Image composition processing]
Next, the image composition process of S1613 will be described.

FIG. 17 is a flowchart showing an operation example of the synthesis processing unit according to the third embodiment.

Since S1701 to S1706 are the same as S501 to S506 in FIG. 5, the description thereof will be omitted.

The shooting location of the search image and the display image is the same (S1702; Yes), the shooting date and time is within the predetermined time (S1703; Yes), the background is the same (S1704; Yes), and the same person is not included (S1705; No). ) And when the people do not overlap (S1706; No),
The registered person in the search image is extracted by contour detection and pasted on the display image for composition. Alternatively, the registered person in the display image is extracted by contour detection, pasted on the search image, and combined (S).
1707).

Further, the background of the search image at the position where the unregistered person of the display image exists is extracted and pasted on the display image (background data is combined) to delete the unregistered person. Alternatively, the background of the display image at the position where the unregistered person in the search image exists is extracted and pasted on the search image to delete the unregistered person.

Since S1708 to S1709 are the same as S508 to S509 of FIG. 5, the description thereof will be omitted.

After synthesizing the images (S1707), or when there is no other image on the same shooting date (S1708;
No), the synthesis processing unit 1027 confirms whether or not there is an unregistered person in the synthesized image data (S1710).

When there is an unregistered person in the combined image data (S1710; Yes), the composition processing unit 1027 confirms whether or not there is another image (S1711).

If there is another image (S1711; Yes), another image is searched and extracted (S171).
2) If there is no other image (S1711; No), the process ends.

The compositing processing unit 1027 confirms whether or not the shooting location of the search image and the shooting location of the composited image are the same (S1713).

When the shooting locations are the same (S1713; Yes), the compositing processing unit 1027 confirms whether or not the shooting date and time of the search image and the shooting date and time of the composited image are within a predetermined time (S1714). The predetermined time is, for example, about 3 minutes and may be changed by the user.

When the shooting date and time is within the predetermined time (S1714; Yes), the composition processing unit 1027 confirms whether or not the background of the search image and the background of the display image are the same (S1715).

When the background is the same (S1715; Yes), the synthesis processing unit 1027 deletes the unregistered person (S1716) and returns to S1710.

When the unregistered person is not included (S1710; No) or when there is no other image on the same shooting date (S1711; No), the process ends.

By performing the above processes S1700 to S1716, the image to be pasted on the display image is searched, the registered person is extracted and combined, and the unregistered person is deleted.

FIG. 19 is an explanatory diagram showing an example of searching and synthesizing image data.

Image data 1901-1903 has latitude 35 ° 24'53 "north and longitude 138 ° 51'31 east.
The image data is taken in order from 10:02: 00 to 10:02:50 on 2015.03.16. The image data 1901 to 1903 have the same background.

The image data 1901 and 1902 are image data in which the person B is photographed by the person A, but the unregistered person D is also photographed in the image data 1902. Since the person B in the image data 1901 does not have a smile, it indicates that the image data 1902 has been re-photographed by the person A. The image data 1903 is image data in which the person A is photographed by the person B, but the unregistered persons C and D are also photographed. Persons A and B are registered as important persons in advance. Further, when a person is photographed by touching a person displayed on the display unit 112 provided with a touch sensor instead of the shutter button 113-2, the person may be registered as an important person.

When the shutter button 113-2 of the image pickup device 1 is pressed, the subject is photographed and the display unit 112
Display the captured image in. Then, when the composite button 113-6 is pressed, the images are composited.

The image data 1904 is image data obtained by synthesizing a registered person and deleting an unregistered person C from the image data 1902 and 1903.

For example, the person B of the image data 1902 is extracted by contour detection and pasted on the image data 1903 with reference to the background, thereby synthesizing the person B with the image data 1902. Further, the unregistered person C can be deleted by pasting the background of the image data 1902 at the position where the unregistered person C of the image data 1903 exists on the unregistered person C of the image data 1903. Alternatively, the person A of the image data 1903 may be extracted by contour detection and combined with the image data 1902 with reference to the background to generate the combined image data 1904. When there is an unregistered person in either one or both image data, it is preferable to extract the registered person from the image data having many unregistered people and paste it in the image data having few registered people.

Next, the unregistered person D is deleted by pasting the background of the image data 1901 at the position where the unregistered person D of the image data 1904 exists on the unregistered person D of the image data 1904.

The image data 1905 is image data obtained by deleting the unregistered person D from the image data 1904 and 1901.

As described above, according to the present embodiment, the same effect as that of the first embodiment can be obtained, and further, only the registered person can be added to the image and the unregistered person can be deleted. In this embodiment, it should be noted that
Although the example of adding the registered person to the image and deleting the unregistered person is shown, the unregistered person can be deleted without adding the registered person to the image.

<Fourth Embodiment>
The fourth embodiment is an image of only the background in which a person is not captured (hereinafter referred to as "background image").
In the embodiment, the background image is searched when the background image is imaged, and the image data in which the background imaged is published on the network is searched when there is no background image, and the unregistered person is deleted using the background image. be.

[Hardware configuration of imaging device]
In FIG. 20A, the same processing portions as those in FIG. 1A are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 20A is a block diagram showing a configuration example of the image pickup apparatus according to the fourth embodiment. FIG. 20A
The image pickup device 1 shown in FIG. 2 has a wireless communication unit 120, is connected to the external network 3 via the access point device 2, and transmits / receives data to / from the server device 4 on the external network 3. The connection with the access point device 2 shall be made by wireless connection using Wi-Fi (registered trademark) or the like. The access point device 2 may be a base station of a mobile communication operator. Further, the image pickup device 1 may not directly send / receive data to / from the access point 2, but may send / receive data via a smartphone or a mobile terminal.

The server device 4 has a plurality of image data with position information and a plurality of image data without position information, and the image pickup device 1 can acquire various image data via the access point device 2 and the network 3.

[Software configuration of imaging device]
FIG. 20B is a software configuration diagram of the image pickup apparatus 1 of this embodiment, and the same processing portions as those in FIG. 1B are designated by the same reference numerals, and the description thereof will be omitted.

It is a figure which shows the software block diagram of the image pickup apparatus which concerns on 4th Embodiment shown in FIG. 20B. R
The OM unit 102 has a wireless connection processing unit 1029, and controls the wireless communication unit 120 to perform wireless connection processing with the access point device 2. In addition, the position information acquisition unit 102
2. Performs a process of updating the position information of the metadata of the image file recorded in the storage 104 of the main body or the external recording medium 106.

[Image composition processing]
Next, the image composition process will be described. FIG. 21 is a flowchart showing an operation example of the synthesis processing unit according to the fourth embodiment.

Since S2101 to S2110 are the same as S1701 to S1710 in FIG. 17, the description thereof will be omitted.

When there is an unregistered person in the image data (S2110; Yes), the storage unit 1
It is confirmed whether or not there is another image in the image storage area 1042 of 04 (S2111).

If there is another image (S2111; Yes), another image is searched (S2112), and if there is no other image (S2111; No), the process proceeds to S2117.

Since S2113 to S2116 are the same as S1713 to S1716 in FIG. 17, the description thereof will be omitted.

When there is no other image in the image storage area 1042 of the storage unit 104 (S2111; No.
) Connects to the external network 3 via the access point device 2 and searches for an image of the server device 4 on the external network 3 (S2117). Then, the same process as in S2116 is performed using the image on the network, and the unregistered person is deleted (S2118).

By performing the above processes S2101 to S2118, the image to be pasted on the display image is searched, the registered person is extracted and combined, and the unregistered person is deleted.

FIG. 22 is an explanatory diagram showing an example of searching and synthesizing image data according to the fourth embodiment.

Image data 2201 to 2203 are 35 ° 24'53 "north latitude and 138 ° 51'31 east longitude.
The image data are taken in order from 10:02: 00 to 10:02:50 on 2015.03.16. The image data 2201 to 2203 have the same background.

The image data 2201 is image data in which the person B is photographed by the person A, but the unregistered person D is also photographed. The image data 2202 is image data in which the person A is photographed by the person B, but the unregistered persons C and D are also photographed. Persons A and B are registered as important persons in advance. Further, when a person is photographed by touching a person displayed on the display unit 112 provided with a touch sensor instead of the shutter button 113-2, the person may be registered as an important person.

When the shutter button 113-2 of the image pickup apparatus 1 is pressed, the subject is photographed and the display unit 11
The image taken in 2 is displayed. Then, when the composite button 113-6 is pressed, the images are composited.

The composite image data 2203 is image data obtained by synthesizing a registered person and deleting an unregistered person C from the image data 2201 and 2202.

For example, the person B of the image data 2201 is extracted by contour detection and pasted on the image data 2202 with reference to the background, and the background of the image data 2201 at the position where the unregistered person C of the image data 2202 exists is the image data 2202. The unregistered person C is deleted by pasting it on the unregistered person C. Alternatively, the person A of the image data 2202 is extracted by contour detection and pasted on the image data 2201 with reference to the background. When there is an unregistered person in either one or both image data, it is preferable to extract the registered person from the image data having many unregistered people and paste it in the image data having few registered people.

Then, the unregistered person D can be deleted by re-shooting and pasting an image (background image) in which the unregistered person D does not exist in the same background.

However, if the unregistered person D does not leave the place for a long time, it is not possible to take an image (background image) in which the unregistered person D does not exist in the same background.

Therefore, the image data published on the network is searched, and the searched image data 2
At 204 (35 ° 24'53 "north latitude, 138 ° 51'31" east longitude, 2015.03.1.
0 09:30) is pasted and the unregistered person D is deleted. Image data is searched by pattern matching to find similar image data. You may search by shooting location, shooting season, shooting date and time, and so on.

The composite image data 2205 is image data in which the unregistered person D is deleted from the composite image data 2203 using the image data 2204 searched on the network.

The unregistered person D is deleted by pasting the background of the image data 2204 at the position where the unregistered person D of the composite image data 2203 exists on the unregistered person D of the composite image data 2203.

As described above, in the present embodiment, the same effect as that of the third embodiment can be obtained, and further, the image data published on the network can be searched, and the unregistered person can be deleted by using the searched image data. ..

Further, the user may be able to combine with the image data taken at the same place based on the shooting place of the image data published on the network.

Although the embodiment of the present invention has been described above, it goes without saying that the configuration for realizing the technique of the present invention is not limited to the above embodiment, and various modifications can be considered. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the category of the present invention. In addition, numerical values and messages appearing in sentences and figures are merely examples, and the effects of the present invention may not be impaired even if different ones are used.

Further, the programs described in each processing example may be independent programs, or a plurality of programs may constitute one application program. Further, the order in which each process is performed may be changed and executed.

The above-mentioned functions of the present invention and the like may be realized by hardware, for example, by designing a part or all of them by an integrated circuit. Further, it may be realized by software by interpreting and executing an operation program in which the microprocessor unit or the like realizes each function or the like. Hardware and software may be used together.

In addition, the control lines and information lines shown in the figure indicate those considered necessary for explanation, and do not necessarily indicate all the control lines and information lines on the product. In practice, it can be considered that almost all configurations are interconnected.

1: Imaging device, 101: Main control unit, 102: ROM, 103: RAM, 104: Storage unit, 106: External recording medium, 110: Imaging unit, 115: GPS receiving unit, 120: Wireless communication unit, 1022: Position Information acquisition unit, 1023: Imaging processing unit, 1024: Image recording unit, 10
25: Image reproduction unit, 1027: Synthesis processing unit, 1028: Face recognition processing unit

Claims (1)

An imaging processing unit that captures an image of a subject and generates image data,
A face recognition processing unit that executes face recognition processing of the image data,
An image recording unit that records the image data and
It is provided with a compositing processing unit that generates a compositing data by executing a compositing process so that a person imaged in each of a plurality of image data is included in one image data.
The face recognition processing unit executes the face recognition process on the first image data to recognize the face of the first person.
A second image that is shot at an arbitrary shooting timing different from the shooting timing of the first image data, has the same background as the background of the first image data, and is different from the first person. When the second image data in which a person is captured is recorded in the image recording unit,
The synthesis processing unit uses the first image data and the second image data to generate the composite data in which the first person and the second person are superimposed on the same background.
An imaging device characterized by this.

Images