JP6412400B2 - Image composition apparatus and image composition program - Google Patents

Description

  The present invention relates to an image composition apparatus and an image composition program for compositing a foreground of a photographed image and a background photographed in advance.

  2. Description of the Related Art Conventionally, there is a data format in which shooting information such as shooting date / time and shooting conditions (focal length, aperture value, etc.) is recorded in association with an image shot by a camera. This data format is, for example, Exif (Exchangeable image file format) standardized by the Japan Electronics and Information Technology Industries Association (JEITA) (see Non-Patent Document 1). This Exif is mainly a metadata file format for recording the state at the time of shooting by the camera. In addition, Exif can record shooting information such as shooting date and time, shooting conditions and the like, and position information such as latitude, longitude, and altitude measured by GPS corresponding to a camera with GPS.

  Shooting information (including position information) associated with such an image is not only used by a photographer for confirmation after shooting, but can be used in various applications. For example, as disclosed in Patent Document 1, a subject image (foreground image) captured by a photographer and a landscape image (background image) stored in advance in a storage device corresponding to the position of the measured shooting information ) And can be used to synthesize.

  In this way, by combining the background image corresponding to the position of the shooting information with the foreground image, for example, when taking a commemorative photo, a landscape photo, etc. in a tourist spot, the image is not taken beautifully due to bad weather etc. Even in such a case, it is possible to synthesize a beautiful landscape image taken in good weather stored in the storage device in advance as a background image.

Japanese Patent Laid-Open No. 2005-210412

"Image file format standard for digital still camera Exif2.3", created by AV & IT Standardization Committee, published by Japan Electronics and Information Technology Industries Association, revised in May 2013

However, the above-mentioned Exif is not stipulated to record the azimuth information indicating which direction the camera is facing, even though the position information can be recorded as the shooting information of the camera.
For this reason, for example, an image taken by a photograph of a mountain as a landscape photograph at a certain position and an image taken by a camera of a building facing in the opposite direction are completely different images. The position information recorded in is almost the same information.
That is, when a landscape image (background image) is synthesized by the method described in Patent Document 1 using information recorded in conventional Exif, a background image that is completely different from the image desired by the photographer depending on the orientation of the camera. Is combined with the foreground image.

  The present invention has been made in view of such problems, and it is possible to synthesize a background image suitable for the position, orientation, orientation, and other shooting conditions of a camera with a foreground image of a person photographed by a photographer. It is an object of the present invention to provide a possible image composition device and an image composition program.

  In order to solve the above-described problem, an image composition device according to the present invention is an image composition device that combines images taken at different times with a foreground of a photographed image as a background, and includes a photographing unit, photographing information acquisition unit, and foreground. An area detecting unit, a foreground image cutting unit, a background candidate image acquiring unit, a background candidate image selecting unit, and a synthesizing unit are provided.

In such a configuration, the image synthesizing apparatus captures an image of the subject by the imaging unit via the optical system including the lens. In addition, the image composition device uses the photographing information acquisition means as the photographing information of the image composition device, position information indicating the latitude, longitude, and altitude when the subject is photographed, direction information indicating the photographing direction, and posture (tilt) ) Is acquired via the detecting means. Further, the image synthesizing apparatus acquires lens information (focal length, focus position, aperture value, angle of view, etc.) of the optical system by photographing information acquisition means.
As a result, the image composition apparatus acquires a captured image obtained by photographing the subject and photographing information obtained when the subject is photographed at the same timing.

  Then, the image composition apparatus detects the foreground region in the captured image based on the depth information in which the distance to the subject is associated with the pixel of the captured image captured by the capturing unit by the foreground region detecting unit. The depth information can be obtained by a general method such as parallax of a stereo image or measurement by a laser. In the foreground, an area corresponding to a predetermined depth from the closest depth (frontmost) of depth information in the photographed image may be regarded as a subject area, or the foreground is focused with reference to the lens information. It is also possible to regard the area that is present as the area of the subject. Then, the image composition apparatus generates a foreground image by cutting out the foreground area from the captured image by the foreground image cutting means.

In addition, the image composition device uses the background candidate image acquisition means to obtain background photographing information and background candidate images including position information, orientation information, posture information, and lens information of the camera that has photographed the background candidate image as a background candidate. A background candidate image that matches a search condition in which shooting information and background shooting information approximate within a predetermined range is acquired from a previously associated database via a communication control unit.
As a result, the image composition device can acquire a background candidate image photographed under a condition approximate to the photographing condition photographed at the present time.

Then, the image composition apparatus displays the background candidate image acquired by the background candidate image acquisition unit on the screen of the display unit by the background candidate image selection unit, and accepts the selection from the photographer. As a result, one background image to be combined is determined.
Then, the image synthesizing device synthesizes the background candidate image selected by the background candidate image selecting unit with the foreground image generated by the foreground image cutting unit using the background candidate image selected by the background candidate image selecting unit. As a result, the image composition device can synthesize the foreground area of the photographed image photographed at the present time and the background image photographed at different times that approximate the photographing conditions.

The present invention has the following excellent effects.
According to the present invention, it is possible to synthesize a background image captured at different times in advance close to the captured foreground, such as the position, orientation, orientation, and lens imaging conditions. As a result, the present invention uses the subject desired by the photographer as the background even when the photographer wants to take a commemorative photo or a landscape photograph at a sightseeing spot or the like, even if the photographer's desired background is not photographed beautifully due to weather conditions. A beautiful background image taken in advance can be synthesized.

It is a system outline figure for explaining an outline of an image composition system concerning an embodiment of the present invention. It is a block block diagram which shows the structure of the image provision apparatus of FIG. It is a data structure figure for demonstrating the structure of the database memorize | stored in the image storage means of FIG. It is explanatory drawing for demonstrating the attitude | position information in the database memorize | stored in the image storage means of FIG. It is a block block diagram which shows the structure of the image imaging device (image composition apparatus) of FIG. It is a figure which shows the example of a display of the screen which selects the background image which the background candidate image selection means of FIG. 5 displays. FIG. 6 is an explanatory diagram for explaining the foreground image cutout by the foreground image cutout unit in FIG. 5. It is explanatory drawing for demonstrating the synthesis | combination of a foreground image and a background image in the synthetic | combination means of FIG. It is a flowchart which shows the operation | movement of the image registration of the image composition system which concerns on embodiment of this invention. It is a flowchart which shows the operation | movement of an image composition of the image composition system which concerns on embodiment of this invention. FIG. 5 is a diagram for explaining a modification of the present invention, in which (a) shows the difference in the angle of view between the captured image and the background candidate image, and (b) shows an example in which the background candidate image is deformed according to the angle of view. . FIG. 6 is a diagram for explaining a modification of the present invention, where (a) shows a difference in image due to a difference in posture between a captured image and a background candidate image, (b) shows an image obtained by rotating (a), c) shows an example of transforming the background candidate image from the image of (b) according to the angle of view.

Embodiments of the present invention will be described below with reference to the drawings.
≪Outline of image composition system≫
First, an overview of the image composition system S according to the embodiment of the present invention will be described with reference to FIG.

The image composition system S combines a foreground image of an image photographed by the image photographing device (image composition device) 2 with a background image photographed in advance under conditions close to the photographing conditions of the image photographing device 2.
Here, the image composition system S includes an image providing device 1 and an image photographing device 2 that can transmit data by communication.

  The image providing apparatus 1 provides the image photographing apparatus 2 with background candidate images BG that have been photographed in advance at a plurality of points and serve as background image candidates. Here, the image providing apparatus 1 registers a background candidate image BG in the database in the image storage unit 10 in association with shooting information (background shooting information) indicating shooting conditions. Then, the image providing apparatus 1 provides the image capturing apparatus 2 with a background candidate image BG that approximates the shooting information (shooting conditions) notified from the image capturing apparatus 2 based on a predetermined criterion.

An image photographing device (image synthesizing device, camera) 2 photographs a subject. Here, the subject is the foreground F and the other background B that the image capturing apparatus 2 captures in focus.
The image photographing device 2 notifies the image providing device 1 through communication, using a photographing condition approximate to its own photographing information as a search condition. Then, the image capturing device 2 acquires a background candidate image BG that matches the search condition from the image providing device 1.
Then, the image capturing device 2, the foreground image corresponding to the foreground F of the image G ₁ taken at this time, by combining the acquired candidate background image BG, to generate a new image G _2.

As a result, the image composition system S actually uses a background image (background candidate image BG) that was captured in advance in good weather even when the background was not captured beautifully due to cloudy weather or fog in a tourist spot. It is possible to synthesize the foreground of the captured image.
Here, it is assumed that the image capturing apparatus 2 has a function of registering a captured image in the image providing apparatus 1 as a background candidate image BG, in addition to the function of combining the foreground image and the background image.
Hereinafter, the configuration of the image providing device 1 and the configuration of the image capturing device 2 that constitute the image composition system S will be sequentially described.

[Configuration of image providing device]
First, the configuration of the image providing apparatus 1 will be described with reference to FIG. 2 (refer to FIG. 1 as appropriate).
As shown in FIG. 2, the image providing apparatus 1 includes an image storage unit 10, a communication control unit 11, and an image registration search unit 12.

  The image storage means 10 stores a database in which shooting information D indicating image shooting conditions is associated with a background candidate image BG that is a background image candidate. The image storage means 10 is a general storage device such as a hard disk storage device, and here, the image registration search means 12 writes and reads out the photographing information D and the background candidate image BG.

Here, the data structure of the database stored in the image storage means 10 will be described with reference to FIG.
As shown in FIG. 3, shooting information (background shooting information) D and a background candidate image BG are stored in association with each other in the database of the image storage unit 10.
Here, the shooting information D and the background candidate image BG are associated with each other by an identifier ID (ID1, ID2, ID3,...) That can uniquely specify the background candidate image BG.

The shooting information (background shooting information) D is information indicating shooting conditions when the corresponding background candidate image BG is shot. Here, the shooting information D is image information, position information, orientation information, posture information, lens information, and the like.
“Image information” is information for specifying the background candidate image BG. This image information includes, for example, an identifier that uniquely identifies the background candidate image BG, the date and time when the background candidate image BG was captured, the resolution of the background candidate image BG, the image format, and the like.
“Position information” is information indicating the position of the image capturing apparatus 2 when the background candidate image BG is captured. The position information includes latitude, longitude, altitude, and the like that can be acquired by the image capturing apparatus 2 using GPS (Global Positioning System).
“Direction information” is information indicating the orientation (imaging direction) of the image capturing apparatus 2 when the background candidate image BG is captured. This azimuth information is a azimuth that can be acquired by the geomagnetism sensor or the like in the image capturing apparatus 2.

“Attitude information” is information indicating the attitude (tilt) of the image capturing apparatus 2 when the background candidate image BG is captured. This posture information is a rotation angle (φ, θ, ψ) in each of the roll, pitch, and yaw directions that can be acquired by the inertial sensor (gyro sensor, acceleration sensor, etc.) in the image capturing apparatus 2.
As shown in FIG. 4, the “roll direction” is a rotation direction around an imaging direction axis (X axis in the drawing) of the image capturing apparatus 2. The “pitch direction” indicates a rotation direction around the horizontal axis (Y axis in the figure) of the image capturing apparatus 2. The “yaw direction” indicates a rotation direction around the vertical axis (Z axis in the drawing) of the image capturing apparatus 2.

“Lens information” is information on the lens of the image capturing apparatus 2 when the background candidate image BG is captured. The lens information includes a focal length, a focus position, an aperture value (F value), a field angle, and the like. However, it is not necessary to directly store the value of the angle of view, and information capable of calculating the angle of view may be stored. For example, if the resolution of the image sensor of the image capturing device 2 is stored, the angle of view can be calculated from the resolution and the focal length.
The photographing information D is not limited to such information, and may include information defined by conventional Exif.

Thus, by registering at least position information, azimuth information, and posture information, the shooting information D corresponds to the image shooting device 2 in which position, in which direction, in any posture. This is information that can uniquely identify whether the background candidate image BG is taken.
Further, by registering the lens information, the shooting information D becomes information that can uniquely identify the range (view angle) of the image shooting device 2 in which the subject is shot.
Returning to FIG. 2, the description of the configuration of the image providing apparatus 1 will be continued.

The communication control unit 11 transmits / receives data to / from the image capturing device 2 through communication. The communication control unit 11 transmits and receives data using a wireless LAN such as Wi-Fi (Wireless Fidelity).
Here, the communication control unit 11 outputs the data received from the image capturing device 2 to the image registration search unit 12, and transmits the data input from the image registration search unit 12 to the image capturing device 2.

The image registration search means 12 performs registration and search of a database stored in the image storage means 10. In other words, the image registration / retrieval means 12 has a database management system function that responds to database management and data access requests.
The image registration / retrieval unit 12 acquires, as a registration request, shooting data including the shooting information D and the background candidate image BG from the image shooting device 2 via the communication control unit 11 and registers them in the database of the image storage unit 10. (Remember. At this time, the image registration search unit 12 sequentially assigns a unique value for the identifier ID in the image information of the shooting information D.
The image registration / retrieval means 12 uses a recording medium such as an SD card in which the photographing information D and the background candidate image BG are recorded, and the photographing information D and the background candidate image BG via a recording medium reading device (not shown). May be read out and registered in the database.

In addition, when the image registration search unit 12 acquires a search request using the shooting information as a search condition from the image shooting apparatus 2 via the communication control unit 11, a background that matches the search condition from the database of the image storage unit 10. Candidate image BG is read. Then, the image registration / retrieval unit 12 transmits the read background candidate image BG to the image capturing apparatus 2 that has made the search request via the communication control unit 11.
Here, the search condition is, for example, which information in the shooting information D is the search condition such as the latitude, longitude, and altitude, each value and permissible range, and the azimuth and permissible range. Attribute, its value and tolerance.

[Configuration of Image Shooting Device (Image Composition Device)]
Next, referring to FIG. 5 (refer to FIG. 1 as appropriate), the configuration of the image capturing device (image composition device) 2 will be described. As shown in FIG. 5, the image capturing apparatus 2 includes a shooting information acquisition unit 20, a shooting unit 21, a foreground area detection unit 22, a foreground / background synthesis unit 23, a communication control unit 24, a display unit 25, A storage unit 26, an image writing unit 27, and an image registration unit 28 are provided. The image photographing apparatus 2 photographs a subject by pressing the shutter switch SW.

Here, the image capturing apparatus 2 includes therein a switching unit T (T ₁ , T ₂ ) that switches whether to synthesize a background image provided by the image providing apparatus 1, and switches the path to the a terminal side. Thus, the background image is synthesized and the route is switched to the b terminal side so that the background image is not synthesized. This switching unit T may be a hardware changeover switch set from the outside, or may be a switching control means by software control that performs switching by setting from a setting screen.

The shooting information acquisition means 20 acquires shooting information indicating the position, orientation, posture, and lens state of the image shooting apparatus 2 when the subject is shot.
Here, the photographing information acquisition means 20 includes a position detection means 201, a position information acquisition means 202, an orientation detection means 203, an orientation information acquisition means 204, an orientation detection means 205, an orientation information acquisition means 206, and a lens. Information acquisition means 207.

The position detection unit 201 detects the position of the image capturing device 2. The position detection unit 201 includes a GPS antenna (not shown), and calculates the latitude, longitude, and altitude of the image capturing device 2 from the time lag of radio waves received from a plurality of GPS satellites (not shown).
The position detection unit 201 calculates latitude, longitude, and altitude at a timing when a detection request is received from the position information acquisition unit 202, and outputs the calculated latitude, longitude, and altitude to the position information acquisition unit 202 as position information.

The position information acquisition unit 202 acquires the position information of the image capturing device 2 from the position detection unit 201. The position information acquisition unit 202 requests position information from the position detection unit 201 and acquires the position information at a timing when shooting is instructed by the shutter switch SW. Then, the position information acquisition unit 202, when the switching unit T ₁ is set to a terminal side, and outputs the position information to the foreground background synthesis means 23, if it is set to the terminal b, the position information image Output to the writing means 27.

The direction detection means 203 detects the direction of the image capturing device 2. This azimuth detecting means 203 can be constituted by a triaxial geomagnetic sensor, for example. In the case of a triaxial geomagnetic sensor, the front direction of the image capturing device 2 is detected by detecting a reference direction (for example, the north direction) from the front, back, left, and right top geomagnetism.
The azimuth detecting unit 203 sequentially outputs the current azimuth of the image capturing device 2 to the azimuth information acquiring unit 204.

The direction information acquisition unit 204 acquires the direction information of the image capturing device 2 from the direction detection unit 203. The azimuth information acquisition unit 204 acquires the azimuth information output from the azimuth detection unit 203 at the timing when shooting is instructed by the shutter switch SW.
The orientation information acquisition means 204, when the switching unit T ₁ is set to a terminal side, if the outputs azimuth information to the foreground background synthesis means 23, is set to the terminal b, the image azimuth information Output to the writing means 27.

The attitude detection unit 205 detects the attitude of the image capturing apparatus 2. The posture detection means 205 can be constituted by an inertial sensor such as a gyro sensor or an acceleration sensor, for example. The attitude detection unit 205 detects the attitude of the image capturing apparatus 2 by detecting the rotation angles (φ, θ, ψ) in the roll, pitch, and yaw directions described in FIG.
The posture detection unit 205 sequentially outputs the current posture of the image capturing device 2 to the posture information acquisition unit 206.

The posture information acquisition unit 206 acquires the posture information of the image capturing device 2 from the posture detection unit 205. The posture information acquisition unit 206 acquires the posture information output from the posture detection unit 205 at the timing when shooting is instructed by the shutter switch SW.
The orientation obtaining unit 206, when the switching unit T ₁ is set to a terminal side, if the outputs posture information to the foreground background synthesis means 23, is set to the terminal b, the orientation information image Output to the writing means 27.

The lens information acquisition unit 207 acquires lens information of the photographing unit 21. The lens information acquisition unit 207 acquires the focal length, the focus position, the aperture value, and the like that are the state of the lens of the imaging unit 21 at the timing when imaging is instructed by the shutter switch SW.
The lens information acquisition unit 207 determines the angle of view (vertical angle of view and horizontal angle of view) from the image sensor size (horizontal size and vertical size) built in the previously known photographing unit 21 and the acquired focal length. Calculate as part of the information.
Then, the lens information acquisition unit 207, when the switching unit T ₁ is set to a terminal side, and outputs the lens information to the foreground background synthesis means 23, if it is set to the terminal b, the lens information image Output to the writing means 27.
As described above, the shooting information acquisition unit 20 can acquire shooting information (position information, azimuth information, posture information, and lens information) at the timing when the image shooting device 2 shots the subject.

The photographing means 21 photographs the subject at a timing when photographing is instructed by the shutter switch SW. The photographing means 21 is a photographing device including a lens (optical system) and an image sensor (not shown), and is a lens operated by a photographer or controlled by a known technique such as autofocus. The lens information such as the focal length and the aperture is output to the lens information acquisition unit 207.
The imaging unit 21, when the switching unit T ₁ is set to a terminal side, if the output an imaged image obtained by imaging the foreground background synthesis means 23, is set to the terminal b, the photographed image image Output to the writing means 27.

The foreground area detection unit 22 detects the foreground area in the captured image captured by the imaging unit 21 based on the depth information to the subject and the lens information when the subject is captured.
The foreground area detection unit 22 shoots a subject with an optical system having a parallax different from that of the imaging unit 21, for example, and generates depth information (depth image) from the two images (stereo images) based on the parallax. . Then, the foreground area detecting means 22 obtains the shooting distance to the foreground in focus specified by the focal length and the focus position of the lens, and the area of the shot image corresponding to the depth of the depth information corresponds to the shooting distance. A key signal is generated. For example, a key signal in which a pixel value “0xFF” is assigned to the foreground area and a pixel value “0” is assigned to the background area is generated.

  At this time, it is preferable that the foreground area detection means 22 provides a depth width of the depth of field before and after the foreground depth. This depth of field can be obtained from the focal length, aperture value (F value), and shooting distance of the lens when shooting. Of course, as the depth width, a predetermined width (for example, 30 cm or the like) may be used.

Here, the foreground area detection means 22 calculates the depth by the parallax, but may measure the depth by a laser sensor or an infrared sensor.
Further, the foreground area detection means 22 is not limited to the focus position, and the captured image corresponding to a depth (for example, 30 cm) in a predetermined range from a distance (frontmost) closest to the image capturing apparatus 2 in the depth information. These areas may be detected as foreground areas.

The foreground area detection unit 22 outputs the generated foreground key signal to the foreground / background composition unit 23. Note that this key signal does not use only when synthesizing the background image, if the path to the terminal b by the switching portion T ₂ are set, and is not performed to generate the key signal.

The foreground / background synthesizing unit 23, based on the shooting information acquired by the shooting information acquiring unit 20, the shot image shot by the shooting unit 21, and the key signal generated by the foreground area detection unit 22, The foreground area and the background candidate image photographed in advance are synthesized.
Here, the foreground / background composition unit 23 includes a background candidate image acquisition unit 231, a background candidate image selection unit 232, a foreground image cutout unit 233, and a composition unit 234.

The background candidate image acquisition unit 231 receives, from the image providing apparatus 1, a background candidate image that has been captured under a shooting condition specified by the shooting information acquired by the shooting information acquisition unit 20 and a shooting condition approximated within a predetermined range. To get.
The background candidate image acquisition unit 231 uses information obtained by providing a predetermined allowable range for the shooting information (position information, orientation information, posture information, lens information) acquired by the shooting information acquisition unit 20 as a search condition, and performs communication control. A background candidate image is requested to the image providing apparatus 1 via the means 24.
For example, for position information, an error within “1 m” for each of latitude, longitude, and altitude is used as a search condition. Further, for each information of the azimuth information and the posture information, an error within “5 degrees” is used as a search condition.

Then, the background candidate image acquisition unit 231 acquires a background candidate image via the communication control unit 24. At this time, if the number of background candidate images that match the search condition is smaller than the predetermined number, the background candidate image acquisition unit 231 sets the latitude, longitude, and altitude errors within “5 m”, “ It is possible to provide a level for the search condition by gradually relaxing the search condition to within 10 m ".
The background candidate image acquisition unit 231 outputs the acquired background candidate image to the background candidate image selection unit 232. At this time, when a level is provided in the search condition, the background candidate image acquisition unit 231 outputs the level to the background candidate image selection unit 232 as well.

The background candidate image selection unit 232 causes the photographer to select one image as a background image for synthesis from the plurality of background candidate images acquired by the background candidate image acquisition unit 231.
The background candidate image selection unit 232 causes the display unit 25 to display the acquired plurality of background candidate images on the screen, and outputs an image (background image) selected by the photographer to the synthesis unit 234.
For example, as illustrated in FIG. 6, the background candidate image selection unit 232 displays a plurality of background candidate images BG ₁ , BG ₂ , BG ₃ , BG ₄ ... On the screen of the display unit 25. At this time, if all the background candidate images cannot be displayed on the screen, the background candidate image selecting unit 232 may provide the scroll bar SB and switch the background candidate images to be displayed by the photographer's selection.

If the level is set in the search condition, the background candidate image selecting means 232 provides a level display area P indicating the level as shown in FIG. Present. For example, the background candidate image selection means 232 turns on green in the level display area P for the background candidate images that match the strictest search conditions, and in the case of search conditions that are looser than that, the background candidate images are sequentially displayed in yellow, red, etc. Present the level.
This level display allows the photographer to determine which background candidate image approximates the location where he / she photographed, the photographing direction of the image photographing device 2, the angle of view, and the like.
Returning to FIG. 5, the description of the configuration of the image capturing device 2 will be continued.

The foreground image cutout unit 233 cuts out the foreground from the photographed image.
Here, based on the key signal generated by the foreground area detection unit 22, the foreground image extraction unit 233 extracts the area designated as the foreground area from the captured image captured by the imaging unit 21. That is, the foreground image cutout unit 233 cuts out the foreground image FG from the photographed image PG based on the foreground area indicated by the key signal K as shown in FIG.
The foreground image cutout unit 233 outputs the cut out foreground image FG to the combining unit 234.

The synthesizing unit 234 synthesizes the foreground image extracted by the foreground image extracting unit 233 with the background image selected by the background candidate image selecting unit 232.
That is, as shown in FIG. 8, the synthesizing unit 234 actually overwrites the background image BG selected by the background candidate image selecting unit 232 with the foreground image FG extracted by the foreground image extracting unit 233. A synthesized image G in which the background is replaced with respect to the captured image is generated.
The synthesizing unit 234 writes the generated synthesized image G in the storage unit 26. At this time, the synthesizing unit 234 presents the synthesized image G written in the storage unit 26 to the photographer via the display unit 25.
Here, it is assumed that the synthesizing unit 234 writes the shooting information acquired by the shooting information acquiring unit 20 to the generated combined image G in the storage unit 26. Of course, the synthesizing unit 234 may separately acquire the shooting information corresponding to the background image used for the synthesis from the image providing apparatus 1 via the communication control unit 24 and store it together.

The communication control unit 24 transmits / receives data to / from the image providing apparatus 1 through communication. For example, the communication control unit 24 transmits and receives data through a wireless LAN such as Wi-Fi. Of course, the communication standard of the communication control unit 24 is the same as the communication standard of the communication control unit 11 (see FIG. 2) of the image providing apparatus 1.
Here, the communication control unit 24 transmits the data (background candidate image request) input from the foreground / background composition unit 23 to the image providing device 1 and uses the data (background candidate image) received from the image providing device 1 as the foreground / background. The result is output to the combining unit 23.
Here, when there is a background image registration request from the image registration unit 28 described later, the communication control unit 24 transmits the background image together with the background image to the image providing apparatus 1.

The display means 25 displays a plurality of background candidate images output from the foreground / background composition means 23 (background candidate image selection means 232) and accepts a photographer's selection.
The display means 25 includes a touch panel (not shown), and displays a plurality of background candidate images as shown in FIG. Then, the position on the screen selected by the photographer is output to the foreground / background composition means 23. Thereby, the foreground / background composition unit 23 can recognize which background candidate image has been selected.
The display means 25 is also for displaying the composite image synthesized by the synthesis means 234. Needless to say, the display means 25 has a function of monitoring a photographed image photographed by the photographing means 21, although the signal lines are not shown.

The storage means 26 stores a captured image in association with shooting information, and is a general recording medium such as an SD card.
The storage means 26 stores the composite image generated by the foreground / background composition means 23 together with the shooting information, and stores the photographed image that is not subjected to image composition by the image writing means 27 described later together with the shooting information.

When the background image provided by the image providing apparatus 1 is not synthesized, the image writing unit 27 writes the captured image and the shooting information at that time in association with each other in the storage unit 26.
That is, the image writing unit 27 functions when the switching unit T (T ₁ , T ₂ ) is set to the b terminal side, and the captured image captured by the capturing unit 21 and the captured information acquisition unit 20 are used. The acquired photographing information is associated with and written in the storage unit 26.

The image registration unit 28 transmits and registers the captured image and the captured information stored in the storage unit 26 to the image providing apparatus 1 via the communication control unit 24.
The image registration unit 28 registers the photographed image stored by the image writing unit 27 as a background candidate image with the switching unit T (T ₁ , T ₂ ) set to the b terminal side. Although not shown, the image registration unit 28 displays the captured image stored in the storage unit 26 on the display unit 25, and requests registration of shooting data including the background candidate image selected by the photographer and the shooting information. Is transmitted to the image providing apparatus 1.
As a result, the image providing apparatus 1 sequentially stores background candidate images taken at remote locations, and the database is enriched.

  The image capturing apparatus 2 described above includes a computer (not shown), and can be operated by a program that causes the computer to function as each unit described above. Specifically, the image capturing apparatus 2 can realize a function of synthesizing the foreground image and the background image by an image synthesis program that causes the computer to function as each unit of the foreground / background synthesis unit 23.

  By configuring the image capturing device 2 in this way, the image capturing device 2 can synthesize the foreground region of the captured image captured at the present time and the background image captured at different times. As a result, the image capturing apparatus 2 can synthesize a background image that has been captured in advance with good visibility even when the visibility is poor due to weather conditions or the like.

≪Operation of image composition system≫
Next, the operation of the image composition system S will be described with reference to FIGS. Here, the operation of the image capturing apparatus 2 will be mainly described.
[Image registration operation]
First, a background image registration operation to the image providing apparatus 1 by the image capturing apparatus 2 will be described with reference to FIG. During this operation, the switching unit T (T ₁ , T ₂ ) is set to the b terminal side.

First, the image photographing device 2 photographs a subject by the photographing means 21 at a timing when the shutter switch SW is pressed (step S10).
In addition, the image capturing apparatus 2 captures shooting information (position information, position information, position information, position, orientation, and posture of the image capturing apparatus 2 when the subject is captured by the capturing information acquisition unit 20 at the same timing as step S10. (Azimuth information, posture information, lens information) is acquired (step S11).

That is, the image capturing apparatus 2 acquires position information (latitude, longitude, altitude) detected by the position detection unit 201 by the position information acquisition unit 202.
Further, the image capturing apparatus 2 acquires the front direction of the image capturing apparatus 2 detected by the direction detecting unit 203 by the direction information acquiring unit 204.
Further, the image capturing apparatus 2 acquires posture information (rotation angles in the roll, pitch, and yaw directions) detected by the posture detection unit 205 by the posture information acquisition unit 206.
Further, the image capturing apparatus 2 acquires lens information (focal length, focus position, aperture value, etc.) of the image capturing unit 21 by the lens information acquiring unit 207.

  Then, the image photographing device 2 writes and stores the photographed image photographed in step S10 and the photographing information obtained in step S11 in the storage unit 26 in association with each other by the image writing unit 27 (step S12).

  Thereafter, the image photographing device 2 transmits image data including the photographed image and the photographing information stored in the storage unit 26 in step S12 to the image providing device 1 by the image registration unit 28 as a registration request. Register in the database in the providing device 1 (step S13).

  Through the above operation, the image capturing apparatus 2 registers the captured image as a background candidate image in the database in the image providing apparatus 1 in association with the captured information including the position, orientation, orientation, and lens information at the time of capturing. be able to.

(Image composition operation)
Next, with reference to FIG. 10 (refer to FIGS. 1 and 5 as appropriate for the configuration), the composition operation of the foreground image and the background image by the image capturing device 2 will be described. During this operation, the switching unit T (T ₁ , T ₂ ) is set to the a terminal side.

First, the image photographing device 2 photographs the subject by the photographing means 21 at the timing when the shutter switch SW is pressed (step S20).
Then, the image capturing device 2 generates a key signal indicating the foreground of the captured image captured in step S20 by the foreground area detection unit 22 (step S21).

  Further, the image capturing apparatus 2 captures image information (position information, position information, position information, position and orientation) of the image capturing apparatus 2 when the subject is captured by the capturing information acquisition unit 20 at the same timing as step S20. (Azimuth information, posture information, lens information) is acquired (step S22). The operation in step S22 is the same as the operation in step S11 in FIG.

Then, the image photographing device 2 cuts out the foreground image from the photographed image photographed in step S20 based on the key signal generated in step S21 by the foreground image clipping means 233 of the foreground / background composition means 23 (step S21). S23).
Further, the image capturing apparatus 2 provides a predetermined allowable range for the shooting information (position information, orientation information, posture information, lens information) acquired in step S22 by the background candidate image acquisition unit 231 of the foreground / background composition unit 23. Using the obtained information as a search condition, a background candidate image is acquired from the database of the image providing apparatus 1 (step S24).

Then, the image capturing apparatus 2 arranges a plurality of background candidate images acquired in step S24 on the display unit 25 by the background candidate image selecting unit 232 of the foreground / background synthesizing unit 23, and accepts selection of the background image from the photographer ( Step S25).
The image capturing apparatus 2 then combines the foreground image cut out in step S23 with the background image selected in step S25 by the combining unit 234 of the foreground / background combining unit 23 (step S26).

With the above operation, the image capturing apparatus 2 can synthesize the foreground image with the background image captured at different times, approximating the position, orientation, orientation, and lens information at the time of capturing.
The configuration and operation of the image composition system S according to the embodiment of the present invention have been described above, but the present invention is not limited to this embodiment.

(Modification 1)
For example, here, the image capturing apparatus 2 has a function of combining a foreground image at the time of capturing with a background image registered in the database of the image providing apparatus 1, and background image candidates are stored in the database of the image providing apparatus 1. Although it is configured as an apparatus having a function to register, it may be configured to have only one of the functions.
For example, if the function of registering the background image is unnecessary, the image capturing device 2 may be configured by omitting the image registration unit 28 from the configuration described in FIG. If the function for performing image composition is unnecessary, the image capturing apparatus 2 may be configured by omitting the foreground / background composition unit 23 and the foreground area detection unit 22 from the configuration described in FIG.

(Modification 2)
Here, the background candidate image is stored in the database stored in the image storage unit 10 of the image providing apparatus 1, but in addition to this, a thumbnail image obtained by reducing the image is also stored. It is good as well.
In that case, the image capturing apparatus 2 acquires a thumbnail image of a candidate background candidate image from the image providing apparatus 1 by the background candidate image acquisition unit 231. Then, when one background candidate image (thumbnail image) is selected by the background candidate image selection unit 232, the image photographing device 2 uses the background candidate image acquisition unit 231 to generate the original background candidate image from the image providing device 1. What is necessary is just to acquire an image. As a result, the communication amount and communication time between the image providing device 1 and the image capturing device 2 can be reduced.

(Modification 3)
In this example, the combining unit 234 of the image capturing apparatus 2 acquires the selected background candidate image from the image providing apparatus 1 and combines the selected background candidate image with the foreground image as it is as a background image. The background candidate image may be transformed and combined.
For example, as shown in FIG. 11 (a), when the angle of view AGa at the time of actual photographing is smaller than the angle of view AGb of the selected background candidate image, the synthesizing unit 234, as shown in FIG. 11 (b). Then, an image for the angle of view AGa is cut out from the background candidate image, and is converted (enlarged) as necessary to obtain a background image for synthesis.

Here, when the angle of view at the time of actual photographing is larger than the angle of view of the selected background image, the image desired by the photographer may not exist in the background image. Therefore, when the image is deformed as described above, it is preferable that the background candidate image acquisition unit 231 acquires a background candidate image having an angle of view larger than the angle of view at the time of actual photographing.
If the image desired by the photographer cannot be used as a background image depending on the angle of view, a message to that effect may be displayed on the display unit 25 as an error message.

  For example, as shown in FIG. 12A, when the posture information (here, the rotation angle φ in the roll direction) differs between the actual captured image and the background candidate image, As shown in b), the background candidate image BG is rotated by the rotation angle φ in the roll direction. Then, as shown in FIG. 12C, the synthesizing unit 234 cuts out an image corresponding to the angle of view of the actual captured image from the image rotated in FIG. 12B, and performs image conversion (enlargement) as necessary. By doing so, a background image for synthesis is obtained.

Further, when there is a difference in position between the actual captured image and the background candidate image, the synthesizing unit 234 may translate the background candidate image by the amount of the deviation to obtain a background image for synthesis.
Further, when there is a deviation in the orientation between the actual captured image and the background candidate image, the composition unit 234 may perform projective transformation on the background image region shifted by the orientation to obtain a background image for composition. Of course, if the position, orientation, and orientation are different, they may be combined and transformed.

  As described above, when the background image to be used is partially used depending on the position, orientation, and orientation, the image desired by the photographer may not exist in the background image depending on the size of the angle of view. Therefore, in Modification 3, it is desirable to register a large number of background candidate images with different shooting information including the angle of view in the database of the image providing apparatus 1.

(Modification 4)
In this example, the image capturing device 2 is a device that captures a still image. However, if the image capturing device 2 is fixedly installed, it may be a device that captures a moving image.
In this case, the foreground image cutout unit 233 cuts out the foreground area in units of frames, and the combining unit 234 may combine the foreground image and the background image in units of frames. The operation until the background image is selected is the same as that when a still image is taken.

S Image composition system 1 Image providing device 10 Image storage means (database)
DESCRIPTION OF SYMBOLS 11 Communication control means 12 Image registration search means 2 Image imaging device (image composition apparatus, camera)
20 photographing information acquisition means 201 position detection means (detection means)
202 Position information acquisition means 203 Orientation detection means (detection means)
204 Orientation information acquisition means 205 Attitude detection means (detection means)
206 Attitude information acquisition means 207 Lens information acquisition means 21 Imaging means 22 Foreground area detection means 23 Foreground / background composition means 231 Background candidate image acquisition means 232 Background candidate image selection means 233 Foreground image extraction means 234 Composition means 24 Communication control means 25 Display Means 26 Storage means 27 Image writing means 28 Image registration means

Claims (7)

An image synthesizing apparatus that shoots a subject and synthesizes an image taken at a different time on the foreground of the taken image as a background,
Photographing means for photographing the subject via an optical system;
Shooting information acquisition means for acquiring position information, azimuth information and posture information when shooting the subject as detection information through the detection means, and acquiring lens information of the optical system, as shooting information of the image composition device;
Foreground area detecting means for detecting a foreground area in the photographed image based on depth information in which a distance to the subject is associated with a pixel of the photographed image photographed by the photographing means;
Foreground image cutting means for cutting out the foreground area from the captured image to generate a foreground image;
From a database in which background photographing information including position information, orientation information, posture information, and lens information of a camera that has photographed a background candidate image serving as a background candidate is associated with the background candidate image in advance via a communication control unit. A background candidate image acquisition means for acquiring the background candidate image that matches a search condition that approximates the shooting information and the background shooting information within a predetermined range;
Background candidate image selection means for displaying the background candidate image acquired by the background candidate image acquisition means on the screen of the display means and receiving selection from the photographer;
A synthesis means for synthesizing the background candidate image selected by the background candidate image selection means as a background image with the foreground image ,
The background candidate image acquisition means acquires the background candidate image by providing a stepped level in a range that approximates the search condition,
The background candidate image selecting means, the image synthesizing apparatus according to claim you to view on the screen of the display unit the background candidate image in association with the level. An image synthesizing apparatus that shoots a subject and synthesizes an image taken at a different time on the foreground of the taken image as a background,
Photographing means for photographing the subject via an optical system;
Shooting information acquisition means for acquiring position information, azimuth information and posture information when shooting the subject as detection information through the detection means, and acquiring lens information of the optical system, as shooting information of the image composition device;
Foreground area detecting means for detecting a foreground area in the photographed image based on depth information in which a distance to the subject is associated with a pixel of the photographed image photographed by the photographing means;
Foreground image cutting means for cutting out the foreground area from the captured image to generate a foreground image;
A database in which the background candidate image is previously associated with background photographing information including position information, orientation information, posture information, and lens information of a camera that has been photographed in advance at a plurality of points and photographed as a background candidate. A background candidate image obtaining unit that obtains the background candidate image that matches the search condition in which the shooting information and the background shooting information are approximated within a predetermined range from the image providing apparatus including When,
Background candidate image selection means for displaying the background candidate image acquired by the background candidate image acquisition means on the screen of the display means and receiving selection from the photographer;
A combining means for combining the background candidate image selected by the background candidate image selecting means with the foreground image as a background image;
An image composition apparatus comprising: The foreground region detection means, an image synthesizing apparatus according to region of the captured image corresponding to the depth of in focus in the lens information, to claim 1 or claim 2, wherein the detecting as the foreground region . The foreground area detecting means, a region of the captured image corresponding to the depth of a predetermined range from the most front surface of the depth information, one of claims 1 to 3, characterized in that the detection as the foreground region image synthesizing apparatus according to an item or.   5. The composition unit according to claim 1, wherein the composition unit synthesizes an area corresponding to an angle of view specified by lens information of the photographing information in the selected background image with the foreground image. The image composition device according to any one of the above.   6. The synthesis apparatus according to claim 1, wherein the synthesizing unit synthesizes the background image by deforming the background image in accordance with a shift in position, orientation, or posture between the shooting information and the background shooting information. The image composition device according to any one of the above. The computer, image synthesis program for functioning as an image synthesizing device according to any one of claims 1 to 6.

Images