JP6685814B2 - Imaging device and control method thereof - Google Patents

Description

  The present invention relates to an image pickup apparatus, and more particularly to an image pickup apparatus for taking a composite image and a control method thereof.

  2. Description of the Related Art In recent years, an imaging device has been developed that can obtain a composite image that cannot be obtained only by a real image captured by a lens and an image sensor by combining captured images. Many of these image pickup apparatuses compare imaged image data and synthesize image data so that the size, inclination, etc. of a subject are matched between the image data.

  In addition, an apparatus that can display a composite image in real time has been developed by applying the technology of augmented reality. These devices calculate spatial coordinates within the live view image being captured. Then, the size, inclination, etc. of the synthesis target are adjusted so as to be consistent with the spatial coordinates in the live view image, and the adjusted synthesis target is synthesized with the live view image.

  Patent Document 1 discloses an imaging device capable of changing the composition (background) of a subject. The imaging device of Patent Document 1 displays a subject to be combined on a live view image. The user determines the composition of the subject while viewing the live view image, and captures the composite image.

  Patent Document 2 discloses an image display device capable of maintaining consistency between an object in the real world and an object in the virtual world when experiencing augmented reality. The image display device of Patent Document 2 aligns the image data of the virtual object with the captured image data so that the geometrical positional relationship is consistent.

JP, 2014-93732, A JP, 2005-7722, A

In the imaging device of Patent Document 1, the entire image of the subject is displayed on the live view image, and the direction and the angle of view of the live view image are changed according to the displayed subject. In the image display device of Patent Document 2, the visual angle of the user and the angle of view of the image pickup device are matched, and the virtual object is combined with the image of the real space within the angle of view of the image pickup device. In the devices of Patent Document 1 and Patent Document 2, it is premised that the object image to be combined is arranged within the angle of view of the imaging device, and it is assumed that the object image is arranged outside the angle of view. It has not been.
In the augmented reality technology, since the coordinates are set only in the captured image that has been captured, if the object image is arranged outside the captured image, it becomes difficult to correctly calculate the position of the object image. Therefore, it may be difficult to adjust the arranged object image to an appropriate size.
SUMMARY OF THE INVENTION In view of the above problems, it is an object of the present invention to provide an image pickup apparatus capable of appropriately synthesizing an object image arranged outside the angle of view of the image pickup apparatus and a control method thereof.

  An image pickup apparatus according to the present invention includes an image pickup section that picks up an image pickup space through an optical system, an image generation section that generates a picked-up image based on a signal from the image pickup section, a position of the image pickup section, and a posture of the image pickup section. And a measurement unit that acquires measurement information including a distance from the image capturing unit to a subject in the image capturing space, and real coordinates that represent the image capturing space and a space outside the image capturing space, from the captured image and the measurement information. A coordinate generation unit for designating, a designating unit for designating a synthesis position of an object image to be synthesized with the captured image, and processing the object image according to the synthesis position based on the actual coordinates, and the object image is And an image processing unit for synthesizing at a synthesizing position.

  A control method of an image pickup apparatus according to the present invention includes a step of, in an image pickup unit, capturing an image of an image pickup space through an optical system; a step of generating a picked-up image based on a signal from the image pickup unit; a position of the image pickup unit; A step of obtaining measurement information including a posture of a section and a distance from the image pickup section to a subject in the image pickup space; and an actual image representing the image pickup space and a space outside the image pickup space from the picked-up image and the measurement information. A step of generating coordinates, a step of designating a combination position of an object image to be combined with the captured image, a step of processing the object image according to the combination position based on the actual coordinates, And a step of combining at a combination position.

  According to the present invention, by generating real coordinates representing the imaging space being imaged through the lens and the space outside the imaging space, and processing the composition target according to the composition position in the real coordinates, it is possible to move the image outside the angle of view of the imaging device. The arranged object images can be appropriately combined.

It is a block diagram showing the composition of the imaging device concerning a 1st embodiment of the present invention. It is a figure for demonstrating the imaging coordinate which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the real coordinate which made the imaging device which concerns on 1st Embodiment of this invention a base point. It is a figure which shows the image data containing the object image which concerns on 1st Embodiment of this invention. It is a figure which shows the object image which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the arrangement method of the object image which concerns on 1st Embodiment of this invention. It is a figure for demonstrating the arrangement method of the object image which concerns on 1st Embodiment of this invention. It is a figure which shows the positional relationship of the imaging device and object image which concern on 1st Embodiment of this invention. It is a figure which shows the positional relationship of the imaging device and object image which concern on 1st Embodiment of this invention. 3 is a flowchart showing a control method of the image pickup apparatus according to the first embodiment of the present invention. 3 is a flowchart showing a control method of the image pickup apparatus according to the first embodiment of the present invention. It is a figure which shows the position which can arrange the object image in an imaging coordinate.

[First Embodiment]
FIG. 1 is a block diagram showing the configuration of the image pickup apparatus 100 according to this embodiment. The imaging device 100 includes a lens unit 101, an image sensor unit 102, an A / D conversion unit 103, a signal processing unit 104, a memory 105, an image generation unit 106, a data synthesis unit 107, a display unit 108, and a CPU 109. The image pickup apparatus 100 further includes a GPS sensor 110, a height sensor 111, a tilt sensor 112, a position measurement unit 113, a distance sensor 114, a distance measurement unit 115, a data processing unit 116, an extraction unit 117, a storage unit 118, and a selection unit 119. The imaging control unit 120 is provided.

  The lens unit 101 is an optical system including a focus lens, a zoom lens, a diaphragm, and the like. The lens unit 101 has a predetermined angle of view (viewing angle) α, and forms light from the imaging space within the angle of view on the light receiving surface of the image sensor unit 102. The lens unit 101 can adjust the focus of the focus lens by moving the focus lens along the optical axis of the lens with an actuator (not shown). Similarly, the lens unit 101 can adjust a zoom lens, a diaphragm, and the like. The image sensor unit (imaging unit) 102 includes a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), and the like. The image sensor unit 102 converts the light received from the lens unit 101 into an electric signal and outputs it as an analog image pickup signal. For the sake of simplicity, the angle of view of the image sensor unit 102 is equal to the angle of view α of the lens unit 101.

  The A / D (Analog / Digital) conversion unit 103 converts the analog image pickup signal input from the image sensor unit 102 into digital image pickup data. For example, the signal processing unit 104 performs image processing such as white balance adjustment, interpolation, contour enhancement, gamma correction, gradation conversion, and distortion correction on the digital image pickup data, and stores the processed image pickup data in the memory 105. Let The A / D conversion unit 103 may be provided in the image sensor unit 102.

  The memory 105 includes, for example, a DRAM (Dynamic Random Access Memory), a flash memory, and the like, and stores a control program for operating the imaging device 100. The memory 105 is used as a temporary storage area for various data such as image pickup data and also as a work area when each unit of the image pickup apparatus 100 performs calculation. A CPU (Central Processing Unit) 109 is a processor that controls the entire imaging device 100, and realizes the functions of the respective units of the imaging device 100 by reading and executing a predetermined program from the memory 105.

  The image generation unit 106 acquires captured image data from the signal processing unit 104 or the memory 105 and generates a captured image for display. For example, the image generation unit 106 generates a captured image according to the resolution of the screen of the display unit 108 and outputs it to the data synthesis unit 107. In addition, the image generation unit 106 uses the acquired image pickup data as MPEG-4, H.264, H.264, H.264, and the like. It is compressed according to the format of a moving image file such as H.264 and is stored in the storage unit 118 via the data synthesis unit 107. Further, the image generation unit 106 acquires captured image data from the signal processing unit 104 or the memory 105 and generates a captured image (still image). For example, the image generation unit 106 compresses the captured data according to the format of a still image file such as JPEG.

  The data synthesizing unit (image processing unit) 107 superimposes (synthesizes) the object image from the data processing unit 116 on the captured image. The data composition unit 107 outputs the captured image input from the image generation unit 106 or the captured image in which the object image is composited to the display unit 108 and the storage unit 118.

  The display unit (designating unit) 108 includes a liquid crystal display, an organic light emitting device, and the like, and a captured image input from the data combining unit 107, various data stored in the memory 105, and a menu screen for operating the image capturing apparatus 100. Is displayed. Further, a transmissive touch sensor is provided on the screen of the display unit 108. By touching an arbitrary position of the captured image displayed on the screen with a finger or the like, the user can specify the position where the object image is arranged, the position where the focus point is set, and the like. In addition, the user can arrange a part of the object image outside the angle of view of the captured image by dragging the object image to the outside of the screen. According to the present embodiment, by setting the real coordinates capable of calculating the position outside the angle of view of the captured image, even when a part of the object image is arranged outside the angle of view of the captured image, It can be appropriately combined with the captured image.

  A GPS (Global Positioning System) sensor 110 receives radio waves from a plurality of GPS satellites and measures the coordinates (latitude, longitude, altitude) of the imaging device 100. The height sensor 111 is composed of an infrared sensor, an ultrasonic sensor, etc., and the tilt sensor 112 is composed of an acceleration sensor, a gyro sensor, etc. The image pickup apparatus 100 may further include a direction sensor and the like.

  The position measuring unit 113 measures the coordinates (latitude, longitude, altitude) of the imaging device 100 and the height from the ground based on the signals from the GPS sensor 110 and the height sensor 111. In addition, the position measuring unit 113 acquires a forward tilt / backward tilt angle (elevation depression angle) and a left / right tilt (roll angle) of the imaging device 100 based on a signal from the tilt sensor 112.

  The distance sensor 114 is composed of an infrared sensor, an ultrasonic sensor, etc., irradiates an object with infrared rays and ultrasonic waves, and receives a reflected wave thereof. The distance measuring unit 115 measures the distance based on the time from when the distance sensor 114 irradiates infrared rays or ultrasonic waves to when the reflected wave is received. That is, the distance measuring unit 115 can measure the distance from the imaging device 100 to the subject.

  The data processing unit (coordinate generation unit) 116 analyzes the picked-up image from the image generation unit 106 and generates picked-up coordinates (X ′, Y ′, Z ′) (see FIG. 2). The imaging coordinates are three-dimensional coordinates that represent the imaging space captured by the lens unit 101 and the image sensor unit 102 as an image, and have coordinate values corresponding to each pixel of the captured image. The imaging coordinates can be calculated, for example, based on the geometrical features included in the captured image. Specifically, an augmented reality technique such as a SfM (Structure from Motion) method and a SLAM (Simultaneous Localization And Mapping) method can be applied. In addition, in order to improve the accuracy of the imaging coordinates, lens information such as the focal length of the lens unit 101 and the angle of view, the position and orientation of the imaging device 100 acquired from the position measuring unit 113, and the object acquired from the distance measuring unit 115 to the object. Measurement information such as distance may be used.

  Further, the data processing unit 116 calculates three-dimensional real coordinates (X, Y, Z) centered on the imaging device 100 based on the measurement information from the position measuring unit 113 and the distance measuring unit 115. The real coordinates are coordinates in a real three-dimensional space, and the distance between any two points can be calculated based on the real coordinates. The data processing unit 116 associates each coordinate value of the imaging coordinates with the coordinate value of the actual coordinates and sets the imaging coordinates in the actual coordinates (see FIG. 3). That is, the space represented by the real coordinates is a space obtained by expanding the imaging space represented by the imaging coordinates.

  Further, the data processing unit (image processing unit) 116 performs processing processing such as size change and deformation on the object image. Specifically, the data processing unit 116 performs processing such as rotation, roll, deformation, enlargement, reduction, etc. on the object image according to the arrangement designated by the user, and determines the size, inclination, etc. of the object image as a captured image. Align. The data processing unit 116 can also perform the blurring process on the object image according to the focus point (the focused area) of the captured image. For example, when the object image is arranged at a position (background or the like) that is not the focus point of the captured image, the data processing unit 116 uses a spatial filter such as a Gaussian filter to add blurring of intensity according to the focus shift. To do.

  In addition, when the processed object image is outside the captured image, that is, when a part of the object image is arranged outside the image capturing coordinates, the data processing unit 116 extracts only the part at the image capturing coordinates from the object image. Trimming. In other words, the data processing unit 116 trims only the area corresponding to the angle of view α of the lens unit 101 from the object image. In this way, the data processing unit 116 constitutes a coordinate generation unit and an image processing unit.

  The extraction unit 117 reads out the image data from the storage unit 118 and extracts the object image designated by the user. The extraction unit 117 may determine a main subject from the image data and extract it as an object image. In addition, the extraction unit 117 calculates the actual size such as the actual height, width, and depth of the object image based on the lens information and the measurement information at the time of imaging included in the read image data. The actual size of the object image may be stored in the storage unit 118 in association with the image data (captured image) at the time of capturing. The extraction unit 117 outputs the object image and its actual size to the data processing unit 116.

  The storage unit 118 is a storage device such as a hard disk, and stores various image data such as a captured image generated by the imaging device 100. The storage unit 118 includes a removable storage medium such as a removable memory card, and can store image data imaged or generated by a device other than the imaging device 100.

  The selection unit 119 includes input devices such as operation keys and an enter button, and selects image data including an object image from the image data stored in the storage unit 118. For example, the selection unit 119 causes the display unit 108 to display a list of image data stored in the storage unit 118. The object image is not limited to the one included in the captured image, but may be a generated image such as a character created by CG (Computer Graphics) as long as the actual size of the object is defined. Is also good. Further, the selection unit 119 may be configured as an operation key on the display displayed on the display unit 108 or an enter button instead of an actual button.

  The imaging control unit 120 controls the focus lens, the zoom lens, the diaphragm, and the like of the lens unit 101 to adjust the focus, change the zoom magnification, and adjust the brightness. The imaging control unit 120 also acquires information such as the focus position, subject distance, and angle of view regarding the image being captured from the lens unit 101, and outputs the information to the data processing unit 116. Further, the imaging control unit 120 instructs the signal processing unit 104 when to acquire the image captured by the lens unit 101 and the image sensor unit 102 as a still image.

  FIG. 2 is a diagram for explaining imaging coordinates according to the present embodiment. FIG. 2 shows an example of the captured image 200 generated by the image generation unit 106, and the captured image 200 has image capturing coordinates (X ′) in an orthogonal coordinate system composed of an X ′ axis, a Y ′ axis, and a Z ′ axis. , Y ′, Z ′) is generated. In the imaging coordinates shown in FIG. 2, the vicinity of the feet of the person in front is set as the origin, the plane including the ground (floor surface) is set as the X'Y 'plane, and the vertical direction (height direction) is set as the Z'axis. . Each pixel of the captured image 200 is represented by imaging coordinates. It should be noted that the coordinate axes of the image capturing coordinates are not limited to the example of FIG. 2, and are appropriately set according to the composition of the captured image 200.

  FIG. 3 is a diagram for explaining actual coordinates with the image pickup apparatus 100 according to the present embodiment as a base point. As shown in FIG. 3, real coordinates (X, Y, Z) of a Cartesian coordinate system composed of X-axis, Y-axis, and Z-axis are generated, and imaging coordinates (X ′, Y ′, Z) are included in the real coordinates. ') Is set. For example, each coordinate axis of the imaging coordinates matches the direction of each coordinate axis of the actual coordinates, and each coordinate value of the imaging coordinates is associated with the coordinate value of the actual coordinates. The space represented by the imaging coordinates matches the space captured by the lens unit 101 as an image, that is, the space within the angle of view of the lens unit 101, and is included in the space represented by the actual coordinates. . In the actual coordinates shown in FIG. 3, the position of the imaging device 100 is set as the origin, and the X axis and the Y axis that are orthogonal to each other in a horizontal plane including the ground, and the vertical direction (height direction) to the ground are set as the Z axis. . The coordinate axes of the real coordinates are not limited to the example of FIG. 3, and are appropriately set according to the position of the image pickup apparatus 100, the image pickup direction, and the like. Each coordinate value of the actual coordinates is represented by a combination of latitude, longitude, and altitude, for example.

  FIG. 4 is a diagram showing the image data 300 including the object image 301 according to this embodiment. The image data 300 shown in FIG. 4 is selected by the user in the selection unit 119 from the captured images stored in the storage unit 118. The image data 300 includes an object image 301 to be combined. The object image 301 is designated by the user on the display unit 108, for example.

  FIG. 5 is a diagram showing an object image 301 according to this embodiment. The object image 301 of FIG. 5 is extracted by the extraction unit 117 from the selected image data 300. In the object image 301, a reference point 301a, which serves as a reference when arranging the object image 301, is set. The reference point 301a is set at a position such as a foot and a center of gravity of the object image 301, for example. The reference point 301a may be designated by the user on the display unit 108, or may be calculated by the extraction unit 117 based on the shape of the object image 301.

  FIG. 6 is a diagram for explaining a method of arranging the object image 301 according to this embodiment. FIG. 6 is an example displayed on the display unit 108, and shows the object image 301 combined with the captured image 200. When the user designates an arbitrary position on the captured image 200 with the finger, the object image 301 is arranged at the designated position on the captured image 200. Here, for example, when the reference point 301a is set at the foot of the object image 301, it is assumed that the user arranges the object image 301 on the ground and floor of the captured image 200. When the object image 301 is arranged, the actual composite position where the object image 301 is arranged is calculated from the imaging device 100 based on the actual coordinates corresponding to the reference point 301a. Then, the processing process is performed on the object image 301 according to the actual composite position of the object image 301. For example, the object image 301 is enlarged or reduced based on the actual size (for example, height) of the object image 301.

  FIG. 7 is a diagram for explaining a method of arranging the object image 301 according to this embodiment. In FIG. 7, the position of the object image 301 once arranged in the captured image 200 is changed by a user operation. Specifically, the user moves the object image 301 to a position in front of the captured image 200 by dragging the object image 301 arranged near the center of the captured image 200 with a finger as shown in FIG. . As shown in FIG. 7, the user arranges the object image 301 even when a part of the object image 301 protrudes from the captured image 200, particularly when the reference point 301 a is located outside the captured image 200. You can While the object image 301 is being moved by dragging, processing such as resizing and deformation of the object image 301 is performed in a predetermined cycle. In the example of FIG. 7, since the object image 301 is brought closer to the image capturing apparatus 100, the size of the object image 301 is enlarged.

  FIG. 8 is a diagram showing a positional relationship between the image capturing apparatus 100 and the object image 301 according to this embodiment. FIG. 8 shows the arrangement of the object image 301 corresponding to FIG. 6 on the real coordinates (X, Y, Z), and shows the real coordinates viewed from the direction horizontal to the ground (X-axis direction). The YZ plane is shown. The imaging device 100 is located at the height Hs, and images the imaging space within the angle of view α through the lens unit 101. The object image 301 has a height (actual size) H and is arranged within the angle of view α of the lens unit 101. In the object image 301, the focus point F is set at the position of height Hf. Based on the actual coordinates of the focus point F of the imaging device 100 and the object image 301, the actual distance L from the imaging device 100 to the object image 301 is calculated. When the focus point F is not set on the object image 301, the actual distance from the imaging device 100 to the composite position of the object image 301 is calculated based on the reference point 301a of the object image 301, for example. The actual distance to this combined position may be regarded as the actual distance L from the imaging device 100 to the object image 301.

  FIG. 9 is a diagram showing a positional relationship between the image capturing apparatus 100 and the object image 301 according to this embodiment. FIG. 9 shows the arrangement of the object image 301 corresponding to FIG. 7 on the real coordinates (X, Y, Z), and shows the real coordinates viewed from the direction horizontal to the ground (X axis direction). The YZ plane is shown. Similar to the example of FIG. 8, the imaging device 100 is located at the height Hs and images the imaging space within the angle of view α through the lens unit 101. The object image 301 has a height (actual size) H, and the object image 301 has a focus point F set at a position of height Hf. In FIG. 9, the object image 301 is arranged outside the angle of view α of the lens unit 101. Here, when the reference point 301a is located outside the angle of view α of the lens unit 101, it is expressed that the object image 301 is located outside the angle of view α. Even when the object image 301 is arranged outside the angle of view α of the lens unit 101, the actual distance L to the object image 301 can be calculated based on the actual coordinates, as in the example of FIG.

  FIG. 10 is a flowchart showing a control method of the image pickup apparatus 100 according to this embodiment. The process of this flowchart is repeatedly performed while the image capturing control unit 120 is not interrupting the signal processing unit 104 and while the image sensor unit 102 is capturing an image of the image capturing space.

  The imaging apparatus 100 starts the processing of the flowchart when the user operates to start imaging. First, the image sensor unit 102 images the imaging space within the angle of view α of the lens unit 101 (step S101). Next, the A / D converter 103 converts the analog image pickup signal from the image sensor unit 102 into digital image pickup data. The signal processing unit 104 performs image processing on the captured data, and the image generation unit 106 generates the captured image 200 from the captured data (step S102). The image generation unit 106 outputs the captured image 200 to the data processing unit 116 and also transmits it to the display unit 108 via the data synthesis unit 107. The display unit 108 displays the transmitted captured image 200 (step S103).

  Next, the data processing unit 116 receives the captured image 200 from the image generation unit 106 and acquires distance information regarding the captured image 200 from the distance measurement unit 115. Further, the data processing unit 116 acquires measurement information from the position measuring unit 113 (step S104). Subsequently, the data processing unit 116 generates real coordinates (X, Y, Z) including the image pickup coordinates (X ', Y', Z ') (step S105). That is, the data processing unit 116 analyzes the captured image 200, generates imaging coordinates, calculates real coordinates centered on the imaging device 100 using the measurement information, and sets the imaging coordinates in the real coordinates. .

  Next, the data processing unit 116 determines whether the object image 301 is arranged in the captured image 200 (step S106). Specifically, the data processing unit 116 determines whether or not the position where the object image 301 is to be combined is designated with respect to the captured image 200 displayed on the display unit 108. When the object image 301 is not arranged (NO in step S106), the data processing unit 116 shifts the process of the flowchart to step S101.

  When the object image 301 is arranged (YES in step S106), the data processing unit 116 calculates the actual distance to the object image 301 (step S107). That is, the data processing unit 116 uses the generated actual coordinates to calculate the actual distance from the lens unit 101 of the imaging device 100 to the combined position where the object image 301 is arranged or the focus point of the object image 301.

  Next, the data processing unit 116 determines whether the actual distance to the object image 301 has changed (step S108). That is, the data processing unit 116 compares the actual distance to the object image 301 calculated last time with the actual distance to the object image 301 calculated this time. When the actual distance to the object image 301 has not changed (NO in step S108), the data processing unit 116 shifts the process of the flowchart to step S110.

  When the actual distance to the object image 301 has changed, or when the actual distance to the object image 301 is calculated for the first time (YES in step S108), the data processing unit 116 performs the processing process on the object image 301. (Step S109). Specifically, the data processing unit 116, based on the generated real coordinates, the real size of the object image 301, and the real distance to the object image 301, the object image 301 is an object having a size, shape, and inclination suitable for the captured image 200. Convert to image. The data processing unit 116 is included in the captured image 200, that is, the region projected from the image sensor unit 102 in the region of the converted object image 301 based on the actual distance to the object image 301 and the angle of view α of the lens unit 101. Calculate the area. The data processing unit 116 trims the area calculated from the object image 301 and outputs it to the data synthesizing unit 107.

  The data combination unit 107 combines the object image 301 with the captured image 200, and the display unit 108 displays the combined image with the object image 301 combined (step S110). Finally, the imaging device 100 determines whether or not the user has instructed to stop imaging (step S111). If there is no instruction to stop imaging (NO in step S111), the imaging apparatus 100 moves the process of the flowchart to step S101. If there is an instruction to stop imaging (YES in step S111), the imaging apparatus 100 ends the process of the flowchart.

  FIG. 11 is a flowchart showing a control method of the image pickup apparatus 100 according to this embodiment. The process of the flowchart of FIG. 11 is a process for saving the combined image in which the object image 301 is combined as a still image, and is executed as an interrupt process to the process of the flowchart of FIG. 10. Therefore, when the process of the flowchart of FIG. 11 is started, the image capturing coordinates and the actual coordinates including the image capturing apparatus 100 have already been generated.

  First, when the user performs a still image capturing start operation, the image capturing control unit 120 notifies the image generating unit 106 of a still image generation instruction (step S201). The image generation unit 106 acquires the imaging data from the signal processing unit 104 at the timing of the notification from the imaging control unit 120 (step S202). The image generation unit 106 converts the acquired captured image data and generates a still image (captured image) according to the resolution of the image sensor unit 102 (step S203). The image generation unit 106 outputs the generated still image to the data processing unit 116.

  The data processing unit 116 converts the object image 301 acquired from the extraction unit 117 according to the resolution of the still image and performs the same processing as step S109 of FIG. 10 (step S204). Then, the data synthesizing unit 107 synthesizes the processed object image 301 with the still image from the image generating unit 106. The data synthesizing unit 107 stores the synthesized image in which the object image 301 is synthesized in the storage unit 118 (step S205).

  As described above, according to the present embodiment, by calculating the real coordinates around the image pickup device and setting the image pickup coordinates within the range of the real coordinates, a space outside the angle of view of the lens unit of the image pickup device can be obtained. A wider coordinate space is defined that contains. As a result, it becomes possible to arrange the object images to be combined beyond the range of the image pickup coordinates.

  FIG. 12 shows positions where the object image can be arranged in the image pickup coordinates. As described above, when the object image arranged so as to fit in the image pickup coordinates is brought closer to the image pickup apparatus, the size of the object image displayed on the display unit becomes larger as the arrangement position of the object image gets closer to the image pickup apparatus. In the conventional image pickup apparatus having the combining function, when the object image is moved beyond the image pickup coordinates, the distance to the object image cannot be determined, which makes it difficult to appropriately perform the combining process. On the other hand, in the present embodiment, the imaging coordinates obtained through the lens and the image sensor are expanded to the actual coordinates with the imaging device itself as a base point, so that the object image placed at a position that cannot be recognized in the captured image has an appropriate shape. Can be synthesized with. Therefore, it is possible to obtain a composite image in which the object image is arranged outside the angle of view of the captured image, and only part of it is reflected in the captured image. Further, by setting the focus point on the object image, it is possible to capture an image in which the object image is in focus.

[Modified Embodiment]
The present invention is not limited to the above embodiment, and various modifications can be made. For example, in the process of the flowchart shown in FIG. 10, the actual distance to the object image 301 is calculated, but the inclination of the object image 301 with respect to the ground may be calculated. In addition, when the object image 301 is combined, image processing is performed such that the object image 301 is the foreground or the background based on the positional relationship between the combined position of the object image 301 and each subject in the captured image 200. Is also possible.

100 image pickup apparatus 101 lens section 102 image sensor section 103 A / D conversion section 104 signal processing section 106 image generation section 107 data synthesis section 108 display section 113 position measurement section 115 distance measurement section 116 data processing section 200 captured image 301 object image

Claims (15)

An image generating unit that generates a captured image based on a signal from an image capturing unit that captures an image capturing space through an optical system,
Measuring means for acquiring the measurement information including the distance to the subject in the imaging space position and orientation of the imaging unit, and the imaging unit of the imaging means,
Coordinate space generation means for generating an actual coordinate space representing the imaging space and a space outside the imaging space from the captured image and the measurement information;
By specifying the position on the captured image, and designation means for designating a combining position of the object image to be combined with the captured image,
The specified combination position in the designation unit, thereby processing the object image in accordance with a position in the real coordinate space, and an image processing means for synthesizing the object image in the combined position in the captured image,
Wherein the image processing means, the processing corresponding to the specified combination position in the designation unit image processing apparatus characterized that you synthesis in the captured image by performing on the object image. A display control unit that displays the composite image generated by the image processing unit on a display unit;
The image processing apparatus according to claim 1, wherein the designation unit designates the composite position by designating a position on the captured image displayed on the display unit. The image processing apparatus according to claim 2, wherein the designation unit includes a touch sensor provided on the display unit. The said designation | designated means changes the said synthetic | combination position of the said object image by accepting the operation which drags the object image synthesize | combined on the said captured image, The any one of Claim 1 thru | or 3 characterized by the above-mentioned. Image processing device. The image processing unit according to claim 1, wherein the image processing unit performs a processing process on the object image at a predetermined cycle in response to a drag operation received by the designating unit. Processing equipment. The image processing apparatus according to claim 1, wherein the object image is stored in a memory in association with size information of the object image. The image processing apparatus according to claim 1, wherein the object image is an image of a subject extracted from image data. The image processing means calculates an actual distance from the image pickup means to the composite position using the actual coordinate space , and adjusts the size of the object image according to the actual distance and the actual size of the object image. the image processing apparatus according to any one of claims 1 to 7, characterized in that. The image processing apparatus according to claim 1, further comprising the image pickup unit. The image processing apparatus according to claim 9 , wherein the imaging unit adjusts the focus of the optical system to the actual distance to the object image when a focus point is set on the object image. 11. The image processing means adds a blur to the object image according to a focus shift of the optical system with respect to an actual distance to the object image, according to any one of claims 1 to 10. Image processing device. Wherein the image processing means, the image processing apparatus according to any one of claims 1 to 11 trims an area falling within the angle of view of the optical system from the object image. The coordinate space generation means, image processing apparatus according to any one of claims 1 to 12, characterized in that generating the real coordinate space as a base point of the imaging means. The measurement information, latitude indicating the position of the imaging means, longitude, image processing apparatus according to any one of claims 1 to 13, characterized in that it comprises a high degree of information. A step of generating a picked-up image obtained by picking up an image pickup space through an optical system by the image pickup means ;
Acquiring measurement information including the distance to the subject in the imaging space position and orientation of the imaging unit, and the imaging unit of the imaging means,
Generating a real coordinate space representing the imaging space and a space outside the imaging space from the captured image and the measurement information;
Specifying a position on the captured image to specify a composite position of an object image to be combined with the captured image,
A step of synthesizing the object image according to the position in the real coordinate space of the synthesis position designated in the step of designating, and synthesizing the object image at the synthesis position in the captured image ,
Step, a method of controlling an image processing apparatus characterized by comprising the steps you combining said captured image processing corresponding to the changed combination position in the step of specifying go to the object image to the composite.

Images