JP5691632B2 - Image composition method - Google Patents

Description

  The present invention relates to an image composition method for compositing a captured real image and a generated virtual image.

  As an image composition method for synthesizing a captured real image and a generated virtual image, for example, an additional image such as a virtual image associated with a specific subject included in the captured captured image is set at the position of the specific subject. Accordingly, an image composition method is known in which a composite image is generated by being arranged on a captured image (see, for example, Patent Document 1).

JP 2010-187052 A

  In the above conventional image composition method, in the composite image obtained by combining the real image and the virtual image, the real object on which the virtual image is superimposed cannot be seen because it is hidden by the virtual image. There is a problem that it is not possible to confirm the change.

  The present invention has been made in view of such a problem, and an object of the present invention is to confirm an actual change in a real part on a composite image in a composite image in which a virtual image is combined with a real image. It is an object of the present invention to provide an image synthesis method capable of performing the above-described process.

In order to achieve this object, the image composition method of the present invention includes:
A step of preliminarily storing an image obtained by photographing a state inside the wall as a virtual image;
Said wall virtual image includes a first image corresponding object associated, by shooting at shooting ranges, to generate a first combined images by combining the real images captured the wall and the virtual image A first image composition step;
When a writing instrument as a second image-corresponding subject in which a composite element that can be combined with the first composite image is associated with the first composite image is detected within the photographing range, the composite element is combined with the first composite image and the first composite image is combined. A second image composition step for generating two composite images;
When the writing instrument is moved, changing the second composite image based on the trajectory that the writing instrument moves, and bringing the writing instrument into contact with the wall to actually mark the wall;
An image composition method characterized by comprising:

In this image composition method, it is preferable that the composition element associated with the second image-corresponding subject is a second virtual image showing the writing instrument provided with the second image-corresponding subject.

In this image composition method, it is desirable that the composition element associated with the second image corresponding subject is an actual image of the writing instrument provided with the second image corresponding subject.

In this image composition method, the composition element associated with the second image-corresponding subject is a region for removing the virtual image at a position associated with the writing instrument on which the second image-corresponding subject is provided. It is desirable that the figure be shown.

In this image composition method, the composition element associated with the second image-corresponding subject is a figure indicating a region in which the virtual image is removed from the shape of the writing instrument on which the second image-corresponding subject is provided. It is desirable to be.

  ADVANTAGE OF THE INVENTION According to this invention, it is possible to provide the image composition method which can confirm the actual change in a real part on a composite image in the composite image by which the virtual image was synthesize | combined with the real image.

It is a block diagram which shows an example of the apparatus for implement | achieving the image composition method which concerns on this invention. It is a flowchart for demonstrating the image composition method of this embodiment. FIG. 3A is a diagram illustrating the state of the first shooting range, and FIG. 3B is a diagram illustrating the state of the shooting range in which the writing instrument is moved within the shooting range, and FIG. These are figures which show the state after a mark is provided. 4A is a diagram showing the first composite image, FIG. 4B is a diagram showing the second composite image, and FIG. 4C is a diagram showing the tip of the writing instrument on the second composite image. It is a figure which shows the state which added the orbit. It is a figure which shows the example made into the figure which shows the area | region which removes a rebar virtual image by matching a synthetic | combination element with the track | orbit of the front-end | tip of a writing instrument.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
In the image composition method of the embodiment, a photographing range including an image corresponding subject to which information of a virtual image generated in advance is associated is photographed by an image composition device, and a virtual image based on information of the photographed image corresponding subject is captured. And a photographed real image are combined to generate a first composite image, and a second image-corresponding subject in which a composite element that can be combined with the first composite image is associated with the first composite image within the photographing range of the first composite image. When detected, the composite element is combined with the first composite image to generate a second composite image.

  This embodiment demonstrates the example implemented when forming a through-hole in the wall by which the reinforcing bar was laid inside, for example. Specifically, a photograph of the state of the reinforcement inside the wall at the time of the reinforcement work for the wall to be constructed is stored in advance as a rebar virtual image, and an actual image obtained by continuously photographing the wall to be constructed A first composite image obtained by synthesizing the virtual reinforcing bar image is generated for each photographing, and when a writing instrument as a movable member such as an oil pen is moved in the composite image, a composite element associated with the writing instrument is used. An example of generating a second composite image in which a pen-shaped virtual image having a pen shape is combined with the first composite image in correspondence with the movement of the writing tool will be described.

FIG. 1 is a block diagram showing an example of an apparatus for realizing an image composition method according to this embodiment.
An image composition apparatus 1 for realizing the image composition method of the present embodiment includes an image composition unit 10 that photographs a subject, and a process that combines a real image of the photographed subject and a pre-formed virtual image. A control unit 12 that controls the apparatus 1, a storage unit 14 that stores various information and data necessary for combining images, a display unit 16 that displays the combined image, and the present image combining apparatus 1. And an operation unit 18 for operation.

  The image composition device 1 of the present embodiment has, for example, an imaging unit 10 and a display unit 16 that displays a captured image, and is portable, for example, a digital camera, a digital video camera, a mobile phone, a personal digital assistant (PDA). ), A program capable of executing image composition processing is stored in a mobile computer or the like.

  Then, when the user of the image composition device 1 photographs the subject, the captured actual image is displayed on the display unit 16. At this time, the photographed real image 20 (FIG. 4) includes a marker 21 as a subject corresponding to an image indicating predetermined composite information such as a QR code (registered trademark) or a barcode set in advance. If it is, the composite image obtained by detecting the marker 21 and acquiring composite information, and combining the captured real image 20 and the pre-stored virtual image 23 (FIG. 4) based on the acquired composite information. 24 and 25 (FIG. 4) are generated, and the generated combined images 24 and 25 are configured to be displayed on the display unit 16.

  In the first embodiment, two types of markers 21 are provided. Hereinafter, for convenience of description, the two types of markers will be described separately as an A marker 21a and a B marker 21b. One of the two types of markers 21 is an A marker 21a as a subject corresponding to the first image, and an internal reinforcing bar 31 at the time of bar arrangement work of a wall (hereinafter referred to as a construction target wall) 30 that forms a through hole. Reinforcing bar virtual image 23a made up of a photograph of the state is associated. The other of the two types of markers 21 is a B marker 21b as a subject corresponding to the second image, and is associated with a pen-type virtual image 23b simulating a writing instrument 27 for applying a mark 26 to the construction target wall 30. .

  The A marker 21a is accurately measured and arranged at a predetermined position set in advance on the construction target wall 30, and the positional information on the A marker 21a and the relative relationship between the A marker 21a and the reinforcing bar virtual image 23a. The position information indicating the position is stored in association with each other. The B marker 21 b is provided on the surface of the writing instrument 27, and information indicating the distance between the position of the B marker 21 b on the writing instrument 27 and the tip of the writing instrument 27 is stored in association with each other.

  The imaging unit 10 converts optical information into electrical information by an imaging element such as a CCD sensor that generates image data of a plurality of pixels constituting an image by photographing, such as a digital camera or a digital video camera. It is configured as follows.

  The display unit 16 is configured by, for example, a liquid crystal panel that can display image data of a plurality of pixels constituting an image corresponding to each element. Here, the image data of each pixel is, for example, data indicating each component obtained by decomposing each pixel into an R (red) component, a G (green) component, and a B (blue) component.

  The operation unit 18 is a part where a user who uses the image composition device 1 performs an operation such as photographing or changing settings on the image composition device 1, and photographs at least the actual image 20. A so-called shutter, and a color change switch for changing the color of the image data of the pixel corresponding to the tip of the writing instrument 27. The color change switch can change the image data of the pixel corresponding to the position of the tip in the pen-type virtual image 23b to a predetermined color, for example, black, by the user operating the operation unit 18, for example. Is set.

  The storage unit 14 stores various types of information for generating a composite image in the image composition apparatus 1, and also stores an information storage area 14a in which information is stored and an image in the process of generating the composite image. An image virtual development area 14b that is a work area in which data is rewritten, and an actual image storage area 14c that stores image data of a captured real image until image data of the actual image is generated by the next shooting. have.

  In the information storage area 14a, information for specifying the rebar virtual image 23a associated with the A marker 21a included in the subject, information for specifying the pen-type virtual image 23b associated with the B marker 21b, rebar virtual The image data of the image 23a and the pen-type virtual image 23b, the position information indicating the position to be arranged with respect to the real image 20 of the reinforcing bar virtual image 23a, and the arrangement of the pen-type virtual image 23b in conjunction with the B marker 21b. Composite information such as position information and information indicating the distance between the B marker 21b and the tip of the pen-type virtual image 23b is stored. The information storage area 14a also stores calculated information such as information indicating the distance from the viewpoint of the imaging unit 10 to the subject measured at the time of shooting and information indicating the direction and inclination of the B marker 21b.

  The image virtual development area 14b and the real image storage area 14c are storage areas in which photographed image data is stored for each pixel. As described above, since the image data of each pixel has data indicating three components of R, G, and B, the image virtual development area 14b and the actual image storage area 14c are three times the number of pixels constituting the composite image. Storage area.

  The image data stored in the image virtual development area 14b is rewritten with the corresponding pixel image data as necessary when generating a composite image. The three data stored for each pixel in the image virtual development area 14b are rewritten respectively for the R component, the G component, and the B component when the image data of each pixel is rewritten. On the other hand, the image data of the actual image stored in the actual image storage area 14c is not rewritten, but is used when analyzing the image. The image virtual development area 14b and the real image storage area 14c do not have to be arranged in the same manner as the display section 16 or the elements of the imaging section 10, and the image data of each storage area is stored in the display section 16 or the imaging section. It suffices if it is stored in association with 10 elements.

  The control unit 12 controls the imaging unit 10 and the display unit 16 by operating the operation unit 18 by the user, or detects the marker 21 from the image data of the captured real image 20. Various processes such as an analysis process for analyzing the displayed information and an image synthesis process for synthesizing the actual image 20 and the virtual image 23 stored in the storage unit 14 are executed based on the analyzed information.

Next, an image composition method by the control unit 12 in the present embodiment will be described.
The image composition method according to the present embodiment is provided in the image composition apparatus 1, and a process is started by operating a shutter which is an operation unit 18 for photographing with the imaging unit 10. At this time, the operation unit 18 may be provided with a switch for switching processing between when the image composition device 1 simply captures a real image and when a composite image is generated, such as a digital camera. Further, it is set to always execute the process of generating the composite image, and the process of generating the composite image may be executed only when the marker 21 is detected in the captured real image.

  As described above, in the first embodiment, when the image capturing unit 10 of the image composition device 1 captures an image, the image corresponding subject indicating information such as a QR code (registered trademark) or a barcode is present in the capturing range F1. The combined images 24 and 25 are generated according to the state in which the markers 21a and 21b are included.

  FIG. 2 is a flowchart for explaining the image composition method according to the first embodiment.

  FIG. 3A is a diagram illustrating a state of the first shooting range, FIG. 3B is a diagram illustrating a state where the writing instrument is moved to the shooting range, and FIG. It is a figure which shows the state after giving. 4A is a diagram showing the first composite image, FIG. 4B is a diagram showing the second composite image, and FIG. 4C is the tip of the writing instrument 27 on the second composite image. It is a figure which shows the state showing these orbits.

  As shown in FIG. 3A, when the shutter is operated with the A marker 21 a included in the imaging range F <b> 1 by the user, the control unit 12 images the construction target wall 30 with the imaging unit 10 ( S1), image data of each pixel constituting the photographed real image 20 is generated, and the generated image data is stored in the image virtual development area 14b and the real image storage area 14c of the storage unit 14 (S2).

  Further, the control unit 12 uses the principle of an autofocus mechanism mounted on a general camera based on information acquired from the imaging unit 10 at the time of capturing the actual image 20, for example, the upper half and the lower half of the lens. The distance between the subject and the viewpoint position of the imaging unit 10 is measured using the difference in the center of gravity position that occurs in half, and the measured distance information is stored in the information storage area 14a.

  Next, the control unit 12 analyzes whether or not the A marker 21a as the first image corresponding subject is captured in the image data of the actual image 20 stored in the actual image storage area 14c (S3). At this time, if the A marker 21a is captured in the image data of the real image 20 (S4), information associated with the A marker 21a is acquired from the information storage area 14a (S5). Here, the acquired information includes information for specifying the virtual image 23 associated with the A marker 21a, position information for combining the captured real image 20 and the virtual image 23, and the like.

  At this time, the image data of the real image 20 stored in the real image storage area 14c is analyzed, and if the A marker 21a is not captured in the image data of the real image 20, the image virtual development area 14b is displayed. Based on the stored image data of the real image 20, the real image 20 is displayed on the display unit 16 (S13).

  In the case of this embodiment, the image data of the actual image 20 stored in the actual image storage area 14c is analyzed (S3), the A marker 21a is detected in the image data of the actual image 20 (S4), and A Information associated with the marker 21a is acquired from the storage unit 14 (S5). At this time, information specifying the rebar virtual image 23a to be combined with the captured real image 20, position information for combining the rebar virtual image 23a, that is, the position of the A marker 21a is aligned with the position of the rebar virtual image 23a. Information indicating that the process is to be performed is acquired (S5).

  Next, based on the acquired information, the control unit 12 associates the image data of the target pixel of the real image 20 stored in the image virtual development area 14b with the A marker 21a in the information storage area 14a. The first synthesized image 24 is generated by rewriting the stored image data of the reinforcing bar virtual image 23a (S6). Here, the process from the step S1 for photographing the subject to the step S6 for synthesizing the real image 20 and the reinforcing bar virtual image 23a corresponds to the first image synthesizing step.

  Next, the control unit 12 analyzes whether or not the B marker 21b as the second image corresponding subject is captured in the image data of the actual image 20 stored in the actual image storage area 14c (S7).

  In the first shooting in the present embodiment, as shown in FIG. 3A, the writing instrument 27 is not included in the shooting range F1, and as shown in FIG. 4A synthesized in the image virtual development area 14b. The first composite image 24 is displayed on the display unit 16 (S13).

  Such processing from the photographing step S1 to the display step (S13) is repeatedly executed at predetermined time intervals. For example, the above process is repeated 15 times per second to rewrite the image on the display unit 16 so that a moving image with a frame rate of 15 fps is displayed on the display unit 16.

  Then, when the user moves the writing instrument 27 having the B marker 21b provided on the surface thereof within the imaging range F1, as shown in FIG. 3B, the writing instrument 27 moves within the imaging range F1 during repeated imaging. A real image 20 included in the image is taken. During repeated imaging, the control unit 12 performs the first image synthesis process (S1 to S6) for generating the first synthesized image described above, and when the B marker 21b is detected, imaging is performed. The distance between the viewpoint position of the unit 10 and the B marker 21b is also measured, and the measured distance information is stored in the information storage area 14a.

  After the first composite image 24 is formed for each shooting by repeated shooting, the control unit 12 adds the B marker as the second image corresponding subject to the image data of the actual image 20 stored in the actual image storage area 14c. It is analyzed whether 21b is photographed (S7). At this time, since the B marker 21b is captured in the image data of the real image 20 (S7), information associated with the B marker 21b is acquired from the storage unit 14 (S8). At this time, the photographed writing instrument 27 is used to give a mark to the construction target wall 30, and therefore is not arranged in parallel with the construction target wall 30. For this reason, the control part 12 is based on the position of the B marker 21b specified at the time of imaging, the width | variety and inclination of B marker 21b, and the distance measured at the time of imaging | photography, The pen-type virtual image 23b matched with the B marker 21b. A correction process for matching the image data of the image data with the orientation of the writing instrument 27 is executed (S9).

  Next, the control unit 12 specifies a pixel on which the image data of the corrected pen-type virtual image 23b is synthesized based on the position of the B marker 21b, and the first synthesized image stored in the image virtual development area 14b. The image data of the target pixel is rewritten to the image data of the corrected pen-type virtual image 23b to generate the second composite image 25 as shown in FIG. 4B (S10). Here, the process from step S7 for detecting the B marker 21b to step S10 for synthesizing the first synthesized image and the corrected pen-type virtual image 23b corresponds to the second image synthesizing step.

  At this time, the control unit 12 detects whether or not a signal is input by the color change switch (S11). When the user operates the color change switch when the tip of the pen-type virtual image 23b is placed at a desired position, that is, a position where a mark is desired to be added, the control unit 12 receives a signal from the color change switch. When this is detected (S11), the image data of the pixel corresponding to the position of the tip in the pen-type virtual image 23b of the second composite image 25 stored in the image virtual development area 14b is changed to black (S12). ).

  Then, the second synthesized image 25 as shown in FIG. 4B synthesized in the image virtual development area 14b is displayed on the display unit 16 (S13).

  Thereafter, when the user moves the writing instrument 27, the movement of the pen-type virtual image 23b corresponding to the trajectory of the writing instrument 27 is displayed on the display unit 16, and the black color is displayed along with the trajectory of the tip of the pen-type virtual image 23b. An image 28 showing a mark as shown in FIG. 4C is displayed by connecting the changed pixel data. That is, when the user moves the writing tool 27 within the shooting range F1, the pen-type virtual image 23b also moves in conjunction with the writing tool 27, so that the writing tool 27 looks as if it is looking inside the construction target wall 30. An image that appears to move is displayed. Then, when the user brings the writing instrument 27 into contact with the construction target wall 30 at a desired position, it is possible to actually give the mark 26 to the construction target wall 30 as shown in FIG. .

  According to the image composition method of the present embodiment, the range of the first composite image 24 obtained by photographing the A marker 21a associated with the rebar virtual image 23a and composing the photographed real image 20 and the rebar virtual image 23a. When the B marker 21b associated with the first composite image 24 and the pen-type virtual image 23b as a composite element that can be combined is detected, the pen-type virtual image 23b is combined with the first composite image 24. Since the second composite image 25 is generated, it is possible to generate the second composite image 25 by combining the real image, the reinforcing bar virtual image 23a, and the pen-type virtual image 23b. For this reason, the 2nd synthetic | combination produced | generated by synthesize | combining the pen type | mold virtual image 23b matched with the detected B marker 21b with the 1st synthetic | combination image 24 which synthesize | combined the rebar virtual image 23a with the real image 20. FIG. It is possible to reflect the actual change in the actual part in the image 25.

  For this reason, when trying to construct a through hole in the construction target wall as described above, for example, a position that does not damage the reinforcing bar 31 embedded in the construction target wall 30 is easily identified. Is possible.

  Further, when the movable writing instrument 27 provided with the B marker 21b is detected within the range of the first composite image 24 obtained by combining the real image 20 and the rebar virtual image 23a, the B marker 21b is provided. Since the pen-type virtual image 23b of the writing instrument 27 is synthesized with the first synthesized image 24, the second synthesis is made to image the writing instrument 27 that actually exists on the real image 20 synthesized with the reinforcing bar virtual image 23a. The image 25 can be synthesized.

  In the above embodiment, the composite element that can be combined with the first composite image 24 is the pen-type virtual image 23b. However, the present invention is not limited to this. For example, the image data of the actual image of the writing instrument 27 that is a part corresponding to the writing instrument 27 of the photographed actual image 20 may be combined with the first composite image 24 as a combining element.

  FIG. 5 is a diagram showing an example in which a composite element is made into a figure indicating a region for removing a reinforcing bar virtual image in accordance with the trajectory of the tip of the writing instrument.

  Further, the composite element may be a graphic 29 as shown in FIG. 5 showing a region where the reinforcing bar virtual image 23a is removed from the first composite image by aligning the periphery of the tip of the writing instrument 27 with the trajectory of the tip of the writing instrument 27. Absent. In this case, since the construction target wall 30 is displayed at the site where the rebar virtual image 23a has been removed, the mark 26 actually attached to the construction target wall 30 can be visually recognized on the display unit 16. . Further, the shape of the figure indicating the region from which the reinforcing bar virtual image 23 a is removed may be the shape indicating the writing instrument 27. In this case, since the rebar virtual image 23a is removed from the shape of the writing instrument 27, the writing instrument 27 can be imaged on the second composite image 25, and the second composite synthesized with the rebar virtual image 23a. Even in the case of the image 25, it is possible to generate the second composite image 25 so that the actual image can be seen at the position where the writing instrument 27 is present.

  The above embodiment is for facilitating the understanding of the present invention, and is not intended to limit the present invention. The present invention can be changed and improved without departing from the gist thereof, and it is needless to say that the present invention includes equivalents thereof.

  In the above embodiment, an example in which the writing tool 27 is not included in the shooting range F1 has been described at first. However, the writing tool 27 may be included in the shooting range F1 from the beginning.

  In the above embodiment, the process of changing the color of the pixel where the tip of the writing instrument 27 is positioned in correspondence with the trajectory of the tip of the writing instrument 27 is the operation of the color change switch provided in the operation unit 18. This is not a limitation. For example, a writing instrument is provided with a transmission unit and a transmission switch, and an image composition device is provided with a reception unit. For example, when a mark is attached with a writing instrument, the image composition device receives a signal by infrared rays or the like. Then, the process may be executed. Alternatively, the writing tool and the image composition device may be connected by a cable, and a processing start signal may be transmitted to the image composition device by an operation with the writing tool. Further, the processing may be started when the user detects an operation of applying a mark using a pressure sensor or the like.

  The process of changing the color of the pixel at which the tip of the writing instrument 27 is located in correspondence with the trajectory of the tip of the writing instrument 27 described in the above embodiment is performed by matching the periphery of the tip of the writing instrument 27 with the trajectory of the tip of the writing instrument 27. Even when the reinforcing bar virtual image 23a is removed from the first combined image, the combined image reflected on the reinforcing bar virtual image may be stored in the storage unit separately from the image virtual development area. This is because, for example, when the writing instrument is moved after marking the wall to be constructed, the region from which the reinforcing bar virtual image is removed also moves, so that the part actually marked on the construction target wall is rebar virtual image This is to prevent the position of the mark on the construction target wall from being lost. In other words, if the writing instrument is moved after attaching the mark, and the part where the mark is actually attached to the construction target wall is covered with the reinforcing bar virtual image, the mark on the reinforcing bar virtual image can be displayed. It is to do.

  In the above embodiment, an example in which a moving image is displayed has been described. However, a still image may be displayed every time the shutter is operated.

DESCRIPTION OF SYMBOLS 1 Image composition apparatus 10 Imaging part 12 Control part 14 Storage part 14a Information storage area 14b Image virtual expansion | deployment area 14c Real image storage area 16 Display part 18 Operation part 20 Real image 21 Marker 21a A marker 21b B marker 23 Virtual image 23a Reinforcing bar virtual Image 23b Pen-type virtual image 24 First composite image 25 Second composite image 26 Mark 27 Writing instrument 28 Image showing mark 29 Graphic showing region for removing virtual image 30 Construction target wall 31 Rebar F1 Shooting range

Claims (5)

A step of preliminarily storing an image obtained by photographing a state inside the wall as a virtual image;
Said wall virtual image includes a first image corresponding object associated, by shooting at shooting ranges, to generate a first combined images by combining the real images captured the wall and the virtual image A first image composition step;
When a writing instrument as a second image-corresponding subject in which a composite element that can be combined with the first composite image is associated with the first composite image is detected within the photographing range, the composite element is combined with the first composite image and the first composite image is combined. A second image composition step for generating two composite images;
When the writing instrument is moved, changing the second composite image based on the trajectory that the writing instrument moves, and bringing the writing instrument into contact with the wall to actually mark the wall;
An image synthesizing method characterized by comprising: The image composition method according to claim 1 ,
The image composition method, wherein the composition element associated with the second image-corresponding subject is a second virtual image showing the writing instrument provided with the second image-corresponding subject. The image composition method according to claim 1 ,
The image composition method, wherein the composition element associated with the second image-corresponding subject is a real image of the writing instrument provided with the second image-corresponding subject. The image composition method according to claim 1 ,
The composite element associated with the second image-corresponding subject is a figure indicating a region from which the virtual image is removed at a position associated with the writing instrument on which the second image-corresponding subject is provided. An image composition method. The image composition method according to claim 1 ,
The composition element associated with the second image-corresponding subject is a figure indicating a region for removing the virtual image in the shape of the writing instrument provided with the second image-corresponding subject. Method.

Images