JP6360348B2 - Image processing method and image processing program - Google Patents

Description

  The present invention relates to an image processing method and an image processing program.

  AR (Augmented Reality) technology is known in which the real world can be expanded and displayed by superimposing annotations on imaging data generated by imaging with an imaging means such as a camera. An annotation is a virtual object that consists of image data stored in a computer in association with a marker and does not actually exist. In the AR technology, the display position of the virtual object is calculated by recognizing a marker attached to an actually existing article (real object).

  With regard to such an AR technology, for example, according to the image composition method of Patent Document 1, by recognizing a marker on a desk (real object), whether a cup (virtual object) exists on the desk on the display screen. Is displayed. In this image composition method, a step of preparing in advance three-dimensional information of a planned range (desk surface) where a marker is arranged, a step of capturing an image to generate image data, a step of detecting a marker in the image data, The method includes a step of superimposing three-dimensional information on the marker position in the imaging data, and a step of superimposing a virtual object thereon. If this image composition method is used, since the marker is hidden by the three-dimensional information, the real object and the virtual object are displayed without a sense of incongruity.

  Since various image data can be adopted as the virtual object, the display content of the virtual object is not particularly limited. However, in the AR technology in which a virtual object is added to a real object for display, the real object needs to continue displaying actual video (imaging data). That is, conventionally, the real object itself cannot be displayed in various forms.

JP 2005-234757 A

  An object of the present invention is to provide an image processing method and an image processing program that can display a real object itself in various modes.

The image processing method of the present invention is included in the imaging step in which the imaging unit captures an image at a substantially constant imaging position and orientation and generates imaging data, and the imaging data generated after the arrangement of the real object having the marker. A first detection step of detecting the marker and specifying the arrangement position and arrangement orientation of the real object with respect to the imaging position and the imaging orientation; and an object image of a virtual object stored in a storage unit in association with the marker A first compositing step of reading out data and superimposing the object image data on the imaging data generated before the arrangement of the real object to generate first composite data; And continuously displaying the imaging data until the arrangement posture is specified, and the arrangement position and arrangement figure of the real object Of anda display switching step of displaying the first combined data after a specific,
In the first composition step, the object image data is superimposed at a position and posture corresponding to the placement position and placement posture of the real object specified in the first detection step.

  The image processing method of the present invention further includes a monitoring step for monitoring an imaging situation, and a storage step for storing the imaging data when the imaging situation has not changed as imaging data before arrangement of a real object. It is characterized by that.

  The image processing method of the present invention is characterized in that the imaging data stored immediately before the marker is detected in the first detection step is used as imaging data before arrangement of a real object.

  The image processing method of the present invention further includes a monitoring step for monitoring an imaging situation, and when the imaging situation has changed, the imaging data is displayed in the display switching step.

  In the image processing method of the present invention, further, based on the imaging data generated after the imaging situation has changed, the position and orientation corresponding to the arrangement position and orientation of the real object after the imaging situation has changed. A second composition step of generating second composite data by superimposing the object image data of the virtual object that continuously transitions with the imaging data before the placement of the real object, and in the display switching step, The display is switched from the two composite data to the imaging data.

  The image processing program of the present invention is a program that causes a computer to execute the above steps.

The following data is displayed on the display means of the present invention. Before the real object is arranged, the imaging data a generated by the imaging means is displayed. After the real object is arranged, the imaging data b including the video of the real object is displayed until the position and orientation are specified. After specifying the position and orientation, the first composite data formed by superimposing the object image data of the virtual object on the imaging data a before the placement of the real object is displayed as a background video.
Here, since the imaging means captures images at a certain imaging position and orientation and generates imaging data a and b, the imaging data a and b are the same video except for the presence or absence of the video of the real object. That is, the same background video is always displayed on the display means.
In addition, the real object is not displayed in the first composite data, and instead, a virtual object superimposed at a position and posture corresponding to the arrangement position and arrangement posture of the real object is displayed.
Therefore, the display means displays only the video of the displayed real object as if it was replaced with a virtual object. Since various image data can be adopted as the object image data of the virtual object, according to the present invention, only the real object in the imaging scene can be displayed in various modes.

It is a figure which shows the usage example of the image processing apparatus of this invention. 1 is a block diagram of an image processing apparatus of the present invention. It is a flowchart of the image processing method of this invention. It is explanatory drawing of an effect | action of this invention. It is a block diagram of the other image processing apparatus of this invention. It is a flowchart of the other image processing method of this invention. 10 is a flowchart of still another image processing method of the present invention. 10 is a flowchart of still another image processing method of the present invention.

  An image processing apparatus, an image processing method, and an image processing program of the present invention will be described. In addition, the same code | symbol shown over each figure shows the same or the same thing.

  As shown in FIG. 1, for example, the image processing apparatus 1 includes a display unit 3 such as a liquid crystal display panel, an imaging unit 5, and a computer that is communicably connected thereto. This is a portable information terminal having the display means 3 on the front and the imaging means 5 on the back. It is used with its posture held substantially constant by a table 7 or the like placed on the base surface 6.

  The imaging means 5 captures a predetermined range at a substantially constant position and orientation, and generates imaging data 21 (FIG. 4). The imaging means 5 is typically a camera module that has a lens and an image sensor and images a predetermined range. The generated imaging data 21 is imaging data 21a (FIG. 4A) before the real object 15 is arranged and imaging data 21b (FIG. 4B) after the real object 15 is arranged. . The imaging data 21 may be generated by imaging at a predetermined frame rate. Further, the imaging data 21 is acquired by the control means 11 described later and stored in an internal storage (not shown).

  As shown in FIG. 2, the computer includes storage means 9 and control means 11.

  The storage means 9 stores the object image data of the virtual object 17 (FIG. 4C) in association with the marker 13, and is an auxiliary storage such as a flash memory. The marker 13 has a predetermined pattern and color determined in order that the detection unit 19 described later detects the presence of the real object 15 and specifies its arrangement position and arrangement posture. It is preferable that the image is displayed without a sense of incongruity. The real object 15 is a variety of real articles such as dolls, models, stationery, and other daily items that are predetermined as detection targets, and the marker 13 is displayed on the entire surface or a part thereof. The object image data of the virtual object 17 is preferably moving image data for displaying the virtual object 17 in an animation, but is not limited thereto, and may be still image data of the virtual object 17. The virtual object 17 is computer graphics (hereinafter referred to as CG) related to the real object 15. Preferably, it is a CG whose basic stationary form is substantially the same as the form of the real object 15. However, the present invention is not limited to this. For example, a CG of a different color of the real object 15 may be used. Further, it may be a CG of an article similar to the real object 15. The object image data of the virtual object 17 stored in association with one marker 13 is not limited to one, and may be plural.

  As shown in FIG. 2, the control means 11 is connected to the storage means 9, the imaging means 5, and the display means 3 so as to be able to communicate therewith, and controls them, for example, an SoC incorporated in a portable information terminal. (System on a Chip). The control means 11 includes a detection unit 19, a synthesis unit 25, and a display switching unit 23. The control unit 11 has an internal storage (not shown), and stores and reads data such as the imaging data 21 as necessary.

  The detection unit 19 acquires the imaging data 21, detects the marker 13a included in the imaging data 21b generated after the arrangement of the real object 15, and the arrangement position of the real object 15 with respect to the imaging position and imaging posture of the imaging means 5 And the arrangement posture is specified. The synthesizing unit 25 reads out the object image data of the virtual object 17 from the storage means 9 and superimposes the image data 21a generated before the arrangement of the real object 15 on the first synthesized data 27 (FIG. 4C). ) Is generated. The display switching unit 23 continues to display the imaging data 21 on the display means 3 until the arrangement position and orientation of the real object 15 are specified. After the arrangement position and arrangement orientation of the real object 15 are specified, the display switching unit 23 The composite data 27 is displayed.

  Next, as shown in FIG. 3, the image processing method of the present invention includes an imaging step s10, a first detection step s30, a first synthesis step s40, and a display switching step. This display switching step includes an imaging data display step s20 and a composite data display step s50. Hereinafter, each step will be described in the order of the flow.

  The imaging step s10 is a step in which the imaging means 5 captures an image at a substantially constant imaging position and orientation, and generates imaging data 21a before the real object 15 is arranged and imaging data 21b after the real object 15 is arranged. . For example, the imaging unit 5 captures an image at a predetermined frame rate and generates imaging data 21.

  The imaging data display step s20 is a step in which the display switching unit 23 acquires the imaging data 21 and continues to display the imaging data 21 on the display means 3. The imaging data 21 is displayed until the arrangement position and the arrangement posture of the real object 15 are specified in a first detection step s30 described later. When the imaging data 21 is captured and generated at a predetermined frame rate, a real-time video is displayed.

  In the first detection step s30, the detection unit 19 acquires the imaging data 21, detects the marker 13a (FIG. 4B) included in the imaging data 21b generated after the arrangement of the real object 15, and performs imaging. This is a step of specifying the arrangement position and arrangement posture of the real object 15 with respect to the position. For the detection of the marker 13a, known image recognition in the AR technique is used. For example, a plurality of feature points included in the imaging data 21 are extracted, and a figure (hereinafter referred to as figure A) formed by connecting the feature points and a basic figure (hereinafter referred to as figure) of the marker 13 stored in the storage means 9. The presence or absence of the marker 13a can be detected by comparing B). The arrangement position and arrangement orientation of the real object 15 are specified by detecting the position and coordinate system of the marker 13a based on the position of the figure A, the size and distortion of the figure A with respect to the figure B, for example. The identified arrangement position and arrangement orientation of the real object 15 are stored in the internal memory because they are used in a first synthesis step s40 described later. As described above, when the detection of the marker 13a is successful and the arrangement position and the arrangement posture are specified (arrow Y in s31), the process proceeds to the first synthesis step s40. On the other hand, when the marker 13a cannot be detected from the imaging data 21 or the arrangement position and orientation cannot be specified (arrow N in s31), the imaging data 21 is stored in the internal memory as the imaging data 21a before arrangement of the real object 15. The process returns to the imaging step s10 via the storing step s32.

  In the first synthesis step s40, the synthesis unit 25 acquires the imaging data 21a before the arrangement of the real object 15, reads out the object image data of the virtual object 17 stored in the storage unit 9, and adds the acquired imaging data 21a to the acquired imaging data 21a. In this step, the first composite data 27 is generated by superimposing the object image data. As the object image data, data corresponding to the type of the marker 13a detected in the first detection step s30 is retrieved and read. When superimposing the object image data, the object image data is superimposed on the position and orientation corresponding to the arrangement position and arrangement orientation of the real object 15 specified in the first detection step s30. For example, the reference point and coordinate system of the virtual object 17 can be overlapped with the detected position and coordinate system of the marker 13a. When the object image data is moving image data, the superimposition start position and superposition start attitude of the virtual object 17 are superimposed in correspondence with the arrangement position and arrangement attitude of the real object 15.

  The composite data display step s50 is a step in which the display switching unit 23 acquires the first composite data 27 from the composite unit 25 and displays it on the display means 3 after specifying the arrangement position and orientation of the real object 15.

  The image processing program of the present invention is a program that causes the imaging unit 5, the display switching unit 23, the detection unit 19, and the synthesis unit 25 to execute the above steps.

  The operation and effect of the present invention will be described with reference to FIG. In FIG. 4, a bus model is used as the real object 15 having the marker 13, and the bus model and the image processing apparatus 1 of the present invention are placed on a desk 29.

(1) First, as shown in FIG. 4A, in the state where the bus model is not arranged, the imaging data 21a of only the desk 29a generated by the imaging step s10 is displayed by the imaging data display step s20. 3 is displayed. Further, since the bus model is not disposed within the imaging range, the marker 13a is not detected by the first detection step s30. Therefore, the imaging data 21a is stored in the internal storage as the imaging data 21a before the real object 15 is arranged by the storage step s32. Then, the process returns to the imaging step s10.
(2) Next, as shown in FIG. 4B, when the bus model is arranged, the arrangement position and the arrangement attitude are specified by the first detection step s30. Until the arrangement position and orientation are specified, the imaging data display step s20 displays the imaging data 21b including the desk 29a and the bus model (real object 15a) placed thereon on the display means 3. Is displayed.
(3) When the position and orientation of the bus model are specified in the first detection step s30, the bus CG is compared with the imaging data 21a before the bus model is arranged in the first synthesis step s40. The first synthesized data 27 is generated by superimposing the moving image data that moves. Then, in the combined data display step s50, as shown in FIG. 4C, a moving image from the bus image shown by the broken line to the bus image shown by the solid line is displayed on the display means 3. It should be noted that it is preferable to use video techniques such as morphing and crossfading during the switching in order to produce a natural switching between the imaging data 21a and the first composite data 27.

As described above, the imaging data 21a generated by the imaging means 5 is displayed before the real object 15 is arranged. After the real object 15 is arranged, the imaging data 21b including the real object video 15a is displayed until the position and orientation thereof are specified. After specifying the position and orientation, the first composite data 27 formed by superimposing the object image data of the virtual object 17 on the imaging data 21a before the real object 15 is arranged is displayed.
Here, since the imaging means 5 captures images at a constant imaging position and orientation and generates imaging data 21a and 21b, the imaging data 21a and 21b are the same video except for the presence or absence of the video 15a of the real object. That is, the same background video is always displayed on the display means 3.
Further, the real object video 15a is not displayed in the first composite data 27, and instead, the virtual object 17 is displayed in the display position and the display posture of the real object video 15a in the imaging data 21b.
Therefore, only the video 15a of the displayed real object is displayed as if it was replaced with the virtual object 17. In the above example, moving image data in which a bus of the same form as the real object 15 travels is used as the object image data, so that the bus placed on the desk is displayed as if it has started traveling alone.
Since various image data can be adopted as the object image data of the virtual object 17, according to the present invention, only the real object 15 in the imaging scene can be displayed in various modes. For example, moving image data that transforms a bus into a robot may be adopted as object image data.

  The image processing apparatus, the image processing method, and the image processing program of the present invention are not limited to the above embodiments.

  For example, the image processing apparatus 1 of the present invention is not limited to the position and posture holding mode as shown in FIG. 1, and may be held by the user in his / her hand.

  As shown in FIG. 5, the control means 33 of the image processing apparatus 31 of the present invention may include a monitoring unit 35 that monitors the imaging state. The monitoring unit 35 acquires the imaging data 21 from the imaging unit 5 and inputs the imaging data 21 to the detection unit 19 and the synthesis unit 25 according to the imaging situation.

  Further, as shown in FIGS. 6 and 7, the image processing method of the present invention may include monitoring steps s60 and s70 in which the monitoring unit 35 monitors the imaging state. The monitoring steps s60 and s70 typically include a fluctuation monitoring step and / or a moving body monitoring step.

  The fluctuation monitoring step is a step in which the monitoring unit 35 monitors the imaging position and the imaging posture. For example, the fluctuation monitoring step acquires the imaging data 21 from the imaging means 5 at a predetermined frame rate, compares the continuous imaging data 21 to calculate the fluctuation amount, and based on the fluctuation amount, the imaging position and the imaging posture are calculated. The presence or absence of fluctuation is detected. In calculating the amount of variation, first, a reference point is determined in one imaging data 21, and its coordinates are acquired. Next, the reference point in the other imaging data 21 continuous to the one imaging data 21 is detected, and the coordinates thereof are acquired. Then, a difference is obtained for the coordinates of the reference point between the imaging data 21. Thereby, the fluctuation amount is calculated. If the fluctuation amount is within a predetermined range and this state continues for a predetermined time, it is determined that the imaging position and the imaging posture have not changed, that is, the image processing device 31 itself is in a stationary state. On the other hand, when the fluctuation amount is outside the predetermined range, it is determined that the imaging position and the imaging posture are fluctuating, that is, the image processing device 31 itself is not in a stationary state.

  Note that the fluctuation monitoring step is not limited to monitoring the imaging position and the imaging posture based on the imaging data 21. For example, the image processing device 31 includes an acceleration sensor (not shown) that is communicably connected to the control means 11, and the monitoring unit 35 varies the difference based on the difference between the output values of the acceleration sensor acquired at a predetermined frame rate. You may monitor.

  The moving body monitoring step is a step in which the monitoring unit 35 monitors the presence or absence of an object that moves within the imaging range (hereinafter referred to as a moving body). The presence or absence of a moving object is detected as follows, for example. First, the imaging data 21 is acquired from the imaging means 5 at a predetermined frame rate. Next, a difference image between continuous imaging data 21 is obtained. Then, it is determined that there is a moving body when a portion where the gradation value is not 0 (or a portion equal to or greater than a predetermined value close to 0) in the difference image is continuously greater than or equal to a predetermined area ratio. On the other hand, when the portion where the gradation value is not 0 continuously does not exceed the predetermined area ratio, it is determined that there is no moving body.

  Note that the imaging state monitored by the monitoring unit 35 is not limited to the change in the imaging position and the imaging posture and / or the presence or absence of a moving body. For example, the change in the brightness of the imaging range may be monitored by comparing the respective imaging data 21 acquired from the imaging means 5 at a predetermined frame rate.

  In the image processing method of the present invention, the monitoring step s60 is provided between the imaging data display step s20 and the first detection step s30 as shown in FIG. If a change is detected in the monitoring step s60 (arrow N in s61), the process returns to the imaging step s10. On the other hand, if no change is detected (arrow Y in s61), the process proceeds to the first detection step s30. If the marker 13a in the imaging data 21 cannot be detected through the first detection step s30, or if the arrangement position and orientation cannot be specified (arrow N in s31), the process proceeds to the storage step s32, and the imaging data 21a is Stored in internal memory. In other words, the storage step s32 is the imaging data 21a generated when the imaging situation has not changed (typically, when the imaging position and the imaging posture are not changed and there is no moving object within the imaging range). Is stored as the imaging data 21a before the real object 15 is arranged.

Here, according to the monitoring step s60 including the above-described fluctuation monitoring step, the process proceeds to the first detection step s30 when there is no fluctuation in the image processing device 31. Therefore, when it is in a stationary state, the imaging data 21a before the arrangement of the real object 15 can be instantly stored, and the background image of the first composite data 27 can be prepared without being noticed by the user.
Further, according to the monitoring step s60 including the above-described moving body monitoring step, when there is a moving body within the imaging range, the process does not proceed to the first detection step s30. Therefore, the imaging data 21 generated during the arrangement of the real object 15 is not stored, and is automatically excluded from the target of the background video in the first composite data 27.

  Further, in the image processing method of the present invention including the monitoring step s60 described above, the synthesis unit 25 uses the imaging data 21a stored in the internal storage immediately before the marker 13a is detected as the imaging data 21a before the real object 15 is arranged. It is preferable to use as. As a result, the first composite data 27 is generated using the imaging data 21a generated immediately before the composition as a background video. Therefore, when the display content is switched from the imaging data 21b to the first composite data 27 by the display switching step, the background video can be switched without a sense of incongruity.

  Further, in the image processing method of the present invention including the monitoring step s70 described above, the display switching step may display the imaging data 21 when the imaging state changes. Specifically, the monitoring step s70 is provided in the combined data display step s50 as shown in FIG. If no change is detected in this monitoring step s70 (arrow Y in s71), the first combined data display s51 is continued. On the other hand, when a change is detected (arrow N in s71), the combined data display step s50 is completed, the process proceeds to the end, and the process is restarted from the start. Thereby, the imaging data 21 is displayed. When monitoring the change in the imaging state using the imaging data 21b in the monitoring step s70, an imaging step for imaging the predetermined imaging range and generating the imaging data 21b is provided before the monitoring step s70.

  In the present invention including the monitoring step s70, the first combined data display s51 is interrupted when the imaging state changes. Then, when the process is restarted from the start, the imaging data 21 is displayed in the imaging data display step s20. For this reason, the unnatural display that the background image being displayed does not follow the change in the imaging state is prevented, and the user is not aware of the mechanism of the present invention.

  Furthermore, as shown in FIG. 7B, the image processing method of the present invention including the monitoring step s70 described above is the second synthesis step in which the synthesis unit 25 generates the second synthesis data when the imaging state changes. s82 and the step of the display switching unit 23 switching the display from the second composite data to the imaging data 21 may be included.

  The second composite data is obtained by superimposing animation data on the imaging data 21a before the arrangement of the real object 15. This animation data is typically other object image data of the virtual object 17 stored in advance in the storage means 9 in association with the marker 13, and one object image data included in the first composite data 27 Are different. The other object image data is moving image data that continuously transitions to a position and orientation corresponding to the arrangement position and arrangement orientation of the real object 15 after the imaging state changes. The synthesizing unit 25 reads out the animation data and superimposes it on the imaging data 21a before the real object 15 is arranged. At this time, it is preferable to superimpose animation data in accordance with the position and orientation of the virtual object 17 being displayed in the composite data display step s50. When the transition ends, a flag indicating the end of the second composite data may be set.

  The arrangement position and the arrangement posture of the real object 15 after the change of the imaging situation are specified based on the imaging data 21 generated after the change of the imaging situation. For example, as shown in FIG. 7B, the second imaging step s80 and the second detection step s81 are specified.

  The second imaging step s80 is a step in which the imaging means 5 captures an image after the imaging state changes and generates imaging data 21, and is provided after the change is detected by the monitoring step s70 (arrow N in s71). The imaging data 21 may be generated by imaging at a predetermined frame rate.

  The second detection step s81 is a step in which the detection unit 19 specifies the arrangement position and the arrangement posture of the real object 15 with respect to the imaging position and the imaging posture after the change of the imaging situation, based on the imaging data 21 after the imaging situation changes. It is. This identification method can be identified by the same method as in the first detection step s30. When acquiring the imaging data 21 for each predetermined frame rate, the arrangement position and the arrangement posture of the real object 15 may be specified each time.

  In the display switching step, the display switching unit 23 acquires the second synthesized data from the synthesizing unit 25 and switches the display of the first synthesized data 27 to the display of the second synthesized data for display. And a step of switching from the second composite data display s83 to the display of the changed imaging data 21. The second composite data display s83 is displayed until the transition of the virtual object 17 is completed (arrow N in s84). The end of the transition is determined with reference to, for example, a flag set in the second synthesis step s82. When the transition is completed (arrow Y in s84), the process proceeds from the composite data display step s50 to the end, restarts from the start, and the imaging data 21 is displayed (s20). Note that it is preferable to use a video technique such as morphing or crossfading during the switching in order to produce the switching of the display naturally.

  As described above, when the imaging state changes, the second composite data display s83 is interposed between the first composite data display s51 and the changed imaging data display, so that the switching between the virtual object 17 and the real object 15a is natural. It is expressed in

  An image processing program of the present invention causes a computer to execute the above steps.

  The image processing method, the image processing program, and the image processing apparatus of the present invention have been described above. However, the present invention includes various improvements, corrections, and modifications based on the knowledge of those skilled in the art without departing from the spirit of the present invention. These embodiments are all within the scope of the present invention.

  For example, the flow in the image processing method is not limited to the above, and may be the flow shown in FIG. In the flow shown in FIG. 8, each step from the start to the end is performed in one frame, and this flow is executed every frame.

First, imaging data 21 for each frame is acquired in imaging step s10. Next, in the monitoring step s90, a change in imaging state is detected for each frame. Next, the display flag indicating the content to be displayed on the display means 3 and the monitoring result of the imaging state are confirmed (s91). Depending on the result, one of the flows of (Case 1) to (Case 4) shown in FIG. 8 is executed.
(Case 1) When the display flag indicates the imaging data 21 and the imaging situation has changed, the imaging data display step s20 is executed, and the processing of the frame ends.
(Case 2) If the display flag indicates the imaging data 21 and the imaging state has not changed, the process proceeds to the first detection step s30. If the detection fails (arrow N in s31), the storage step s32 The imaging data display step s20 is executed in this order, and the processing of the frame ends. On the other hand, when the detection is successful (arrow Y in s31), the display flag is manipulated (s92), and the first synthesis step s40 described below (case 3) is executed. In the display flag operation (s92), a flag indicating that the first composite data 27 is displayed on the display means is set.
(Case 3) When the display flag indicates the first composite data 27 and the imaging state has not changed, the first composite step s40 and the first composite data display s51 are executed in this order, and the processing of the frame ends. To do.
(Case 4) When the display flag indicates the first composite data 27 and the imaging state changes, or when the display flag indicates the second composite data, the second detection step s81 and the second composite step s82 The second composite data display s83 and the display flag operation s93 are executed in this order. In the display flag operation s93, a flag indicating that the second composite data is displayed on the display means 3 is set when the virtual object 17 is in transition. When the transition of the virtual object 17 is completed, a flag indicating that the imaging data 21 is displayed on the display unit is set.

  Further, the real object 15 used in the present invention is not limited to one type, and may be a plurality of types. The plurality of types of real objects 15 have different markers 13 respectively. In this case, the storage means 9 includes an identification ID for identifying each marker, a basic figure associated with each identification ID for recognizing and detecting the marker 13, and a virtual object associated with each identification ID. Seventeen object image data are stored. In the first detection step s30, the detection unit 19 compares the feature points included in the imaging data 21a with the basic figure, detects the presence / absence of the marker, and identifies the identification ID. In the first composition step s40, the composition unit 25 obtains an identification ID, reads object image data corresponding to the identification ID, and generates first composition data 27. As a result, the virtual object 17 corresponding to the type of the real object 15 is displayed on the display means 3 in various modes.

1,31… Image processing device
3… Display means
5… Imaging means
7… stand
9… Memory means
11,33… Control means
13… Marker displayed on real object
13a… Marker in imaging data
15… Real objects in the real world
15a… Real object in the imaging data
17… virtual objects
19… Detector
21… Imaging data
21a… Imaging data before placement of real object
21b… Imaging data after placement of real object
23… Display switching section
25… Synthesizer
27… First synthesis data
29… desk
29a… Desk in imaging data
35… Monitoring unit

Claims (9)

An imaging step in which the imaging means captures an image at a substantially constant imaging position and orientation and generates imaging data;
A first detection step of detecting the marker included in the imaging data generated after arrangement of a real object having a marker, and specifying the arrangement position and arrangement attitude of the real object with respect to the imaging position and the imaging attitude;
Reads the object image data of the virtual object stored in the storage means in association with the marker, and generates the first composite data by superimposing the object image data on the imaging data generated before the arrangement of the real object A first synthesis step to
A monitoring step for monitoring imaging conditions;
A storage step of storing the imaging data when the imaging situation has not changed as imaging data before arrangement of a real object;
A display switching step of continuously displaying the imaging data until the arrangement position and orientation of the real object are specified on the display means, and displaying the first composite data after specifying the arrangement position and orientation of the real object; ,
Including
The imaging data stored immediately before the marker is detected in the first detection step is used as imaging data before the arrangement of the real object, and the actual data identified by the first detection step in the first synthesis step is used. An image processing method, wherein the object image data is superimposed at a position and orientation corresponding to an arrangement position and orientation of an object. The image processing method according to claim 1, wherein the image processing step returns to the imaging step after the storing step. When the imaging situation is changing, in the display switching step, the image processing method according to claim 1, wherein the displaying the captured data. Based on the imaging data generated after the imaging situation has changed, the object of the virtual object that continuously transitions to a position and orientation corresponding to the arrangement position and orientation of the real object after the imaging situation has changed Including a second synthesis step of generating second synthesized data by superimposing the image data on the imaging data before arrangement of the real object;
The image processing method according to claim 3 , wherein in the display switching step, the display is switched from the second composite data to the imaging data. A storage step of storing object image data of the virtual object in association with the marker in a storage means of the computer;
An imaging step of imaging the imaging means at a substantially constant imaging position and orientation and generating imaging data;
The detection unit of the computer detects the marker included in the imaging data generated after the arrangement of the real object having the marker, and determines the arrangement position and the arrangement attitude of the real object with respect to the imaging position and the imaging attitude. A first detection step to identify,
A monitoring step of monitoring an imaging state in the monitoring unit of the computer;
A storage step of storing, in the internal storage of the computer, the imaging data when the imaging situation has not changed as imaging data before arrangement of a real object;
A first compositing step of reading the object image data to a compositing unit of the computer and generating the first composite data by superimposing the object image data on the imaging data generated before the arrangement of the real object;
The imaging data is continuously displayed on the display means until the arrangement position and orientation of the real object are specified on the display switching unit of the computer, and after the arrangement position and orientation of the real object are specified, the first object is displayed. A display switching step for displaying the composite data,
The imaging data stored immediately before the marker is detected in the first detection step is used as imaging data before the arrangement of the real object, and the actual data identified by the first detection step in the first synthesis step is used. An image processing program that superimposes the object image data at a position and orientation corresponding to an arrangement position and orientation of an object. The image processing program according to claim 5, which causes the imaging unit to execute an imaging step after the storing step. 6. The image processing program according to claim 5 , wherein the imaging data stored immediately before the marker is detected in the first detection step is used as imaging data before arrangement of a real object. Causing the monitoring unit of the computer to execute a monitoring step of monitoring an imaging state;
The image processing program according to claim 5 , wherein the imaging data is displayed in the display switching step when the imaging situation is changed. Based on the imaging data generated after the imaging situation changes, the composition unit of the computer continuously detects the position and orientation corresponding to the arrangement position and orientation of the real object after the imaging situation changes. A second synthesis step of generating second synthesized data by superimposing the object image data of the virtual object transitioning to the imaging data before arrangement of the real object,
9. The image processing program according to claim 8 , wherein in the display switching step, the display is switched from the second composite data to the imaging data.

Images