JP2010231459A - Image synthesizing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image synthesizing apparatus for continuously displaying virtual objects even when the degree of matching of a marker and a reference pattern is deteriorated after the virtual object is displayed. <P>SOLUTION: The image synthesizing apparatus is configured to composite a real image with a virtual object, and to make a display device display the synthetic image, and provided with: a virtual object generation unit for generating a virtual object by using a computer graphics technology; a memory for storing a reference pattern corresponding to the virtual object; a real image generation unit for photographing an object, and for generating a real image; and a control unit for comparing a marker detected from the real image with the reference pattern, and for generating the virtual object corresponding to the reference pattern whose degree of matching is equal to or more than a first threshold by the virtual object generation unit, and for compositing it with the real image, and for, after composition, preventing the virtual object from being deleted until the degree of matching of the marker and the reference pattern becomes not larger than a second threshold which is lower than the first threshold. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image synthesizing apparatus, and more particularly to an image synthesizing apparatus for synthesizing a real image and a virtual object and displaying the synthesized image on a display device.

  In recent years, AR (Augmented Reality), MR (Mixed Reality) that gives supplemental information by superimposing computer-generated information (for example, virtual objects by computer graphics technology) on real image information obtained from the real environment A technique called augmented reality or mixed reality has attracted attention (hereinafter, AR / MR / augmented reality / mixed reality technologies are collectively referred to as “AR technology”). Compared to VR (Virtual Reality), the AR technology adds information in a form linked to the real world, and thus has a feature that it is easy for the user to visually understand the relationship between the virtual object and the real world. As a method of AR technology, a virtual object to be associated with a predetermined mark (marker) is set, and the virtual object is synthesized in the video when the degree of coincidence between the marker extracted from the real image and the reference pattern is high (For example, refer to Patent Document 1).

JP 2003-256876 A

  FIG. 15 is a diagram illustrating an example of an AR technique for displaying a virtual object on a marker. In this case, a marker is extracted from the image before AR processing (FIG. 15A), and the degree of coincidence between the extracted marker and a reference pattern held in advance is calculated. When there is a reference pattern with a high degree of coincidence, a virtual object corresponding to the reference pattern is synthesized and displayed on the extracted marker (FIG. 15B). FIG. 16 is a diagram illustrating the relationship between the degree of coincidence between the marker and the reference pattern and the virtual object display. As shown in FIG. 16, when the degree of coincidence between the marker and the reference pattern exceeds a certain threshold value, the virtual object is displayed, and when it falls below the threshold value, the virtual object is not displayed.

  However, in the conventional AR technique, when an obstacle enters a part of the marker, the degree of coincidence between the marker and the reference pattern is reduced, and thus the virtual object is not displayed. As shown in FIG. 17A, when a UI (User Interface) that accepts an input from the user as a virtual object is displayed on the marker, the user tries to perform an input operation on the virtual object. The input from the user (for example, a hand or an input device) is reflected in the camera image. For this reason, the input from the user becomes an obstacle to the marker, and the degree of coincidence between the marker and the reference pattern decreases, so that the virtual object is not displayed as shown in FIG. FIG. 18 is a diagram illustrating the relationship between the degree of coincidence between the marker and the reference pattern and the virtual object display. In this case, due to an input from the user, the degree of coincidence falls below a threshold value for determining whether or not to display a virtual object at time t1, and the virtual object displayed as the UI disappears.

  As a countermeasure to this problem, there is a method of lowering the threshold for determining whether or not to display a virtual object and avoiding the disappearance of the virtual object due to an obstacle. The probability of misrecognition increases, and there is a problem that virtual objects are displayed erroneously for various markers. For this reason, in order to display the virtual object accurately, it is necessary to set a high threshold value for the degree of coincidence with the marker.

  Therefore, an object of the present invention made in view of such a point is to provide an image composition device capable of continuously displaying a virtual object even when the degree of coincidence between the marker and the reference pattern decreases after the virtual object is displayed. There is to do.

In order to solve the above-described problems, the image composition device according to the first invention provides:
An image synthesizing device for synthesizing a real image and a virtual object and displaying the synthesized image on a display device,
A virtual object generation unit that generates a virtual object using computer graphics technology;
A storage unit for storing a reference pattern corresponding to the virtual object;
A real image generation unit that shoots a subject and generates a real image;
The marker detected from the real image is compared with the reference pattern, a virtual object corresponding to a reference pattern having a matching degree equal to or higher than a first threshold is generated by the virtual object generation unit, and combined with the real image,
And a controller that controls the virtual object not to be erased until the degree of coincidence between the marker and the reference pattern is equal to or lower than a second threshold value lower than the first threshold value after the synthesis.

  In the image composition device according to the present invention, the virtual object generation unit can create the virtual object that can be operated by a user.

  According to the present invention, after a real image and a virtual object are combined and displayed, the virtual object is not erased until the degree of coincidence between the marker and the reference pattern is equal to or lower than a second threshold value lower than the first threshold value. Therefore, even when the degree of coincidence between the marker and the reference pattern decreases after the virtual object is displayed, the virtual object can be continuously displayed.

1 is a diagram illustrating a schematic configuration of an image composition device according to an embodiment of the present invention. It is a process flowchart of the image composition apparatus shown in FIG. It is a process flowchart of the image composition apparatus shown in FIG. It is a process flowchart of the image composition apparatus shown in FIG. It is a process flowchart of the image composition apparatus shown in FIG. It is a figure which shows the image composition environment which concerns on one Embodiment of this invention. It is a figure which shows the marker seen from the image composition apparatus shown in FIG. It is a figure which shows the example of correction of the marker shown in FIG. It is a figure which shows the coincidence degree calculation example of the marker and reference pattern which concern on one Embodiment of this invention. It is a figure which shows the user input example to the virtual object which concerns on one Embodiment of this invention. It is a figure which shows the example of a coincidence calculation with the marker and reference pattern with respect to the user input of FIG. FIG. 10 is a diagram illustrating the relationship between the transition of the degree of coincidence with the user input and the display state of the virtual object. It is a figure which shows the example of a display of the virtual object which concerns on one Embodiment of this invention. It is a figure which shows transition of a coincidence degree in the case of FIG. It is a figure which shows the example of a display of the virtual object on a marker. It is a figure which shows the relationship between transition of a coincidence in the case of FIG. 15, and the display state of a virtual object. It is a figure which shows the example of a display of the virtual object with respect to a user input. It is a figure which shows the relationship between the transition of a coincidence in the case of FIG. 17, and the display state of a virtual object.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram showing a schematic configuration of an image composition device according to an embodiment of the present invention. The image composition device 1 includes an imaging unit 2 that captures a subject and generates real image data, a storage unit 3, a display unit 4, and a control unit 6. The storage unit 3 includes an application program 31 related to AR, reference pattern information 32 to be compared with the marker, and virtual object information 33 to be displayed when the degree of coincidence between the marker and the reference pattern is high. The display unit 4 displays a display related to the application program and a composite image by AR processing. The control unit 6 includes an AR control unit 61, and the AR control unit 61 includes an input image analysis unit 611, a virtual object determination unit 612, a virtual object generation unit 613, and a virtual object operation unit 614. The input image analysis unit 611 analyzes the actual image data generated by the imaging unit 2, and performs marker extraction and calculation of the three-dimensional position and orientation of the marker. The virtual object determination unit 612 calculates the degree of coincidence between the marker extracted by the input image analysis unit 611 and the reference pattern, and if the degree of coincidence exceeds a threshold (generation threshold), the virtual object to be displayed on the marker Select. The virtual object generation unit 613 generates a virtual object selected by the virtual object determination unit 612 on the marker whose matching degree exceeds the generation threshold using a computer graphic technique. When there is an input from the user to the virtual object, the virtual object operation unit 614 performs control corresponding to the user input and the virtual object.

  Here, the imaging unit 2 can be configured by any suitable imaging device such as a camera. The control unit 6 is configured by any suitable processor such as a CPU (Central Processing Unit), and each function unit (AR control unit 61, input image analysis unit 611, virtual object determination unit) of the control unit 6 is configured. 612, the virtual object generation unit 613, and the virtual object operation unit 614) are configured as software executed on the CPU, or a dedicated processor specialized in the processing of each functional unit (for example, a DSP (digital signal processor) )) And the like. The memory | storage part 3 can be comprised by volatile memory (SRAM etc.), non-volatile memory (EEPROM etc.), or another suitable storage medium. The display unit 4 can be configured by any suitable display device such as a liquid crystal display.

  As described above, the control unit 6 includes functional units such as the AR control unit 61, the input image analysis unit 611, the virtual object determination unit 612, the virtual object generation unit 613, and the virtual object operation unit 614. In this case, the control unit 6 including at least the input image analysis unit 611 and the virtual object determination unit 612 constitutes a “control unit” described in the claims. In the case of this embodiment, the imaging unit 2 constitutes a “real image generation unit” described in the claims of the present invention.

  In the image composition device 1 shown in FIG. 1, the display unit 4 is included in the image composition device 1. However, the display unit 4 is provided outside the image composition device 1, for example, from the image composition device 1. It should be noted that the display device can be configured as an independent display device.

  In the present embodiment, the storage unit 3 stores marker information extracted from the actual image as the structure Mrk. The structure Mrk, which is marker information, includes info, position, AR, and PD as member variables. info is a data set of marker coordinates and size extracted from real image data, color information, and the like. The position indicates the position and orientation of the virtual object to be displayed with respect to the extracted marker. AR is a flag indicating the display state of the virtual object with respect to the extracted marker. PD indicates a reference pattern corresponding to the extracted marker.

  Hereinafter, the notation Mrk (t) [i] indicates the structure of the i-th marker included in the real image data at time t. In addition, the notation Mrk (t) [i] .info, Mrk (t) [i] .position, Mrk (t) [i] .AR, Mrk (t) [i] .PD (t) Indicates a reference to a member variable included in [i].

  FIG. 2 is an operation flowchart of the image composition device 1. When processing is started in the image composition device 1, the control unit 6 initializes a parameter t representing elapsed time (step S001) and activates the imaging unit 2 (step S002). While the operation of the image synthesizing apparatus 1 continues, the imaging unit 2 periodically shoots a subject and generates real image data P (t) (steps S003 and S004). The input image analysis unit 611 performs input image analysis processing such as extraction of markers included in the actual image data P (t) on the image data P (t) acquired from the imaging unit 2 (step S005). The virtual object determination unit 612 calculates the degree of coincidence between the marker extracted by the input image analysis process and the reference pattern stored in the storage unit 3, and selects a virtual object to be displayed on the extracted marker. Object determination processing is performed (step S006). The virtual object generation unit 613 performs a virtual object generation process of displaying a virtual object on the extracted marker according to the result of the virtual object determination process (step S007).

  FIG. 3 is a flowchart of the input image analysis process in step S005 of FIG. First, the input image analysis unit 611 initializes a counter i for identifying a marker included in the actual image data P (t) (step S101). Next, the input image analysis unit 611 performs appropriate image processing on the image data P (t), extracts markers included in the actual image data P (t) (step S102), and extracts marker information. Is stored in Mrk (t) [i] .info of the storage unit 3 (step S103). As described above, this Mrk (t) [i] .info is a data set of marker coordinates and sizes extracted from the real image data P (t), color information, and the like. The marker extraction by the input image analysis unit 611 can be realized by various known techniques related to image processing, and will not be described in detail here.

  Next, the input image analysis unit 611 searches whether the information of the same marker as the extracted marker is already stored in the storage unit 3 except at the time of initial activation (t> 0, Yes in step S104). That is, the input image analysis unit 611 includes a marker that is duplicated with the marker extracted at the current time t in the marker information extracted from the actual image data at the previous time t−1 (hereinafter, this marker is referred to as “duplicate marker”). The duplication marker information is indicated by Mrk (t-1).) (Step S105). When the information of the overlapping marker is stored in the storage unit 3, the input image analysis unit 611 refers to Mrk (t-1) .AR and determines whether a virtual object is displayed for the overlapping marker. This is performed (step S106). If a virtual object is already displayed for the overlapping marker (Mrk (t-1) .AR = ON), the input image analysis unit 611 extracts the marker extracted at the current time t in order to maintain the display of the virtual object. Also, a flag for displaying a virtual object is set (Mrk (t) [i] .AR = ON) (step S107). Note that at the time of initial activation (t = 0, NO in step S104), when there is no overlapping marker (NO in step S105), or when a virtual object is not displayed for the overlapping marker (NO in step S106). In step S107, the process in step S107 is skipped.

  Next, the input image analysis unit 611 refers to the extracted marker Mrk (t) [i] .info, and coordinates-converts the position and orientation of the marker to the position and orientation at the viewpoint of the imaging unit 2 (step S108). ), The result of the coordinate transformation is stored in Mrk (t) [i] .position (step S109). As described above, Mrk (t) [i] .position indicates the position and orientation at which the virtual object is displayed.

  The input image analysis unit 611 repeats the loop 1 process (steps S102 to S109) as many times as the number of markers extracted from the actual image data P (t). For this reason, even when the actual image data P (t) includes a plurality of markers, the processing of loop 1 is performed for each marker. Note that each time the processing of loop 1 is repeated, the counter i for identifying the marker included in the actual image data P (t) is incremented (i ++), and therefore the structure Mrk (t ) [i] is not saved.

  FIG. 4 is a flowchart of the virtual object determination process in step S006 of FIG. In the virtual object determination process, the virtual object determination unit 612 first reads the reference pattern Pd from the reference pattern information 32 of the storage unit 3 (step S201), and identifies a marker included in the actual image data P (t). The counter i is initialized (step S202). Next, the virtual object determination unit 612 determines whether there is a valid reference pattern Pd and marker information Mrk in the storage unit 3 (steps S203 and S204), and if they exist (the reference pattern Pd and marker information Mrk are present). If neither is NULL), the following processing is performed.

  First, in loop 1, the virtual object determination unit 612 reads Mrk (t) [i] .info that is marker information from the storage unit 3 (step S205), and initializes a counter k for identifying the reference pattern Pd. Loop 2 is entered (step S206). In loop 2, the virtual object determination unit 612 determines whether the current (kth) reference pattern Pd (k) is valid (step S207), and if Pd (k) is valid, The degree of coincidence RA between the (i-th) marker information Mrk (t) [i] and the current reference pattern Pd (k) is calculated (step S208).

  At this time, the virtual object determination unit 612 refers to Mrk (t) [i] .AR and determines whether a virtual object is being generated for the current marker (step S209). When a virtual object has not yet been generated for the current marker (Mrk (t) [i] .AR = OFF), the virtual object determination unit 612 sets the threshold value S for determining the matching degree RA to A generation threshold G for determining whether to generate a virtual object is substituted (step S210). When a virtual object has already been generated for the marker (Mrk (t) [i] .AR = ON), the virtual object determination unit 612 sets the threshold value S for determining the matching degree RA to A maintenance threshold M for determining whether to maintain the display of the virtual object is set (step S211).

  Here, the generation threshold G is set to a relatively high value in order to avoid displaying an erroneous virtual object with respect to the marker, whereas the maintenance threshold M is set such that the user's hand or the like is applied to the camera. Even when the degree of coincidence RA decreases due to reflection, the value is set to be relatively lower than the generation threshold G because it is for maintaining the display of the virtual object.

  Next, the virtual object determination unit 612 compares the matching degree RA calculated in step S208 with a threshold S for determining the matching degree RA (step S212). When the degree of coincidence RA is higher than the threshold S, the virtual object determination unit 612 displays the current (i-th) marker information Mrk (t) [i] and the current (k-th) reference pattern Pd ( k) and Pd (k) are set in Mrk (t) [i] .PD (step S213). When the degree of coincidence RA is lower than the threshold value S, the virtual object determination unit 612 displays the current (i-th) marker information Mrk (t) [i] and the current (k-th) reference pattern Pb ( k) does not correspond, and Mrk (t) [i] .PD is not set (step S214).

  The virtual object determination unit 612 repeats the processing of loop 1 (steps S205 to S206) as many times as the number of markers extracted from the real image data P (t). Further, the virtual object determination unit 612 performs the processing of loop 2 (steps S207 to S214) by the number of reference patterns stored in the storage unit 3 or until the matching degree RA is higher than the threshold value S. repeat. Therefore, even when the virtual image determination unit 612 includes a plurality of markers in the real image data P (t) and a plurality of reference patterns are stored in the storage unit 3, the virtual object determination unit 612 corresponds to each marker. A pattern can be obtained.

  FIG. 5 is a flowchart of the virtual object generation process in step S007 of FIG. In the virtual object generation process, first, the virtual object generation unit 613 initializes a counter i for identifying a marker included in the actual image data P (t) (step S301). Next, the virtual object generation unit 613 enters loop 1 and checks whether a reference pattern corresponding to the current (i-th) marker is set, so that Mrk (t) [i] .PD in the storage unit 3 is set. Is referred to (step S302). When the reference pattern is set in Mrk (t) [i] .PD, the virtual object generation unit 613 displays the virtual object VRO corresponding to Mrk (t) [i] .PD as virtual object information in the storage unit 3. 33 (step S303), based on the position and orientation of Mrk (t) [i] .position, a process of generating and synthesizing the virtual object VRO on the current marker is performed (step S304). After the synthesis process, the virtual object generation unit 613 turns on Mrk (t) [i] .AR to indicate that the virtual object is displayed on the current marker (step S305). In step S302, if no reference pattern is set in Mrk (t) [i] .PD, the virtual object generation unit 613 indicates that no virtual object is displayed on the current marker. (t) [i] .AR is set to OFF (step S306).

  The virtual object generation unit 613 repeats the loop 1 process (steps S302 to S306) by the number of markers extracted from the real image data P (t). Therefore, even when the actual image data P (t) includes a plurality of markers, a virtual object is generated and synthesized for each marker.

  FIG. 6 is a diagram showing a virtual object display environment according to an embodiment of the present invention. In the present embodiment, the image composition device 1 is configured as a camera-equipped HMD (head mounted display), and the user photographs the marker MK placed on the desk with the image composition device 1 mounted on the head. In the case of FIG. 6, the image composition device 1 includes the display unit 4 as an HMD. For example, the HMD corresponding to the display unit 4, a personal computer with a camera corresponding to the image composition device 1, and the like. It is also possible to configure the information processing terminal as a separate device. When the display unit 4 and the image composition device 1 are configured as separate devices, the display unit 4 can be configured with a lighter display device, and the image composition device 1 can be configured with a higher performance information processing device. is there. Of course, the camera may be configured separately from the information processing terminal.

  FIG. 7 is a diagram illustrating real image data including the marker MK generated by the imaging unit 2 of the image composition device 1. As shown in FIG. 7, in this embodiment, the marker MK is composed of a square double frame and character information (KCC). The input image analysis unit 611 extracts the marker from the actual image data, performs coordinate conversion, and calculates the matching RA with the reference pattern stored in the storage unit 3 as shown in FIG. Correct the image.

  FIG. 9 is a diagram illustrating a calculation example of the degree of coincidence RA between the marker extracted from the actual image data and the reference pattern. In the case of FIG. 9, the virtual object determination unit 612 divides the marker line extracted from the real image data by several feature points, performs matching with each feature point of the reference pattern, The degree of matching RA is calculated. For example, when the 56 feature points of the marker among 64 feature points of the reference pattern match, the matching degree RA is 87.5%.

  FIG. 10 is a diagram illustrating a virtual object that can be operated by the user displayed on the marker and an example of user input to the virtual object. When the user input is performed on the virtual object as illustrated in FIG. 10B, the virtual object determination unit 612 is stored in the storage unit 3 with the marker including the user input as an obstacle as illustrated in FIG. The degree of coincidence RA is calculated by comparing with the reference pattern.

  FIG. 12 is a diagram illustrating a relationship between the matching degree RA between the marker and the reference pattern and the virtual object display when the image composition device 1 sets the maintenance threshold value M. As shown in FIG. 12, since no user input is performed on the virtual object until time t0, the matching degree RA exceeds the generation threshold G, and the virtual object is continuously displayed. When a user input is performed at time t0 as shown in FIG. 10B, the matching degree RA decreases. However, the image composition device 1 according to the present invention sets a maintenance threshold M for the virtual object determination unit to determine whether or not to maintain the display of the virtual object in Step 211 of FIG. For this reason, even if the matching level RA is reduced by the user input, as long as the matching level RA is higher than the maintenance threshold M, the display of the virtual object is maintained.

  As described above, according to the present embodiment, the virtual object determination unit 612 determines whether or not to generate a virtual object at the threshold S for determining the matching degree RA after the virtual object is displayed on the marker. In order to set a maintenance threshold M that is lower than the generation threshold G for determining whether to maintain the display of the virtual object, the matching degree RA between the marker and the reference pattern has decreased after the virtual object is displayed. Even in this case, the virtual object can be continuously displayed. In other words, the control unit 6 synthesizes the real image and the virtual object, and then the second threshold (maintenance threshold M) in which the degree of coincidence RA between the marker and the reference pattern is lower than the first threshold (generation threshold G). Since control is performed so that the virtual object is not erased until the following is reached, the virtual object can be continuously displayed even when the degree of coincidence RA between the marker and the reference pattern decreases after the virtual object is displayed.

  In addition, according to the present embodiment, the virtual object generation unit 613 generates a virtual object as a virtual object that can be operated by the user, so that various UIs associated with the real image can be displayed to the user. It becomes possible. In particular, by configuring a virtual object three-dimensionally, it is possible to provide a user with a UI that is more realistic and has a high affinity with a real image.

  Further, according to the present embodiment, the generation threshold G can be set high. When the generation threshold G is set high, a virtual object is generated only when the matching degree RA is high. Probability can be greatly reduced. In addition, the maintenance threshold M can be set low, and when the maintenance threshold M is set low, the display of the virtual object is maintained even if an obstacle such as the user's finger enters the marker. A virtual UI (virtual object) can be operated without hesitation.

  FIG. 13 is a diagram illustrating a virtual object display example when the threshold value S for determining the matching degree RA is returned from the maintenance threshold value M to the generation threshold value G after the user input to the virtual object is lost. As described above, the image composition device 1 according to the present invention can display virtual objects for a plurality of markers. FIG. 14 is a diagram illustrating the transition of the matching degree RA when there is no user input to the virtual object (an obstacle to the marker). When there is no user input at time t0, the obstacle to the marker is removed, so the matching degree RA increases. When the coincidence RA becomes equal to the generation threshold G at time t1, the virtual object determination unit 612 deletes the maintenance threshold M and returns the threshold S for determining the coincidence RA from the maintenance threshold M to the generation threshold G. .

  As described above, according to the present embodiment, the virtual object determination unit 612 deletes the maintenance threshold M and the threshold S is generated from the maintenance threshold M after the user input to the virtual object (an obstacle to the marker) disappears. When the value is returned to G, the threshold value S increases, so that it is possible to prevent an erroneous virtual object from being displayed with respect to the marker. Therefore, it is possible to avoid the problem that the false recognition rate is increased by lowering the threshold value S, and to improve the usability for the user.

  Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention. For example, each functional unit can be rearranged so as not to be logically contradictory, and a plurality of functional units can be combined into one or divided.

DESCRIPTION OF SYMBOLS 1 Image composition apparatus 2 Imaging part 3 Storage part 31 Application 32 Reference pattern information 33 Virtual object information 6 Control part 61 AR control part 611 Input image analysis part 612 Virtual object determination part 613 Virtual object generation part 614 Virtual object operation part

Claims (2)

An image synthesizing device for synthesizing a real image and a virtual object and displaying the synthesized image on a display device,
A virtual object generation unit that generates a virtual object using computer graphics technology;
A storage unit for storing a reference pattern corresponding to the virtual object;
A real image generation unit that shoots a subject and generates a real image;
The marker detected from the real image is compared with the reference pattern, a virtual object corresponding to a reference pattern having a matching degree equal to or higher than a first threshold is generated by the virtual object generation unit, and combined with the real image,
A controller that controls the virtual object not to be erased until the degree of coincidence between the marker and the reference pattern is equal to or lower than a second threshold lower than the first threshold after the synthesis;
An image synthesizing apparatus.   The image synthesis apparatus according to claim 1, wherein the virtual object generation unit creates the virtual object that can be operated by a user.

Images