JP6938158B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present invention relates to an information processing device, an information processing method, a program and an information processing system that provide mixed reality and augmented reality.

Mixed reality (hereinafter referred to as MR) technology and expanded reality (AR) technology are known as technologies for fusing the real world and the virtual world in real time. These technologies are technologies that seamlessly fuse the real space and the virtual space created by a computer.

As one of the devices for providing MR to users, there is a video see-through type information processing device. This is a device that displays a composite image in which a virtual object is superimposed on an image captured in the real world with a video camera on a display device such as a display in real time and presents it to the user. As an example of such a device, a head-mounted video see-through type head mounted display (hereinafter, referred to as HMD) and the like are known. The video see-through type HMD used for MR measures the position and orientation of the camera in the real space at the time of taking the image every time an image is input from the built-in camera. Then, the HMD draws a computer graphic (hereinafter referred to as CG) based on the position and orientation of the camera and the unique parameters of the camera such as the focal length, and captures the CG image in the real space. Overlay on the image.

Here, as an important function for realizing MR, there is an alignment function. In order for the user to feel that the virtual object actually exists in the real space, it is necessary that the virtual object and the real space are geometrically consistent. That is, from the viewpoint of the user who is experiencing MR, the virtual object must appear to exist at a position where it should exist in the real space. For this reason, the HMD is designed to obtain the position and orientation of the built-in camera in the real space and to obtain the geometrical consistency between the virtual object and the real space.

The position and orientation of the camera can be estimated based on the image information from the camera built in the HMD. For example, an index whose three-dimensional position in the real space is known is photographed by a camera, and based on the correspondence between the position of the index in the captured image and the three-dimensional position of the index in the real space, the camera A method for estimating the position and orientation is known. As a known index, a quadrangle marker, a circular marker, a point marker, or the like is used as an example, and these indexes are artificially arranged in the real space, for example. Non-Patent Document 1 describes a technique that enables the position and orientation of a camera to be estimated by adding (arranging) a feature point marker having a feature that is easy to detect from an image captured by the camera in a real space environment as an index. Is disclosed.

Ryuhei Tenmoku, Akito Nishigami, Fumihisa Shibata, Asako Kimura and Hideyuki Tamura: Balancing design freedom and constraints in wall posters masquerading as AR tracking markers, Proc. Human-Computer Interaction International, 2009 D. Lowe, "Distinctive image features from scale-invariant keypoints.", International Jurnal of Computer Vision (IJCV), 2004 Bay, Herbert, Tinne Tuytelaars, and Luc Van Gool. "Surf: Speeded up robust features.", European Conference on Computer Vision (ECCV), 2006

In the technique described in Non-Patent Document 1 described above, the position and orientation of the camera can be estimated by adding a feature point marker having a feature that is easy to detect from an image captured by the camera to the environment as an index. However, when the feature point marker is attached to the environment, there arises a problem that the landscape of the real space is spoiled by the feature point marker.

Therefore, an object of the present invention is to make it possible to generate an index that does not impair the landscape of the real space.

The present invention provides a brightness determining means for determining the brightness of a pattern given as an index to an experience area in the real space based on a threshold value for detecting a feature of an image, and a hue of each pixel of the captured image in the real space. The main hue of the captured image is obtained from the histogram of the above, and the color determining means for determining the color of the pattern based on the main hue and the brightness of the pattern, and the determined brightness and color of the pattern. It is characterized by having a generation means for generating an image.

According to the present invention, it is possible to generate an index that does not impair the landscape of the real space.

It is a figure which shows the schematic structure of the information processing apparatus in 1st Embodiment. It is a figure which shows an example of the pattern which is the basis of the index pattern. It is a figure used for explaining the combination example of the luminance difference of an index pattern. It is a flowchart of the process of the information processing apparatus of 1st Embodiment. It is a figure used for explaining the size example of a plurality of index patterns. It is a figure used for explaining the combination example of a plurality of index patterns. It is a figure which shows the schematic structure of the information processing apparatus of 2nd Embodiment. It is a figure used for explaining the method of determining the color of a pattern. It is a flowchart of the process of the information processing apparatus of 2nd Embodiment. It is a figure which shows the configuration example which realizes this embodiment by a computer.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.
In the following description, an example of providing a user with mixed reality (MR) and augmented reality (AR) that fuse the real world and the virtual world by computer graphics (CG) in real time will be given. MR and the like can be applied to various fields such as assembly support for superimposing and displaying work procedures and wiring during assembly work, and surgical support for superimposing and displaying the state of the body on the body surface of a patient. As a device for providing MR to a user, a head-mounted video see-through type head mounted display (HMD) or a portable information terminal such as a tablet terminal having a video camera on the back is used. In this embodiment, a video see-through type HMD will be taken as an example. The illustration of the video see-through type HMD is omitted. The information processing system of the present embodiment includes a video see-through type HMD and an information processing device 1000 of the present embodiment described later.

When providing MR to the user in the present embodiment, the video see-through type HMD estimates the position and orientation of the camera based on the image taken by the built-in camera (not shown). The position and orientation of the camera can be measured by a physical sensor having 6 degrees of freedom such as a magnetic sensor, an ultrasonic sensor, and an optical sensor, but in the present embodiment, it is easier and cheaper than the case of using a physical sensor. , The method of estimating the position and orientation of the camera based on the captured image is used. When estimating the position and orientation of the camera based on the captured image, the HMD of the present embodiment uses the feature points of the image such as points, lines, edges, corners, and brightness gradients as natural features from the captured image in the real space. It detects and estimates the position and orientation of the camera based on its natural features.

However, if there are few natural features in the captured image in the real space, it is difficult to estimate the position and orientation of the camera. Therefore, in the present embodiment, a pattern that can be detected as a natural feature is generated, and the image of the pattern is used as an index. Is given to the real space as. In the present embodiment, the generated pattern is referred to as an index pattern. Although the details will be described later, the index pattern of the present embodiment is a pattern generated based on the threshold value used when detecting the natural feature from the captured image in the real space, and can be detected as the natural feature from the captured image. At the same time, it is generated as a pattern that is difficult for the user to see. In the following description, the threshold value used when detecting a natural feature from a captured image is referred to as a feature detection threshold value.

<First Embodiment>
FIG. 1 is a diagram showing a schematic configuration of an information processing apparatus 1000 that generates an index pattern of the first embodiment. The information processing device 1000 includes a pattern acquisition unit 1010, an information storage device 1020, a pattern size determination unit 1030, a pattern brightness determination unit 1040, a feature detection threshold acquisition unit 1050, and a pattern output unit 1060. The pattern acquisition unit 1010 to the pattern output unit 1060 are examples of the determination means and the generation means in the present embodiment. The pattern size reference information input unit 100 may be a device connected to the information processing device 1000, or may be included in the information processing device 1000.

The pattern size reference information input unit 100 inputs information about the experience area in the real space, and holds the information about the experience area in a memory area (not shown). Information about the experience area may be input from, for example, the administrator of the information processing apparatus 1000 via an input device (not shown), or may be obtained by calculation or the like. Although the MR administrator and the user are separated here, the administrator and the user may be the same person. The information about the experience area held in the memory area of the pattern size reference information input unit 100 is sent to the information processing apparatus 1000 as one of the pattern size reference information used when generating the index pattern according to the present embodiment. Be done. In the present embodiment, the information regarding the experience area includes, for example, information on the shortest distance and the longest distance between various objects in the experience area and the camera of the HMD.

The objects in the experience area include, for example, not only the work table and the patient's body when MR is applied to assembly support, surgical support, etc., but also the walls, ceiling, floor, and the like of the room. Further, the information regarding the experience area may include not only the shortest distance and the longest distance between the object in the experience area and the camera, but also information such as the depth, height, and width of the experience area in the real space. ..

The pattern size reference information also includes information such as the angle of view of the lens of the camera that shoots the real space, the focal length, the optical characteristics of the lens, the exposure at the time of video shooting, and the resolution of the captured image. Information such as the angle of view, focal length, optical characteristics, exposure, and resolution of these cameras may be input by an administrator or the like, or may be calculated by the pattern size reference information input unit 100.

The pattern acquisition unit 1010 of the information processing apparatus 1000 acquires pattern size reference information from the pattern size reference information input unit 100. Further, the pattern acquisition unit 1010 acquires the information of the original pattern, which is the source of the index pattern, from the information storage device 1020. The original pattern may be a predetermined pattern or a pattern generated every time a request is made from the pattern acquisition unit 1010. When the original pattern is generated every time a request is made from the pattern acquisition unit 1010, for example, the original pattern is generated by the original pattern generation unit (not shown). The pattern size reference information and the original pattern information acquired by the pattern acquisition unit 1010 are sent to the pattern size determination unit 1030.

The information storage device 1020 stores at least the information of the original pattern and the information of the feature detection threshold value. The information storage device 1020 sends information on the original pattern to the pattern acquisition unit 1010 in response to a request from the pattern acquisition unit 1010, and acquires feature detection threshold information in response to a request from the feature detection threshold acquisition unit 1050. Send to section 1050. The feature detection threshold value acquisition unit 1050 sends the feature detection threshold value information acquired from the information storage device 1020 to the pattern luminance determination unit 1040 described later.

The pattern size determination unit 1030 determines the pattern size Sα of the index pattern based on the pattern size reference information sent from the pattern acquisition unit 1010 and the information of the original pattern.

Here, the size of the index pattern needs to be sized so that the HMD can detect the index pattern as a natural feature from the captured image when the experience area to which the index pattern is given is photographed by the camera. In addition, the index pattern will be given to the object in the experience area. Therefore, the size of the index pattern is determined by the HMD when the index pattern is applied to an object at any position within the range of the shortest distance and the longest distance between each object in the experience area and the camera. The size must be such that the index pattern can be detected as a natural feature from the image. Further, since the index pattern is detected from the captured image in the experience area, for example, if the camera cannot resolve the pattern of the index pattern and the pattern of the index pattern in the captured image is crushed, the index pattern cannot be detected. become. Therefore, the size of the index pattern is the spatial frequency of the pattern of the index pattern regardless of the position within the range of the shortest distance and the longest distance between the object and the camera in the experience area. It is necessary to determine the size that the camera can resolve. However, if the size of the index pattern is larger than necessary, there is a high possibility that the index pattern will be easily visible to MR users, etc. when the index pattern is given in the experience area. Therefore, the index pattern size is necessary. It is necessary to decide on a size that does not become larger than this. Therefore, the pattern size determination unit 1030 can detect the natural feature of the index pattern from the captured image based on each information such as the shortest distance and the longest distance, the spatial frequency of the pattern of the pattern, and the resolution of the camera. Calculate the pattern size Sα that is not larger than necessary.

FIG. 2 shows an example of the original pattern 200. The original pattern 200 of the example of FIG. 2 is a pattern having, for example, a square pattern in which the B region 202 is arranged in the center and the A region 201 is arranged around the B region 202. Further, the A region 201 is, for example, a region having a high density (low brightness), and the B region 202 is, for example, a region having a low density (high brightness). The size S in FIG. 2 is assumed to be a reference size (for example, the size of the original pattern) when the pattern size Sα is calculated by the pattern size determination unit 1030. Hereinafter, a method of determining the pattern size Sα by the pattern size determining unit 1030 will be described with reference to the example of the original pattern 200 shown in FIG.

First, the longest distance Dmax between the object in the experience area and the camera is calculated based on the information on the depth, height, and width of the experience area. The longest distance Dmax may be calculated by the pattern size determination unit 1030, or may be input by the administrator of the information processing apparatus 1000 or the like. When the depth of the experience area is Z, the height is Y, and the width is X, the longest distance Dmax between the object in the experience area and the camera can be calculated by the equation (1). It is assumed that the shortest distance Dmin is input by, for example, the administrator of the information processing apparatus 1000.
Dmax = √ (Z ² + Y ² + X ² ) ・ ・ ・ Equation (1)

Further, when the short side length of the image sensor of the camera is L and the spatial frequency of the pattern of the original pattern 200 is F, the reference size S is represented by the equation (2). Since the resolution of the camera is determined by the ratio of the side length of the image pickup element to the number of pixels, the short side length L of the image pickup element of the camera is used here as a value that simply represents the resolution of the camera. ..
S = L / (2 × F) ・ ・ ・ Equation (2)

Then, the pattern size determination unit 1030 calculates the pattern size Sα by the equations (3) and (4), respectively. Equation (3) is an equation for obtaining the pattern size Sα according to the shortest distance Dmin between the object and the camera in the experience area, and equation (4) is the equation (4) for obtaining the pattern size Sα based on the longest distance Dmax between the object and the camera. This is the formula to be calculated.
Sα = S × Dmin ・ ・ ・ Equation (3)
Sα = S × Dmax ・ ・ ・ Equation (4)

In this way, the pattern size determination unit 1030 obtains a set of pattern sizes Sα corresponding to the shortest distance Dmin and the longest distance Dmax between the object in the experience area and the camera. Then, the pattern size determination unit 1030 sends the information of the set of index patterns in which the pattern size Sα corresponding to the shortest distance Dmin and the longest distance Dmax is determined to the pattern brightness determination unit 1040 together with the pattern size reference information.

The pattern luminance determination unit 1040 receives information on a set of index patterns whose pattern size has been determined by the pattern size determination unit 1030 and pattern size reference information, and also receives information on the feature detection threshold value from the feature detection threshold value acquisition unit 1050. receive. Then, the pattern luminance determination unit 1040 determines the luminance of each index pattern based on the pattern size reference information and the feature detection threshold value. Specifically, the pattern luminance determination unit 1040 determines the luminance of the index pattern so as to exceed the feature detection threshold. However, if the brightness is simply determined so as to exceed the feature detection threshold value, the brightness of the index pattern in the captured image may fall below the feature detection threshold value due to the characteristics of the image pickup element of the camera and the optical characteristics of the lens. There is. Therefore, the pattern brightness determining unit 1040 considers what the brightness of the index pattern will be in the captured image, and is based on the pattern size reference information (information such as the characteristics of the image sensor of the camera and the optical characteristics of the lens). To determine the brightness of the index pattern.

FIG. 3 shows an example of the index pattern 300 after the brightness is determined with respect to the original pattern 200 of FIG. 2 described above. In the index pattern 300 of FIG. 3, for example, when the feature detection threshold value is the value of the luminance “10”, the luminance value of the A region 301 is “0” and the luminance value of the B region 302 is “10”. And. The A region 301 is a black region because the luminance value is "0", and the B region 302 is a gray region having a high density because the luminance value is "10". Hereinafter, the pattern luminance determination method by the pattern luminance determination unit 1040 will be described with reference to the example of the index pattern 300 of FIG.

Here, in order for the HMD to be able to detect the index pattern in the captured image as a natural feature, the index pattern must maintain the brightness condition that can be detected as a natural feature. As an example, a case where a natural feature is detected by a brightness gradient of a captured image will be taken as an example. As a method for detecting a natural feature by a brightness gradient, a known method such as Non-Patent Document 2 or Non-Patent Document 3 can be used. When a natural feature is detected by the brightness gradient, the brightness of the A region 301 and the B region 302 of the index pattern 300 in FIG. 3 must be a combination of brightness that causes a brightness difference equal to or larger than the feature detection threshold. However, if the brightness difference is larger than necessary, there is a high possibility that the MR user or the like can easily see the index pattern when the index pattern is given in the experience area. Therefore, it is necessary to determine the brightness of the A region 301 and the B region 302 of the index pattern 300 to be a combination of brightness that causes a brightness difference equal to or larger than the feature detection threshold value and a brightness that does not increase the brightness difference more than necessary. In the example of FIG. 3, the feature detection threshold value is determined to be “10”, the luminance value of the A region 301 is determined to be “0”, and the luminance value of the B region 302 is determined to be “10” based on the luminance determination conditions. There is.

Although the luminance is taken as an example here, for example, when the luminance does not affect the detection of natural features, the luminance in the A region and the luminance in the B region are maintained as long as the pattern of the original pattern is maintained. There is no limit to the combination of. Further, in the above example, only the "brightness" of the index pattern is described, but if the brightness is expressed in a color space such as an RGB color space or a YUV color space, the brightness in any color space can be used. May be good. Further, the algorithm for detecting a natural feature is an algorithm that defines the condition of another natural feature and detects the natural feature if it can detect the one satisfying the above-mentioned condition of the natural feature. May be good.

As described above, the pattern size Sα is determined by the pattern size determining unit 1030, and the information of the set of index patterns whose brightness is determined by the pattern luminance determining unit 1040 is sent to the pattern output unit 1060. The pattern output unit 1060 generates and outputs an image of the index pattern according to the information of the received set of index patterns. The image output of the index pattern may be performed by, for example, printing on a paper medium or projection by light. In the present embodiment, a paper medium on which the image of the index pattern is printed is placed in the experience area, and the projected light of the image of the index pattern is projected into the experience area to display the index to the experience area in the real space. The pattern is added.

FIG. 4 is a flowchart showing a processing flow from acquisition of pattern size reference information by the pattern size reference information input unit 100 and the information processing device 1000 of FIG. 1 to image generation and output of an index pattern. In the following description, each step ST300 to ST350 in the flowchart is abbreviated as ST300 to ST350, and the same applies to other flowcharts described later. The processing of the flowchart of FIG. 4 is started, for example, when the administrator of the information processing apparatus 1000 instructs the information processing apparatus 1000 to generate an index pattern.

In the flowchart of FIG. 4, in ST300, the pattern size reference information input unit 100 acquires the pattern size reference information including the information regarding the experience area described above. After ST300, the information processing apparatus 1000 performs the processing after ST310. The process of ST310 is a process performed by the pattern acquisition unit 1010 of the information processing apparatus 1000.

In ST310, the pattern acquisition unit 1010 receives the pattern size reference information from the pattern size reference information input unit 100, and also receives the information of the original pattern from the information storage device 1020. Then, the pattern acquisition unit 1010 sends the acquired information to the pattern size determination unit 1030. After S310, the information processing apparatus 1000 proceeds to ST320 performed by the pattern size determination unit 1030.

In ST320, the pattern size determination unit 1030 calculates the pattern size Sα as described above based on the pattern size reference information and the information of the original pattern. Then, the pattern size determination unit 1030 sends the index pattern information determined for the pattern size Sα, the pattern size reference information, and the like to the pattern luminance determination unit 1040. After ST320, the information processing apparatus 1000 proceeds to ST330 performed by the feature detection threshold value acquisition unit 1050.

In ST330, the feature detection threshold value acquisition unit 1050 acquires the feature detection threshold value information from the information storage device 1020 and sends the feature detection threshold value information to the pattern luminance determination unit 1040. After ST330, the information processing apparatus 1000 proceeds to ST340 performed by the pattern luminance determination unit 1040.

In ST340, the pattern luminance determination unit 1040 determines the luminance of the index pattern based on the feature detection threshold value as described above. Then, the pattern luminance determination unit 1040 sends the information of the index pattern whose luminance is determined to the pattern output unit 1060. After ST340, the information processing apparatus 1000 proceeds to ST350 performed by the pattern output unit 1060.

In the ST350, the pattern output unit 1060 generates and outputs an image of the index pattern in which the pattern size and the brightness are determined as described above. After ST350, the information processing apparatus 1000 ends the processing of the flowchart of FIG.

The processing order of ST300 and ST310 may be changed, the processing order of ST320 to ST340 may be changed as appropriate, or they may be executed in parallel.

As described above, the index pattern of the present embodiment is a pattern of a size determined based on the pattern size reference information including the information about the experience area in the real space, and the feature detection when detecting the natural feature. It is a brightness pattern generated based on the threshold value. Therefore, the index pattern of the present embodiment is a pattern that can be detected as a natural feature from the captured image of the experience area in the real space and is difficult for the user to visually recognize. In addition, since the index pattern given to the experience area is a pattern that is difficult to see not only for HMD users but also for third parties who do not use the HMD, the landscape in the real space may be impaired. No.

<Modification 1 of the first embodiment>
In the first embodiment, an example of generating a set of index patterns of the pattern size Sα based on the shortest distance and the longest distance has been given. For example, a plurality of indexes corresponding to the respective sizes S1 to S6 as shown in FIG. A set of patterns consisting of patterns 501 to 506 may be generated. In the case of a set of a plurality of index patterns 501 to 506 as shown in FIG. 5, the pattern sizes S1 to S6 may be determined for each index pattern. The pattern sizes S1 to S6 of each of these index patterns are determined as sizes within the range of an appropriate pattern size Sα based on the shortest distance and the longest distance, the angle of view of the camera, the resolution, and the like described above.

The size of the plurality of index patterns is not limited to the example of FIG. 5, and is set to another size as long as it is within the range of the appropriate pattern size Sα based on the information of the experience area and the external device such as the camera. You may. In the example of FIG. 5, as in the case of the first embodiment described above, the index pattern is a pattern that can be detected as a natural feature from the captured image of the experience area in the real space and is difficult for the user to see. ..

<Modification 2 of the first embodiment>
Further, in the first embodiment, the brightness is determined for a set of index patterns, but if the above-mentioned brightness determination condition is satisfied, for example, as shown in FIG. 6, the index patterns 601 to 606 with a plurality of brightness are used. It may be generated as a set of patterns consisting of. Note that FIG. 6 shows an example in which the brightness of each of the plurality of index patterns of the plurality of sizes described in FIG. 5 is determined. In the example of FIG. 6, the index pattern 601 has 0 in the A region and 10 in the B region, the index pattern 602 has 30 in the A region and 40 in the B region, and the index pattern 603 has the brightness in the A region. Is 40 and the brightness in the B region is 50. Further, the index pattern 604 has a brightness of 10 in the A region and a brightness of the B region of 20, the index pattern 605 has a brightness of 20 in the A region and a brightness of the B region of 30, and the index pattern 606 has a brightness of 50 in the A region and B. The brightness of the area is 60. Also in the example of FIG. 6, similarly to the above, the index pattern is a pattern that can be detected as a natural feature from the captured image of the experience area in the real space and is difficult for the user to see.

<Modification 3 of the first embodiment>
In the first embodiment, the brightness of the index pattern is determined based on the pattern size reference information and the feature detection threshold value, but the brightness of the experience area (for example, a real wall or floor) to which the pattern is given is determined. The brightness of the index pattern may be determined in consideration.

As an example, an example in which an index pattern is assigned to the wall will be described. In the case of the third modification, the information processing apparatus 1000 first acquires an image of an actual wall to which the index pattern is to be applied, and obtains the brightness of the wall from the image. In the case of this example, the captured image of the wall is input to the information processing device 1000 via the pattern size reference information input unit 100. The image pickup device that images the actual wall may be an HMD camera or a separately prepared camera.

Further, the region where the brightness is required is defined as the region around the position where the index pattern is actually applied to the actual wall in the image region of the wall shown in the captured image. Then, the average value of the brightness in the surrounding area is obtained as the brightness of the wall. As an example, the process of obtaining the average value of the luminance is performed by the pattern luminance determination unit 1040 of the information processing apparatus 1000 or the like.

Here, when the obtained wall brightness is 50, the brightness of the index pattern to be arranged is set to, for example, 60. In the index pattern in the case of this modification 3, the brightness of the above-mentioned area A or the brightness of the area B may be set to 60, or a pattern consisting of only one area having a brightness of 60 (for example, a pattern such as a square). It may be.

The brightness of the index pattern may be set manually by the user, but it is a parameter of a filter provided in the HMD or the like (a detection filter used for image processing or the like to detect the index pattern from the image captured by the camera). It may be set automatically according to. This is because it is sufficient that the brightness difference between the brightness of the wall and the brightness of the index pattern can be detected by the detection filter.

When the brightness of the index pattern is automatically set, the information storage device 1020 of the information processing device 1000 also stores information regarding the parameters of the detection filter provided in the HMD or the like. The pattern brightness determination unit 1040 reads information on the parameters of the detection filter from the information storage device 1020, and is an index that the detection filter can detect based on the information on the parameters and the wall brightness information obtained as described above. Determines the brightness of the parameter.

According to the third modification of the first embodiment, the brightness of the index pattern is determined in consideration of the brightness of the actual wall or floor on which the index pattern is arranged, so that the HMD can be easily detected while being used. It is possible to realize an index pattern that is more difficult for a person to see.

<Second embodiment>
In the second embodiment, it is used by considering the difference in brightness between the input / output device such as the input device (imaging device) such as the HMD camera and the output device such as the printing device of the pattern output unit 1060. An example of generating an index pattern that is more difficult to see for a person or the like will be given. In the following description, only the parts different from the configuration and the like of the first embodiment described above will be described.

FIG. 7 is a diagram showing a schematic configuration of the information processing apparatus 1000 that generates the index pattern of the second embodiment. The information processing device 1000 has a pattern color determining unit 1055 in addition to the configuration of the information processing device 1000 of the first embodiment described above. Further, in the case of the configuration of the second embodiment, the experience area image acquisition unit 50 is added.

In the second embodiment, the experience area image acquisition unit 50 acquires the information of the experience area image in which the experience area to which the index pattern will be given later is captured by the imaging device, and obtains the information of the experience area image. It is sent to the pattern size reference information input unit 100. In the case of the present embodiment, the imaging device that acquires the experience area image is an HMD camera.

The pattern size reference information input unit 100 sends the information of the experience area image acquired by the experience area image acquisition unit 50 to the information processing device 1000 in addition to the pattern size reference information described in the first embodiment. Since the pattern acquisition unit 1010 and the pattern size determination unit 1030 are the same as those described in the first embodiment, their description will be omitted.

In the case of the second embodiment, the information storage device 1020 stores the luminance value correction parameters used when correcting the luminance difference between the input / output devices in addition to the original pattern information and the feature detection threshold value described above. There is. The feature detection threshold value acquisition unit 1050 acquires each information of the feature detection threshold value and the brightness value correction parameter and sends the information to the pattern brightness determination unit 1040.

In the case of the second embodiment, the pattern luminance determination unit 1040 includes the index pattern information in which the pattern size Sα is determined as described above from the pattern size determination unit 1030, and the pattern size reference information including the experience area image. To receive. Further, the pattern luminance determination unit 1040 receives the above-mentioned feature detection threshold value and the luminance value correction parameter used when correcting the luminance difference between the input / output devices from the feature detection threshold value acquisition unit 1050.

The pattern luminance determination unit 1040 in the case of the second embodiment determines the luminance of the index pattern based on the received information. Specifically, the pattern luminance determination unit 1040 first obtains the luminance of the index pattern in the same manner as described in the first embodiment described above. Then, in the case of the second embodiment, the pattern luminance determination unit 1040 further corrects the luminance of the index pattern by the luminance value correction parameter corresponding to the experience region image, and determines it as the luminance of the indicator pattern.

Here, the index pattern is output from the pattern output unit 1060 including output devices such as a printing device and a projection device, is given to the experience area, and is photographed by the camera of the HMD which is the input device. Therefore, the brightness of the index pattern output from the pattern output unit 1060 and the brightness of the index pattern in the image captured by the HMD camera are values that depend on the device characteristics of the input / output devices, and are the values of the input / output devices. There is a possibility that the brightness shift due to the characteristics. Therefore, in the second embodiment, in order to correct the brightness deviation of the index pattern due to the input / output device, the brightness of the index pattern obtained by the pattern brightness determining unit 1040 in the same manner as in the first embodiment is used in the experience area image. Correction is performed based on the corresponding brightness value correction parameters.

As the luminance value correction parameter, those generated in advance may be used, or the luminance value correction parameter generated each time the MR is provided may be used. The brightness value correction parameter is, for example, the brightness between the image obtained by assigning a confirmation pattern whose brightness value is determined to a predetermined value to the experience area and capturing the confirmation pattern and the predetermined brightness value of the confirmation pattern. It can be generated as a parameter according to the difference. Therefore, when the index pattern whose brightness value is corrected based on the brightness value correction parameter according to the experience area image is given to the experience area, the brightness difference between the brightness of the experience area and the brightness of the index pattern is the minimum. It becomes a pattern that is difficult for users to see. The information of the index pattern whose brightness is determined as described above is sent to the pattern color determination unit 1055 together with the information of the experience area image.

The pattern color determination unit 1055 determines the color of the index pattern based on the brightness of the index pattern received from the pattern brightness determination unit 1040 and the information of the experience area image acquired by the experience area image acquisition unit 50. For example, as shown in FIG. 8, the hue is calculated for each pixel from the RGB component of the experience region image to generate a histogram, and the most frequent hue in the histogram is determined as the main hue of the experience region image. do. Then, the pattern color determination unit 1055 determines the color obtained by combining the determined hue and the brightness of the index pattern as the color of the index pattern. As a result, the index pattern becomes a pattern that is difficult for the user or the like to visually recognize when it is given to the experience area. The hue may be determined not only by using one experience region image but also by using a plurality of experience region images taken in time series. Further, although an example of obtaining the hue from the RGB component is given here, the hue may be obtained by another method such as a Lab (Lab color space) color space.

FIG. 9 is a flowchart showing the flow of processing from the acquisition of the pattern size reference information by the pattern size reference information input unit 100 and the information processing device 1000 of FIG. 7 of the second embodiment to the image generation and output of the index pattern. .. The processing of the flowchart of FIG. 9 is started, for example, when the administrator of the information processing device 1000 or the like instructs the information processing device 1000 to generate an index pattern. Here, only the processing different from the flowchart of FIG. 4 of the first embodiment described above will be described.

In the flowchart of FIG. 9, in ST300, the pattern size reference information input unit 100 acquires the pattern size reference information including the information of the experience area image. In ST330, the feature detection threshold value acquisition unit 1050 acquires the feature detection threshold value and the above-mentioned luminance value correction parameter. In ST340, the pattern luminance determination unit 1040 determines the luminance of the index pattern based on the feature detection threshold value and the luminance value correction parameter described above. After ST340, the information processing apparatus 1000 proceeds to ST345 performed by the pattern color determination unit 1055. In ST345, the pattern color determination unit 1055 determines the color of the index pattern based on the brightness of the index pattern and the experience area image. After ST345, the information processing apparatus 1000 proceeds to ST350 performed by the pattern output unit 1060. After ST350, the information processing apparatus 1000 ends the processing of the flowchart of FIG.

The processing order of ST300 and ST310 may be changed, the processing order of ST320 to ST345 may be changed as appropriate, or they may be executed in parallel.

As described above, according to the second embodiment, the luminance value is corrected based on the luminance value correction parameter in consideration of the luminance deviation in the input / output device, and the corrected luminance value and the hue of the experience area image are obtained. It is possible to generate an index pattern of a color based on this. The index pattern in the second embodiment can be detected as a natural feature from the captured image of the experience area in the real space, and the natural feature is not only the pattern of the index pattern but also the boundary between the index pattern and the experience area. Detection is possible. Further, since the index pattern of the second embodiment is a pattern that is difficult for the user or other third party to see, the landscape of the real space is not impaired.

<Modification 1 of the second embodiment>
In the second embodiment, the pattern color determination unit 1055 determines the main hue of the experience region image by using the hue histogram, but the method of determining the main hue of the experience region image is the above-mentioned example. It is not limited to. For example, the experience region image may be divided into regions according to the hue, and the hue of the largest region may be determined as the main hue of the experience region image.

<Example of realization by computer>
Each part of FIG. 1 of the first embodiment and FIG. 7 of the second embodiment described above can be realized by either a hardware configuration or a software configuration. For example, the CPU of the computer may be made to execute the information processing program related to the processing of the flowchart of the first or second embodiment described above.

FIG. 10 is a diagram showing a configuration example of a computer provided with a CPU capable of executing the information processing program of the present embodiment. In FIG. 10, the CPU 4001 controls the entire computer using the programs and data stored in the RAM 4002 and the ROM 4003, and also executes the information processing program related to the processing of the first and second embodiments described above. .. The computer of FIG. 10 can perform not only the case of executing the information processing program related to the processing of the first and second embodiments but also the processing for providing the MR to the user. Therefore, FIG. 10 gives an example in which the HMD4100 is also connected to the computer.

The RAM 4002 has an area for temporarily storing various programs, data, and the like including the information processing program of the present embodiment loaded from the external storage device 4007 and the storage medium drive 4008. The RAM 4002 also has an area for temporarily storing data received from an external device via the I / F (interface) 4009. In the case of this example, the external device is a printing device or projection device that prints an image of the index pattern generated as described above, a camera that acquires the experience area image described above, or the like. Further, the RAM 4002 also has a work area used by the CPU 4001 to execute each process. That is, the RAM 4002 can appropriately provide various areas. The ROM 4003 stores computer setting data, a boot program, and the like.

The keyboard 4004 and the mouse 4005 are examples of operation input devices, and various instructions can be input to the CPU 4001 by being operated by a computer user. Of the above-mentioned pattern size reference information, information that can be input by the user is input by the user via the keyboard 4004 and the mouse 4005. The display unit 4006 is composed of a display, and can display the processing result by the CPU 4001 as an image, characters, or the like.

The external storage device 4007 is a large-capacity information storage device typified by a hard disk drive device. The external storage device 4007 stores an OS (operating system) and an information processing program and data for causing the CPU 4001 to execute each process according to the above-described embodiment described as being performed by the information processing device. The information processing program according to the present embodiment includes a program for executing each process of the flowchart of the first or second embodiment described above. Further, the data stored in the external storage device 4007 includes the original pattern, the feature detection threshold, the luminance value correction parameter data, and various other data as stored in the information storage device 1020 described above. There is.

The programs and data stored in the external storage device 4007 are appropriately loaded into the RAM 4002 under the control of the CPU 4001. The CPU 4001 executes each process performed by the information processing apparatus 1000 by executing each process described in the first or second embodiment described above using the loaded program or data.

The storage medium drive 4008 reads programs and data recorded on a storage medium such as a CD-ROM or a DVD-ROM, and writes programs and data to the storage media. A part or all of the programs and data described as being stored in the external storage device 4007 may be recorded in this storage medium. The programs and data read from the storage medium by the storage medium drive 4008 are output to the external storage device 4007 and the RAM 4002.

The I / F 4009 is provided by an analog video port for connecting an external device such as a camera described above, a digital input / output port such as IEEE1394, or a DVI port that outputs an image of an index pattern to the printing device or projection device described above. It is composed. Further, the data received via the I / F 4009 can be input to the RAM 4002 or the external storage device 4007. The pattern size reference information may be acquired via the I / F 4009. Further, when the computer of FIG. 10 provides the MR to the user, the HMD4100 is connected to the I / F4009. The bus 4010 is a bus that connects the above-mentioned parts.

<Other Embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The above-described embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

50 Experience area image acquisition unit, 100 pattern size reference information input unit, 1000 information processing device, 1010 pattern acquisition unit, 1020 information storage device, 1030 pattern size determination unit, 1050 pattern brightness determination unit, 1050 feature detection threshold acquisition unit, 1055 Pattern color determination unit, 1060 pattern output unit

Claims (19)

A brightness determining means for determining the brightness of a pattern given as an index to an experience area in real space based on a threshold value for detecting an image feature.
A color determining means for obtaining the main hue of the captured image from the hue histogram of each pixel of the captured image in the real space and determining the color of the pattern based on the main hue and the brightness.
A generation means for generating an image of the pattern with the determined brightness and color, and
An information processing device characterized by having. The information processing device according to claim 1, wherein the luminance determining means determines the luminance of the pattern based on a luminance threshold when detecting a feature due to the luminance of the image. The luminance determining means determines the luminance difference of the pattern of the pattern based on the luminance difference threshold when detecting the feature due to the luminance gradient of the image.
The information processing apparatus according to claim 2, wherein the generation means generates an image of the pattern having a pattern having the determined luminance difference. The information processing device according to claim 1, wherein the luminance determining means determines the luminance of the pattern based on the luminance of the experience area in the real space to which the pattern is applied. 4. The brightness determining means determines the brightness of the pattern based on the average value of the brightness of the area around the position where the pattern is applied in the experience area of the real space. The information processing device described in. The brightness determining means determines the brightness of the pattern based on the brightness of the experience area in the real space to which the pattern is applied and the parameters of the filter that detects the pattern from the captured image of the experience area in the real space. The information processing apparatus according to claim 4 or 5. A brightness determining means for determining the brightness of a pattern given as an index to an experience area in real space based on a threshold value for detecting an image feature.
The captured image is divided into regions for each hue according to the hue of each pixel of the captured image in the real space, and the main hue of the captured image is obtained based on the size of the divided regions. A color determining means for determining the color of the pattern based on the hue and the brightness.
A generation means for generating an image of the pattern with the determined brightness and color, and
Information processing apparatus characterized by having a. The information processing apparatus according to claim 7 , wherein the color determining means uses the hue of the largest region in the divided regions as the main hue of the captured image. Said color determining means according to claim 1, characterized in that the real space seeking hue becoming the main plurality of experience area images taken and determines the color of the pattern based on said said color intensity The information processing apparatus according to any one of 8 to 8. Claim 1, characterized in that the shortest distance and based on the longest distance between the imaging device for photographing the experience region and the physical space of the real space, further comprising a size determining means for determining the size of said pattern 9. The information processing apparatus according to any one of 9. The information processing device according to claim 10 , wherein the size determining means further determines the size of the pattern based on the angle of view and the resolution of the imaging device. The sizing means determines the size of a plurality of patterns for each pattern.
The information processing apparatus according to claim 10 , wherein the generation means generates an image in which the plurality of patterns whose sizes are determined for each pattern are aggregated. The information processing apparatus according to claim 12 , wherein the brightness determining means determines the brightness of each of the plurality of patterns for each pattern. The brightness determining means corrects the determined brightness based on the threshold value by a correction parameter.
The information processing apparatus according to any one of claims 1 to 13, wherein the generation means generates an image of the pattern having the corrected brightness. The pattern given to the experience area of the real space is used as an index when determining the position and orientation of the image pickup apparatus for photographing the real space, according to any one of claims 1 to 14. The information processing device described. It is an information processing method executed by an information processing device that generates an image of a pattern to be given to the experience area in the real space.
Based on the threshold value when detecting the feature of the image, the brightness of the pattern given as an index to the experience area in the real space is determined.
The main hue of the captured image is obtained from the hue histogram of each pixel of the captured image in the real space, and the color of the pattern is determined based on the main hue and the brightness.
An information processing method comprising generating an image of the pattern with the determined brightness and color. It is an information processing method executed by an information processing device that generates an image of a pattern to be given to the experience area in the real space.
Based on the threshold value when detecting the feature of the image, the brightness of the pattern given as an index to the experience area in the real space is determined.
The captured image is divided into regions for each hue according to the hue of each pixel of the captured image in the real space, and the main hue of the captured image is obtained based on the size of the divided regions. The color of the pattern is determined based on the hue and the brightness.
Generate an image of the pattern with the determined brightness and color
An information processing method characterized by the fact that. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 15. The information processing apparatus according to any one of claims 1 to 15.
A display device that displays a mixed reality image by fusing an image of the real space captured by the imaging means and an image of the virtual world generated by computer graphics on the display means.
An information processing system characterized by having.

Images