JP4745724B2 - Image processing method and image processing apparatus - Google Patents

Description

  The present invention relates to a technique for generating a composite image in which a virtual space image is superimposed on a real space image.

  With a live-action scene in the background, computer graphics (hereinafter referred to as CG) are superimposed on it and presented to the experiencer, and this experience is as if a virtual object exists on the spot. There is a mixed reality (MR) technology that makes

  In order to realize a rich experience using this MR technology, not only simply superimposing and displaying the CG on the background of the live-action scene, but also the actual user is drawn in CG. Interactions such as touching and operating a virtual object (hereinafter sometimes simply referred to as a virtual object) are becoming important. In order to realize such interaction, it is necessary to display the hands of an experienced person who operates the virtual object (hereinafter referred to as a subject) in front of the virtual object (foreground). This is because if the subject that should be in front of the virtual object is hidden by the virtual object, the sense of distance from the virtual object and the sense of reality will break down, impairing the sense of reality.

  In order to solve such a problem, the applicant proposed in Japanese Patent Application Laid-Open No. H11-228707 a technique for preventing the image of the subject to be the foreground from being hidden by the CG image. In this technique, a background and a subject are acquired as a live-action image, and a subject to be displayed in front of a CG image (region having color information as subject detection information) registered in the system manually from the live-action image in advance. Is extracted as a subject region, and drawing of a CG image is prohibited in the subject region. With this technique, the subject that should be the foreground is displayed so as to be in front of the virtual object without being hidden by the CG image, and a mixed reality experience with a high presence can be performed.

Further, as a conventional technique for synthesizing a real image and a computer graphics image, there is a technique in which a background region is colored with a specific color and this background region is replaced with a computer graphics image by chroma key composition. In such a conventional technique, the photographed image is limited to a subject other than the background.
JP 2003-296759 A

  The mixed reality experience system disclosed in Patent Document 1 operates well when there is almost no region having color information similar to the subject in the actual landscape viewed by the user. However, in a landscape where other users are present in the actual landscape viewed by the user, the CG image is drawn even to the subject area (hand or face) of the other user who should originally be behind the CG image. It is prohibited, and the subject area of the other experiencing person is displayed as if it is in front of the CG image, and the actual feeling of the experiencing person may be impaired.

  From this technical background, after extracting the subject area from the live-action image in order to extract only the hands of the experienced person or the specified subject area, the target area (such as the hands of the experienced person) is selected from the extracted area. It is desired to generate a mixed reality image by performing a process of not rendering a CG image only on a selected region.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for appropriately specifying a region in which a virtual space image is not superimposed in a real space image.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, an image processing method performed by an image processing apparatus, in which the acquisition unit included in the image processing apparatus includes a part of the observer in the real space where the observer and another person different from the observer exist. An acquisition step of acquiring a captured image obtained by capturing another person from the viewpoint of the observer, and a registration color that the registration unit of the image processing apparatus determines as a partial color of the observer in the captured image is registered. A registration step, an extraction unit included in the image processing device extracts an object region having the registered subject color in the acquired captured image, and a derivation unit included in the image processing device. a derivation step of obtaining a size of each of the extracted subject region, measuring means for the image processing apparatus has found a measuring step of measuring a distance between the other person and the observer, the image processing Determining means the device has, based on the distance, the image of the virtual space by the other person is not masked, the threshold of the size of the observer's subject region image of the virtual space by a part is masked And a generation unit included in the image processing apparatus regards the extracted subject area having a size larger than the determined threshold as a part of the observer, and uses the subject area as the virtual space. a generation step of generating a mask image for masking the image, superimposing means for the image processing apparatus has found to the captured image, and superimposing step of superimposing the image of the mask by the mask image the virtual space, the image An output unit included in the processing apparatus includes an output step of outputting a captured image on which the image of the virtual space is superimposed.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, an acquisition unit that acquires a captured image obtained by capturing a part of the observer and the other person from the viewpoint of the observer in a real space where the observer and another person different from the observer exist; Registration means for registering a subject color to be determined as a part of the color of the observer in the captured image, extraction means for extracting a subject area having the registered subject color in the acquired captured image, and the extraction Deriving means for determining the size of each of the subject areas, measuring means for measuring the distance between the observer and the other person, and masking the image of the virtual space by the other person based on the distance Sarezu, determining means and, the photographic said extracted larger size than the determined threshold image of the virtual space by a part of the observer to determine the threshold value of the size of the subject region to be masked Considers region and part of the observer, and generation means for generating a mask image for masking the image of the subject region by the virtual space, the captured image, the image of the virtual space that is masked by the mask image And superimposing means for superimposing and output means for outputting a captured image on which the image of the virtual space is superimposed.

  According to the configuration of the present invention, it is possible to appropriately specify an area in which an image in the virtual space is not superimposed on the real space image.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing a functional configuration of a system according to the present embodiment for allowing a user (experience person) to experience a mixed reality space in which a real space and a virtual space are fused. Further, in the system according to the present embodiment, when a virtual subject image is superimposed on a real space image and a predetermined subject is included in the real space image, the subject region is related. The superimposition process is controlled so that the image is always displayed in front of the virtual space image. In the present embodiment, “hand” is used as an example of the predetermined subject.

  As shown in FIG. 1, the system according to the present embodiment includes a video camera 101, an image input unit 102, a subject area extraction unit 103, a camera position and orientation sensor 104, a camera position and orientation measurement unit 105, an image generation unit 106, and a virtual space database. 107, an image composition unit 108, a head-mounted image display device (hereinafter referred to as HMD) 109, a subject color information registration unit 110, and a subject region selection unit 111.

  Moreover, FIG. 2 is a figure which shows a mode that the system which concerns on this embodiment is used. Hereinafter, the configuration of the system according to the present embodiment will be described with reference to FIGS.

  In addition, the experience person in this system assumes one person or multiple persons. In other words, in the situation where the experiencer 201 of FIG. 2 is seated on the chair 202, the experiencer 201 can experience a mixed reality space experience while the other experiencer 208 can be seen by the experiencer 201. .

  The video camera 101 captures a moving image in the real space, and is attached to the HMD 109 as shown in FIG. The attachment position of the video camera 101 is a position close to the eye position of the experience person 201 when the experience person 201 wears the HMD 109 on his / her head 203. In addition, the mounting direction of the video camera 101 is a direction that substantially matches the viewing direction of the experiencing person 201 when the experiencing person 201 wears the HMD 109 on his / her head 203. Therefore, the video camera 101 can be used to capture an image of a real space that can be seen according to the position and orientation of the eye of the experiencer 201. In the following description, the video camera 101 may be referred to as a “viewpoint”. Images of each frame (real space image) captured by the video camera 101 are sequentially input to the image input unit 102 shown in FIG. The image input unit 102 outputs the received image data to the image composition unit 108 and also outputs it to the subject region extraction unit 103.

  The camera position and orientation sensor 104 measures the position and orientation of the video camera 101, and is attached to the video camera 101 as shown in FIG. For example, a magnetic sensor, an optical sensor, an ultrasonic sensor, or the like can be applied to the camera position and orientation sensor 104. Since the position and orientation measurement technique using these sensors is well known, a description thereof will be omitted.

  A signal indicating the position and orientation of the video camera 101 measured by the camera position and orientation sensor 104 is input to the camera position and orientation measurement unit 105 shown in FIG. 1, and the video camera 101 is based on this signal by the camera position and orientation measurement unit 105. Data indicating the position and orientation of the. This data is output to the image generation unit 106 and also to the subject region selection unit 111.

  As described above, the HMD 109 includes the video camera 101 and the camera position / orientation sensor 104, and the HMD 109 is mounted on the head 203 of the experience person 201. In addition, as shown in FIG. 2, the experience person 201 is sitting on a chair 202. In the figure, it is assumed that the experience person 201 is sitting on a chair 202, but even if the experience person 201 takes an arbitrary posture, the essence of the following description is not particularly changed.

  Returning to FIG. 1, the image generation unit 106 uses the data received from the camera position / orientation measurement unit 105 and the data (virtual space data) stored in the virtual space database 107 to display a virtual space that can be viewed from the video camera 101. Generate an image of The virtual space database 107 stores data necessary for generating an image of each virtual object constituting the virtual space. For example, when the virtual object is composed of polygons, the data of each polygon is stored. Normal vector data, polygon color data, coordinate data of each vertex constituting the polygon, and texture data are stored in the virtual space database 107 when texture mapping is performed on a virtual object. It will be.

  In the case of FIG. 2, since 207 is a virtual object, data for generating an image of the virtual object 207 is stored in the virtual space database 107. In this case, the image generation unit 106 generates an image of the virtual object 207 that can be seen from the video camera 101. Since a technique for generating an image of a virtual space that can be seen from a viewpoint having a predetermined position and orientation is well known, a description thereof will be omitted.

  The virtual space image data generated by the image generation unit 106 is output to the image synthesis unit 108. The image composition unit 108 generates a composite image obtained by superimposing the virtual space image received from the image generation unit 106 on the real space image received from the image input unit 102, that is, an image of the mixed reality space. Output to. As a result, an image of the mixed reality space that is visible according to the position and orientation of the viewpoint is displayed in front of the experiencer 201.

  Here, when the user 201 observes the mixed reality space image via the HMD 109 and his / her hand 205 or the hand 210 of another user 208 enters the field of view of the video camera 101, naturally. The hands 205 and 210 are picked up by the video camera 101, and the hands 205 and 210 are displayed on the mixed reality space image displayed on the HMD 109.

  In such a case, since it is preferable for the observer 101 to have only his / her hand 205 as the foreground, it is necessary not to superimpose the virtual space image on the region of the hand 205 on the real space image. It is. In the case of FIG. 2, when the experience person 201 observes the mixed reality space image via the HMD 109, only the hand 205 of the real object is in the foreground, the other real objects 206 a and 206 b, and the hands of other experience persons 208. The background is 210.

  As a configuration for this purpose, the system according to the present embodiment includes a subject region extraction unit 103, a subject color information registration unit 110, and a subject region selection unit 111. Below, each part of the subject region extraction unit 103, the subject color information registration unit 110, and the subject region selection unit 111 will be described.

  In the subject color information registration unit 110, color data (color information) indicating the color of the hand 205 is registered in advance. Here, this color data will be described. The color data registered in the subject color information registration unit 110 can be described as coordinate values in a multidimensional color space. There are various well-known types of color systems such as RGB, YIQ, YCbCr, YUV, HSV, Lu * v *, La * b * (Japanese Standards Association JIS Color Handbook).

  An appropriate one may be used arbitrarily according to the color characteristics of the target subject, but in order to cancel the change in the color characteristics of the subject due to the difference in lighting conditions, the information is separated into luminance information and hue information. It is desirable to use a color system and use only hue information. Typical examples of such a color system are YIQ and YCbCr. In this embodiment, the YCbCr color system is used.

  FIG. 4 is a diagram showing an outline of the color distribution in the CbCr plane of the YCbCr color system. The horizontal axis 401 is Cr and the vertical axis 402 is Cb. The center 403 is a white region, and the saturation increases from the center toward the periphery.

  FIG. 5 is a diagram illustrating an example in which the color information of the subject is expressed as a color space coordinate distribution 501 on the CbCr plane of the YcbCr color system.

  It is desirable that the subject color information is not included in the background color information, but depending on the conditions, it may be difficult to distinguish the subject and the background by color. In such a case, a convenient color may be assigned to the subject, for example, by wearing colored gloves on the hand.

  As a color information registration method in the subject color information registration unit 110, an image of the subject is acquired by imaging the subject with a camera or the like, and pixel values of each pixel constituting the acquired image are distributed on the CbCr plane. A method of specifying the coordinate range of each axis of the color space, a method of sampling the color space for each axis, and registering the value of whether or not each point is a subject, a so-called look-up table Various methods are conceivable, such as using.

  Returning to FIG. 1, the subject region extraction unit 103 refers to each pixel value constituting the image received from the image input unit 102, and “the color of the hand 205” indicated by the data registered in the subject color information registration unit 110. It is determined that a pixel having a pixel value indicating substantially the same color as “a pixel constituting the region of the hand 205”. More specifically, it is determined whether each pixel constituting the image received from the image input unit 102 is a pixel constituting the region of the hand 205, and a determination result is assigned to each pixel. For example, “1” is assigned to the pixels constituting the hand 205 area, and “0” is assigned to the pixels not constituting the hand 205 area.

  FIG. 3 shows the region of the hand 205 from the image 301 when a real space image 301 in which only a hand 205 is a real object having substantially the same color as the hand 205 is obtained from the image input unit 102. It is a figure explaining the process to extract.

  When a process for checking whether or not each pixel constituting the image 301 constitutes the region of the hand 205 is performed, a value of “1” or “0” is assigned to each pixel. An image can be generated. In the image 302, an area indicated by “white” is an area constituted by a pixel group to which “1” is assigned, and is a pixel group constituting the area of the hand 205. On the other hand, an area indicated by “black” in the image 302 is an area constituted by a pixel group to which “0” is assigned, and is a pixel group that does not constitute the area of the hand 205. Such an image 302 is a so-called mask image. Thus, by generating this mask image, the region of the hand 205 in the image of the real space can be extracted.

  FIG. 6 is a flowchart of processing performed by the subject area extraction unit 103. The flowchart of FIG. 6 is a flowchart of the process for the pixel whose image coordinates are at the position (i, j) in the image received from the image input unit 102. Processing according to the flowchart is performed for each pixel constituting the image received from the image input unit 102.

  First, the RGB value of the pixel whose image coordinates are (i, j) in the image received from the image input unit 102 is converted into a YCrCb value (step S601). If the R value of the pixel at the image coordinate (i, j) is R (i, j), the G value is G (i, j), and the B value is B (i, j), the RGB value is YCrCb. R (i, j), G (i, j), and B (i, j) are converted by a function color_conversion () for converting to a value to obtain a Y value, a Cr value, and a Cb value.

  Next, it is determined whether or not the color represented by each value of Y, Cr, and Cb obtained in step S601 is substantially the same as the color indicated by the color data registered in the subject color information registration unit 110. . For example, in the case of the figure, is the color represented by the values of Y, Cr, and Cb obtained in step S601 substantially the same as the color indicated by the color data registered in the subject color information registration unit 110? Whether or not is determined by the function mask_func (). Then, the determination result is assigned to the array mask (i, j). This array mask (i, j) is for storing a value indicating whether or not the pixel whose image coordinates are at the position (i, j) constitutes the subject area. This is for storing the pixel value (1 or 0) of the pixel whose image coordinates are (i, j) in the image.

  As a determination method using the function mask_func (), for example, coordinate values (Cr, Cb) on the CbCr plane defined by Cb and Cr are regions of the subject color distribution 501 registered in the subject color information registration unit 110. It is judged whether it belongs to. The determination result may be expressed as a binary value, for example, 1 if it belongs to the subject color distribution 501 and 0 if it does not belong, but the degree of belonging is expressed by a continuous value from 0 to 1. Also good.

  In the present embodiment, the pixel value of each pixel constituting the image received from the image input unit 102 is represented by RGB, but may be represented by YIQ or YUV. The processing in step S601 is omitted, and the coordinate values in IQ space and UV space may be used instead of (cb, cr) in step S602.

  As described above, the subject area extraction unit 103 generates a mask image indicating whether or not each pixel constituting the image received from the image input unit 102 constitutes a “hand” area.

  However, here, the subject area extraction unit 103 has a pixel value “1” on the mask image for a pixel having substantially the same color as the “color of the hand 205” in the image in the real space. When a real object having substantially the same color as that of the hand 205 is included in the image other than the hand 205, the pixels constituting the real object region also have a pixel value “1” on the mask image. Note that In such a case, a plurality of white areas are scattered in the mask image.

  Accordingly, the subject region selection unit 111 performs processing for specifying which region of the white region in the mask image generated by the subject color information registration unit 110 is the region of the hand 205. Details of this determination processing will be described later.

  Then, when the subject region selection unit 111 specifies the region of the hand 205 in the mask image, this region is changed to a white region. That is, the pixel value of the pixels constituting the region of the hand 205 is set to “1”. Further, a region other than the region of the hand 205 is set to a black region. That is, the pixel value of the pixels constituting the area other than the hand 205 area is set to “0”. As a result, in the mask image, the pixel value of each pixel constituting the hand 205 region is “1”, and the pixel value of each pixel constituting the region other than the hand 205 region is “0”. That is, with this mask image, the region of the hand 205 and the region other than the hand 205 can be distinguished by the pixel value of each pixel constituting the mask image. The mask image generated in this way is output to the image composition unit 108.

  FIG. 7 illustrates that when a virtual space image is superimposed on a real space image, the virtual space image is not superimposed on the region of the hand 205 using the mask image generated by the subject region selection unit 111. It is a figure explaining the process for this. Such processing is performed by the image composition unit 108.

  In the figure, reference numeral 301 denotes an image received from the image input unit 102. In the image 301, the hand 205 of the experience person 201 is shown. On the other hand, reference numeral 701 denotes a virtual space image generated by the image generation unit 106. Here, the image composition unit 108 refers to the mask image 302 when the image composition unit 108 superimposes the image 701 on the image 301. That is, when the image 701 is superimposed on the image 301, a portion corresponding to the hand region (region shown in white) in the image 302 in the image 701 is not drawn on the image 301. Thereby, as shown in the image 702, since the image of the virtual space is not always drawn on the region of the hand 205, the region of the hand 205 can always be the foreground.

  Processing performed by the image composition unit 108 will be described with reference to FIG. 10 showing a flowchart of the processing. The flowchart in FIG. 10 is a flowchart of processing for a pixel whose image coordinates are (i, j) in the image received from the image input unit 102 and the image received from the image generation unit 106. The image composition unit 108 performs the processing according to the flowchart of FIG. 10 for all the ranges that i and j can take.

  In step S100, the pixel data real (i, j) of the pixel having the image coordinate (i, j) in the real space image received from the image input unit 102 is transferred to the frame memory buffer (i, j). . Next, in step S200, the pixel data mask (i, j) of the pixel whose image coordinates are (i, j) in the mask image generated by the subject area selection unit 111 is stored in the image memory for mask processing. Transfer to a stencil buffer.

  In step S300, the pixel data CGI (i, j) of the pixel whose image coordinates are (i, j) in the image of the virtual space received from the image generation unit 106 is transferred to the frame memory buffer (i, j). In this case, the pixel data mask (i, j) in the stencil buffer is referred to.

  If mask (i, j) = 0, real (i, j) is pixel data constituting an area outside the area of the hand 205, and therefore pixel data CGI (i, j) of the image in the virtual space. Is transferred to the frame memory buffer (i, j). Thereby, the frame memory buffer (i, j) stores the pixel data CGI (i, j) of the virtual space image superimposed on the pixel data real (i, j) of the real space image. Will be.

  On the other hand, if mask (i, j) = 1, real (i, j) is the pixel data constituting the area of the hand 205, so the pixel data CGI (i, j) of the image in the virtual space is It is not transferred to the frame memory buffer (i, j). As a result, only the pixel data real (i, j) of the real space image is stored in the frame memory buffer (i, j), and the pixel data CGI (i, j) of the virtual space image is stored. Are not superimposed.

  The image composition unit 108 outputs the image stored in the frame memory in this way to the HMD 109 as a mixed reality space image.

  Next, a process in which the subject region selection unit 111 refers to the mask image generated by the subject region extraction unit 103 and determines which of the white regions in the mask image is the hand 205 region. explain.

  As described above, when a real object having substantially the same color as the color of the hand 205 is included in the image of the real space, the subject area extraction unit 103 also determines the pixels constituting the real object area as follows. Since it is determined that the pixel constitutes the region of the hand 205, a plurality of white regions are scattered in the mask image. When a mixed reality space image is generated using such a mask image, some portions corresponding to white areas in the mask image in the virtual space image are not drawn on the image in the real space. However, since only the area of the hand 205 as the object needs to be the foreground, it is necessary to process the area other than the area of the hand 205 as a black area in the mask image.

  For this purpose, FIG. 8 is a diagram illustrating a process for making a region other than the region of the hand 205 a black region in the mask image. In the case of the figure, noise is generated in the mask image 801 on the outside and inside of the subject area (the area of the hand 205) 850. The outside noise is an area that is misrecognized as the hand 205 in the real space, and the inside noise is an area that is recognized not to be the subject area 850 even though it is the subject area 850.

  In step S1, first, noise outside is removed (a white area outside the subject area 850 is corrected to a black area) to obtain a first noise-removed image 802. For example, the area of the white region (the number of pixels constituting the white region) is obtained using a labeling technique often used in calculating the area of the connected region in the image, and the obtained area is equal to or less than a predetermined threshold value. If there is, it is determined that the white area is not the subject area 850, and the pixel value of each pixel constituting the white area is set to “0”.

  Here, this “predetermined threshold value” is obtained in advance for presenting the mixed reality space image to the experience person 201, and is obtained as follows, for example.

  First, the experienced person 201 makes a fist with his hand 205 while putting the HMD 109 on his / her head 203, and puts out the hand with his hand extended. Then, the video camera 101 captures the state. The image captured by the video camera 101 is an image that can be seen when the fist made with the hand 205 of the experiencer 201 is viewed from the video camera 101.

  The operator inputs the real space image captured by the video camera 101 to the computer and displays it on the display screen of the display device of the computer. Then, this area is manually designated by tracing the area of the fist on the displayed image with the mouse. The number of pixels constituting the designated area is counted by the CPU of the computer, and the counted number is set as the “predetermined threshold value”. The obtained “predetermined threshold value” is stored in the memory.

  At this time, the average value (Cb, Cr value) of each pixel constituting the region selected by the operator is also acquired and registered in the subject color information registration unit 110. As a result, higher-quality subject color information can be registered.

  In the above description, the operator manually specifies the fist region, but the position and orientation of the viewpoint may be fixed, and the area and color information of the fist region may be extracted based on the background difference.

  As described above, the predetermined threshold value can be obtained. Therefore, in step S1, a white area smaller than the size of the fist is regarded as not being an area of the hand 205, and such a white area is removed.

  Thereby, as shown in the first noise-removed image 802, white areas scattered outside the subject area 850 can be removed. Note that if there is no black area in the subject area 850 in the mask image, the processing after step S2 is not necessary.

  Next, in step S2, the pixel value of each pixel constituting the first noise-removed image 802 obtained in step S1 is inverted (a pixel value of “0” is inverted to “1”, and the pixel value is “ “1” is inverted to “0”). Thereby, a bit inverted image 803 is obtained.

  Next, in step S3, labeling is performed again on the bit inverted image 803 obtained in step S2. As a result, the bit-reversed image 803 has a region having the largest area (a region outside the subject region 850 in the bit-reversed image 803 (excluding white regions scattered within the subject region 850)), and other regions. are categorized. Accordingly, the region other than the region having the largest area is regarded as the subject region 850, and the pixel value of each pixel constituting the region other than the region having the largest area is set to “0”. Thereby, the 2nd noise removal image 804 can be obtained. As shown in the second noise-removed image 804, white areas scattered in the subject area 850 can be removed.

  In step S4, the pixel value of each pixel constituting the second noise-removed image 804 obtained in step S3 is inverted (a pixel value of “0” is inverted to “1”, and the pixel value is “1”). "Is reversed to" 0 "). Thereby, a noise-removed image 805 is obtained.

  Therefore, by using such a noise-removed image 805 as a mask image, when the image composition unit 108 superimposes the virtual space image on the real space image, the virtual space image is not superimposed on the region of the hand 205. Can be.

  In the processing example shown in FIG. 8, the processing for specifying the region of the hand 205 is performed by setting a threshold value for the area. However, when the number of subjects is a predetermined number N, When the white areas in the mask image are viewed in order of increasing area, it may be determined that the top N areas are subject areas and the other subject areas are deleted.

  Also, a threshold value may be separately set for an area generated in the skin color area, and an area that is lower than the threshold value may be regarded as noise generated in the hand area and may be deleted. However, in this case, it is necessary to set a sufficiently small threshold value for the hand region.

  In this embodiment, a white region other than the region of the hand 205 in the mask image is removed by adopting a general labeling algorithm. However, a median filter other than labeling, reduction / expansion processing, and the like are used. A white region other than the subject region may be removed using an algorithm or a convex closing process for the subject region candidate, and a suitable means may be arbitrarily selected.

  Further, in the above processing, the processing for specifying the region of the hand 205 in the mask image is performed based on the area of the white region, but matching is performed at the center of gravity position of the left and right subject regions using the principle of binocular stereo. By calculating the depth information, the subject area may be selected by comparing with the information obtained by the camera position / orientation measurement unit 105. Since the method for estimating the depth by binocular stereo is a known technique, the details are omitted.

  Usually, the quality of the presented image can be improved by removing noise by giving a certain threshold value as in the above-described means.

  In the present embodiment, as described above, the “hand” is used as the subject, but it goes without saying that other subjects may be used as the subject.

  FIG. 12 illustrates image processing having functions of an image input unit 102, a subject region extraction unit 103, an image generation unit 106, a virtual space database 107, an image composition unit 108, a subject color information registration unit 110, and a subject region selection unit 111. It is a block diagram which shows the hardware constitutions of an apparatus. As this image processing apparatus, for example, a general PC (personal computer) or WS (workstation) can be applied.

  A CPU 1201 controls the entire apparatus using programs and data stored in the RAM 1202 and the ROM 1203, and executes processing according to the flowchart shown in FIG.

  Reference numeral 1202 denotes a RAM, an area for temporarily storing programs and data loaded from the external storage device 1206, and an area for temporarily storing data received from the outside via an I / F (interface) 1207. Various areas such as a work area used when the CPU 1201 executes various processes and the frame buffer and stencil buffer can be provided as appropriate.

  A ROM 1203 stores setting data, a boot program, and the like of the apparatus.

  An operation unit 1204 includes a keyboard, a mouse, and the like, and various instructions can be input to the CPU 1201 by an operator of the apparatus.

  A display unit 1205 includes a CRT, a liquid crystal screen, and the like, and can display a processing result by the CPU 1201 using an image, text, or the like.

  Reference numeral 1206 denotes an external storage device, which is a large-capacity information storage device typified by a hard disk drive device. Here, an OS (operating system), a program for causing the CPU 1201 to execute processing according to the flowchart shown in FIG. Data etc. are saved. The virtual space database 107 and the subject color information registration unit 110 are also provided in the external storage device 1206. These are loaded into the RAM 1202 according to the control of the CPU 1201 as necessary.

  Reference numeral 1207 denotes an I / F, to which the HMD 109, the video camera 101, the camera position / orientation measurement unit 105, and the like are connected. The HMD 109, the video camera 101, the camera position / orientation measurement unit 105, and the like are connected to the apparatus via the I / F 1207. And data communication. In the figure, the number of I / Fs is one, but an I / F may be provided for each of the HMD 109, the video camera 101, and the camera position / orientation measurement unit 105.

  Reference numeral 1208 denotes a bus connecting the above-described units.

  FIG. 11 is a flowchart of the process described above for generating the mixed reality space image and outputting it to the HMD 109. Since the processing in each step is as described above, it will be briefly described here.

  First, since the real space image captured by the video camera 101 is sequentially input into the apparatus via the I / F 1207 for each frame, the CPU 1201 stores the input image data in the RAM 1202 or the external storage device 1206. (Step S1101).

  On the other hand, since the data indicating the position and orientation of the video camera 101 measured by the camera position and orientation sensor 104 is input into this apparatus via the I / F 1207, the CPU 1201 stores the input data indicating the position and orientation in the RAM 1202 or an external storage. It stores in the device 1206 (step S1102).

  Next, the CPU 1201 uses the virtual space data stored in the virtual space database 107 provided in the external storage device 1206 to generate and arrange each virtual object constituting the virtual space. Then, an image that is seen when such a virtual space is viewed from the viewpoint having the position and orientation acquired in step S1102 is generated (step S1103).

  Next, the CPU 1201 functions as the subject area extraction unit 103, and generates a mask image indicating an area having substantially the same color as the hand 205 in the real space image (step S1104).

  Next, the CPU 1201 performs processing for specifying the region of the hand 205 and the region outside the region of the hand 205 among the regions extracted as regions having the same color as the hand 205 in the generated mask image (step S1105). ). More specifically, for example, in the mask image, the pixel value of each pixel constituting the inside of the hand 205 area is set to “1”, and the pixel value of each pixel constituting the outside of the hand 205 area is set to “0”. .

  Then, by using the mask image generated in step S1105 and superimposing the virtual space image generated in step S1103 on the real space image acquired in step S1101, the mixed reality with the region of the hand 205 as the foreground is provided. A spatial image is generated (step S1106).

  The generated mixed reality space image data is output to the HMD 109 via the I / F 1207 (step S1107).

<Modification>
In the threshold value determination method, as soon as the area of the experiencer's hand disappears from the field of view, the area disappears from the field of view and the reality of the user may be impaired. Therefore, in order to reduce the phenomenon caused by the fixed threshold value, the threshold value is dynamically determined using the positional information of the experience person 201 and the other experience person 208.

  First, the positions of the heads of the experiencer 201 and other experiencers 208 are measured by the camera position / orientation measurement unit 105. Then, the distance between the two persons is calculated from the measured position information, and the area threshold value is dynamically changed according to the calculated distance. The threshold value is the maximum value of the fist size (area on the image) with the hand of the experiential person 201 extended, and the minimum value is the environment in which the system operates (how much color information is similar to that of the subject area in the background). The operator decides depending on whether or not

  Here, the processing beyond obtaining the maximum and minimum threshold values is based on the following assumptions.

  That is, when the skin color region of another experiencer 208 is viewed from the experiencer 201 in a state where the experiencer 201 and the other experiencer 208 are sufficiently separated from each other, the skin color is larger than the fist size of the experiencer 201 The area of the region does not increase. In other words, when the distance between the experiencer 201 and the other experiencer 208 is sufficiently large, the size of the fist of the experiencer 201 is set as a noise threshold (maximum value), and a skin color region smaller than that is regarded as noise. Can be considered.

  Regarding the threshold control method, the threshold is gradually increased when the distance between the experience person 201 and the other experience person 208 approaches. At this time, as a relational expression between the distance and the threshold, for example, an expression for linearly changing the threshold by applying a coefficient to the distance, an expression for changing the threshold nonlinearly in proportion to the square of the distance, and the like are conceivable.

  Further, by using the position information of the camera position / orientation measurement unit 105, it is determined whether or not another experience person exists in the field of view of the experience person. An assumption may be made that the area is the subject area, and a threshold set by the operator may be adopted.

  A subject area having an area equal to or smaller than the threshold value can be regarded as noise from the image information, but it is difficult to determine whether the subject area having an area equal to or larger than the threshold value is noise or the subject area of the user.

  For example, when the subject is in the hands of an experienced person, the face of another experienced person appears larger on the video camera 101 than the hands of the experienced person.

  FIG. 9 is a diagram for explaining the distribution of areas determined as subjects only from color information when other experienced persons are viewed from the field of view of the experienced persons, and display areas and non-display areas. The hand 901 on the right side of FIG. 9 is the hands of the experienced person, and the hands of the experienced person are areas to be displayed. 9 is an area where the heads 902 of other experiencers displayed on the left side of FIG. 9 and the hands and shoes 903 (colors of feet in the case of women) of other experiencers are not displayed.

  In order to extract a subject area from an image acquired by the video camera 101 and selectively remove a non-display area, it cannot be easily removed only by image information. Therefore, in addition to image information, position and orientation information is also used.

  The position and orientation information of the head of another experiencer is acquired from the camera position and orientation measurement unit 105, and a sphere or egg-shaped mask object is placed in the three-dimensional space so as to cover the head of the other experiencer. . As the shape of the mask object to be arranged at this time, any suitable shape may be selected according to the system construction environment. Another experience person's head can mask this area in the subject area by the experience object's mask object. Therefore, the subject area generated on the head of another experiencer can be removed by the mask object placed on the face part of the other experiencer.

  In this modification, the mask object is placed only on the head of another experiencer. However, in a system environment in which the position and orientation sensor can be attached to the hand of another experiencer, the other experiencer's hand is covered. Arrange the mask object like this. This makes it possible to robustly detect the hands of other users and make them non-display areas.

<Modification 2>
Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the function structure of the system which concerns on this embodiment for making a user (experience person) experience the mixed reality space which physical space and virtual space united. It is a figure which shows a mode that the system which concerns on embodiment of this invention is used. When a real space image 301 in which only the hand 205 is a real object having substantially the same color as the color of the hand 205 is obtained from the image input unit 102, a process of extracting the region of the hand 205 from the image 301 is performed. It is a figure explaining. It is a figure which shows the outline | summary of the color distribution in the CbCr plane of a YCbCr color system. It is a figure which shows the example which expressed the color information of a to-be-photographed object as the color space coordinate distribution 501 in the CbCr plane of a YcbCr color system. 5 is a flowchart of processing performed by a subject area extraction unit 103. When superimposing the virtual space image on the real space image, a process for preventing the virtual space image from being superimposed on the hand 205 region using the mask image generated by the subject region selection unit 111. It is a figure explaining. It is a figure explaining the process for making area | regions other than the area | region of the hand 205 into a black area | region in a mask image. It is a figure explaining the distribution of the area | region judged as a to-be-photographed object only from the color information at the time of seeing another experienced person from a visual field of an experienced person, a display area, and a non-display area. 5 is a flowchart of processing by the image composition unit. 10 is a flowchart of a process for generating a mixed reality space image and outputting it to the HMD 109; Hardware of an image processing apparatus having functions of an image input unit 102, a subject region extraction unit 103, an image generation unit 106, a virtual space database 107, an image composition unit 108, a subject color information registration unit 110, and a subject region selection unit 111 It is a block diagram which shows a structure.

Claims (6)

An acquisition means for acquiring a captured image obtained by capturing a part of the observer and the other person from the viewpoint of the observer in a real space where the observer and another person different from the observer exist;
Registration means for registering a subject color to be determined as a partial color of the observer in the captured image;
Extraction means for extracting a subject area having the registered subject color in the acquired captured image;
Derivation means for determining the size of each of the extracted subject areas;
Measuring means for measuring the distance between the observer and the other person ;
Determining means for determining a threshold value of a size of a subject area in which the image of the virtual space is not masked by the other person and the image of the virtual space is masked by a part of the observer based on the distance;
Generating means for considering the extracted subject region having a size larger than the determined threshold as a part of the observer, and generating a mask image for masking the image of the virtual space by the subject region ;
In the captured image, and superimposing means for superimposing the image of the masked the virtual space in the mask image,
An image processing apparatus comprising: output means for outputting a captured image on which an image of the virtual space is superimposed.   The image processing apparatus according to claim 1, wherein the generation unit generates a mask image that prohibits superimposition of an image in the virtual space on a subject area having a size larger than the threshold value.   The image processing apparatus according to claim 1, wherein the determination unit calculates the threshold value such that the threshold value increases as the distance approaches. Position measuring means for measuring the position of a part of the other person ;
An arrangement means for arranging a mask object at the measured position in the virtual space;
4. The superimposing unit superimposes an image in a virtual space on the captured image, with the region where the mask object is arranged in the captured image being a region on which the virtual image may be superimposed. The image processing apparatus according to any one of the above.   The computer program for functioning a computer as each means which the image processing apparatus of any one of Claims 1 thru | or 4 has. An image processing method performed by an image processing apparatus,
The acquired image possessed by the image processing apparatus is a captured image obtained by capturing a part of the observer and the other person from the viewpoint of the observer in a real space where the observer and another person different from the observer exist. An acquisition process for acquiring
A registration step in which a registration unit included in the image processing apparatus registers a subject color to be determined as a partial color of the observer in the captured image;
Extraction means for the image processing apparatus has, in the acquired the captured image, an extraction step of extracting a subject region with the registered subject color,
A deriving step in which a deriving unit included in the image processing apparatus obtains the size of each of the extracted subject areas;
A measuring step that the image processing apparatus has, a measuring step of measuring a distance between the observer and the other person ,
Based on the distance, a determination unit included in the image processing apparatus is configured to determine a subject area in which the virtual space image is not masked by the other person and the virtual space image is masked by a part of the observer . A determining step for determining a size threshold;
A mask image in which the generation unit included in the image processing apparatus regards the extracted subject area having a size larger than the determined threshold as a part of the observer, and masks the image in the virtual space with the subject area. A generating step for generating
Superimposing means for the image processing apparatus has found to the captured image, and superimposing step of superimposing the image of the mask by the mask image the virtual space,
An image processing method comprising: an output unit included in the image processing apparatus that outputs a captured image on which an image of the virtual space is superimposed.

Images