JP2014106647A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To specify a translucent region and information indicating transparency thereof from an image.SOLUTION: A composite image is formed by combining a known first image with an unknown second image by use of an unknown coefficient indicating transparency, in a translucent manner or a non-translucent manner for each pixel. An image processing apparatus which determines whether each of pixels in the composite image has been formed in the translucent manner or not. The image processing apparatus calculates a pixel value of an image corresponding to the second image from pixel values of the composite image in a predetermined region including one pixel and a pixel value of the first image, calculates the sum of differences of the pixel values of the image obtained by calculation between the one pixel and other pixels included in the predetermined region, specifies a coefficient to be used for obtaining the composite image from the sum of differences, and determines that the one pixel has been formed in the translucent manner when a value of the specified coefficient is larger than a predetermined value.

Description

  The present invention relates to image processing technology in mixed reality technology.

  Mixed reality, so-called MR (Mixed Reality) technology is known as a technology that seamlessly fuses the real world and the virtual world in real time. One of MR techniques uses video see-through HMD (Head Mounted Display). In the method using the video see-through HMD, a subject that substantially matches the subject observed from the pupil position of the HMD user is captured by a video camera or the like, and the HMD user superimposes computer graphics (CG) on the captured image. The displayed image can be observed.

  In the video see-through HMD, a subject is imaged by a charge coupled device such as a CCD to obtain digital image data of the subject, and an MR image (mixed reality image) is generated by superimposing a CG image on the captured image. The generated MR image is displayed to the wearer via a display device such as liquid crystal or organic EL. FIG. 14 is a diagram illustrating a state in which the HMD user is wearing the HMD 2401. Although not shown in FIG. 14, there is an external device in addition to the HMD 2401, and the HMD 2401 transmits the position and orientation information of the HMD to the external device. The external device superimposes the CG on the blue background image based on the position and orientation information received from the HMD and transmits the image to the HMD. In the HMD, a CG image received from an external device and a captured image captured by the HMD are superimposed and displayed. Thus, the HMD user can experience the MR space by using the HMD 2401.

  When superimposing a CG image and an image captured by a camera, the CG image may be semi-transparent and superimposed. For example, when there is an object such as a window in the CG image, the CG image is made semi-transparent to this object and the camera image and the chroma key are combined. By translucent and chroma key composition, the HMD user can see an image with a live-action image outside the window. In this case, apart from the CG image, a coefficient representing the transparency called α value necessary for the translucent process is required.

  FIG. 15 is a diagram showing such an example. FIG. 15A shows an image captured by the camera, and FIG. 15B shows a CG image. Reference numeral 2503 denotes a portion where the captured image is translucent and chroma key composition is performed. Reference numeral 2502 is a portion where the image captured at the time of chroma key composition is replaced with a CG image. The result of chroma key composition of (a) and (b) is (c). Since the area 2503 in (c) is translucent and chroma key composition is performed, the camera image is displayed in a transparent state.

  In order to perform semi-transparent and chroma key composition of images acquired from different devices such as an HMD and an external device, it is necessary to send information as to which is translucent. Non-Patent Document 1 describes a method of reducing α value data by embedding an α value in image data.

JP 2005-107780 A

  However, the technique disclosed in Patent Document 1 has a problem in that the α value is embedded in the image data, so that the color information of the image is less than before the α value is embedded. The present invention has been made in view of the above problems, and provides a technique for identifying a translucent area and information indicating the transparency of a CG image including a translucent area without reducing the amount of information of the image. For the purpose.

  In order to achieve the above object, an image processing apparatus according to the present invention converts a known first image and an unknown second image into a translucent or non-semiconductor for each pixel by using an unknown coefficient indicating transparency. An image processing apparatus that determines whether each of the pixels included in the combined image is a semi-transparent combined pixel with respect to a combined image combined with one of the transparent images in a predetermined region including one pixel The pixel value of the image corresponding to the second image when each of a plurality of values is used as the coefficient is calculated from the pixel value of the synthesized image and the pixel value of the first image, and Calculating means for calculating a sum of differences in pixel values of the image corresponding to the second image between one pixel and other pixels included in the predetermined region; and a plurality of values of the coefficient To the sum of the corresponding differences Therefore, the estimation means for estimating the coefficient used to obtain the composite image in the one pixel, and when the estimated value of the coefficient is larger than a predetermined value, the one pixel Determining means for determining that the pixel value of the first image and the pixel value of the second image are translucent and synthesized pixels.

  According to the present invention, for a CG image including a translucent area, the translucent area and its transparency can be specified from the image itself.

The figure which shows the structural example of an image display system. The block diagram which shows the function structural example of an image provision apparatus and HMD. The block diagram which shows the function structural example of an image process part. Schematic which shows CG image which HMD receives from an image provision apparatus. The figure which shows the concept of the semi-transparent composition of a blue back image and a CG image. The figure which shows the relationship between the process target pixel and surrounding pixel in determination of a semi-transparent area | region. The figure which shows the outline | summary of the translucent area | region determination process with respect to the translucent image of a CG image and a background image. The flowchart which shows the synthetic | combination process of a received image and a captured image. The flowchart which shows the detail of the determination process of a semi-transparent area | region. FIG. 4 is a block diagram illustrating an example of a functional configuration of an image providing apparatus, an HMD, and an image display apparatus according to a second embodiment. FIG. 4 is a block diagram illustrating an example of a functional configuration of an image processing unit according to a second embodiment. FIG. 4 is a diagram illustrating an example of a captured image, a received image, and a composite image according to the first embodiment. The figure which shows the example of the image synthesize | combined with various forms. 9 is a flowchart illustrating a synthesis process between a received image and a captured image according to the second embodiment. The figure which shows a mode that the user has mounted | worn HMD. The figure which shows the concept of a synthesis | combination with a captured image and a partially translucent CG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<< Embodiment 1 >>
(System configuration)
FIG. 1 is a diagram illustrating a configuration example of an image display system according to the present embodiment. The image display system includes an HMD 1101 that is a head-mounted image display device and also an image processing device that performs image processing, a controller 1102, and an image providing device 1104 having a display unit 1103.

  The HMD 1101 is mounted on the user's head and includes an image display unit, a communication unit that communicates with the controller 1102, a control unit that controls the communication unit, and the like. The image display unit displays an image generated by the image providing device 1104. The image display unit includes an optical system and is disposed in front of each of the left and right eyes of the user. The HMD 1101 communicates with the controller 1102. The controller 1102 configures a small-scale network such as a WLAN (Wireless Local Area Network) or a WPAN (Wireless Personal Area Network), and performs wireless communication with the HMD 1101. Note that communication between the HMD 1101 and the controller 1102 may be wired communication instead of wireless communication.

  The image providing apparatus 1104 connected to the controller 1102 by wire includes, for example, a reproduction unit that reproduces an image and a storage unit that stores an image to be reproduced. The image providing apparatus 1104 communicates with the HMD 1101 via the controller 1102. The image providing apparatus 1104 includes a keyboard and the like, and inputs data, commands, and the like. The display unit 1103 displays processing results corresponding to the input data and commands.

  In FIG. 1, the image providing apparatus 1104 and the controller 1102 are illustrated as comprising separate hardware, but the functions of the controller 1102 may be mounted and integrated in the image providing apparatus 1104. In addition, a dedicated image providing apparatus may be configured by collecting the functions of the image providing apparatus 1104 and the controller 1102.

(Configuration of each device in the system)
FIG. 2 is a schematic block diagram showing functional configurations of the image providing apparatus and the HMD in the image display system of FIG. The HMD 1101 includes, for example, an imaging unit 1201, a feature point extraction unit 1202, an image processing unit 1203, a display unit 1204, a control unit 1205, a communication unit 1206, and a captured image storage unit 1207. The image providing apparatus 1104 is mounted on an external apparatus different from the HMD 1101 such as a PC or a workstation. For example, the image providing apparatus 1104 includes a communication unit 1221, a CG drawing unit 1222, a content database (DB) 1223, and a position / orientation calculation unit 1224. Have.

  The imaging unit 1201 of the HMD 1101 captures an observation target and acquires an image. The feature point extraction unit 1202 extracts information necessary for obtaining the position and orientation of the HMD from the image captured by the imaging unit 1201. The image processing unit 1203 performs processing on an image received and acquired from the outside, and the display unit 1204 displays the image. A control unit 1205 controls each functional unit of the HMD 1101, and a communication unit 1206 transmits and receives images and control signals to and from the image providing apparatus 1104. The captured image storage unit 1207 temporarily stores the image captured by the imaging unit 1201 in a memory. The HMD 1101 may have other functional units (not shown).

  The communication unit 1221 of the image providing apparatus 1104 transmits and receives images and control signals to and from the HMD 1101. The content DB 1223 stores color graphics (CG) content, and the CG rendering unit 1222 renders a color graphics image (CG image) stored in the content DB. The position / orientation calculation unit 1224 calculates the position / orientation of the HMD 1101 from the feature point information of the captured image received from the HMD 1101.

  Hereinafter, a configuration in which feature points are transmitted from the HMD 1101 to the image providing apparatus 1104 and the position / orientation calculation unit 1224 of the image providing apparatus 1104 calculates information on the position and orientation of the HMD will be described, but the present invention is not limited thereto. For example, the position / orientation calculation unit 1224 of the image providing apparatus 1104 may be mounted on the HMD 1101 to generate position / orientation information of the HMD 1101 in the HMD 1101 and transmit the position / orientation information from the HMD 1101 to the image providing apparatus 1104. .

  In the above-described configuration, the feature point extraction unit 1202 of the HMD 1101 extracts feature points for the image captured by the imaging unit 1201. The extracted feature point information is then transmitted to the image providing apparatus 1104 via the communication unit 1206. The position / orientation calculation unit 1224 of the image providing apparatus 1104 calculates the position and orientation of the HMD 1101 from the received feature point information. The CG rendering unit 1222 renders a CG based on the position and orientation of the HMD 1101 calculated by the position / orientation calculation unit 1224. Thereafter, the image providing apparatus 1104 transmits the drawn CG image to the HMD 1101 via the communication unit 1221. The image processing unit 1203 of the HMD 1101 combines the CG image received and acquired from the image providing apparatus 1104 and the image captured by the imaging unit 1201, and the display unit 1204 displays the combined image.

  In the present embodiment, as described later, the CG image is a semi-transparent or first image (background image) and a second image (CG image) using a coefficient (α value) indicating transparency. This is a non-translucent composite image. That is, the received image has the pixel value of the background image for pixels that do not display CG, the pixel value of CG for pixels that display non-translucent CG, and the pixel values of the background image and CG for translucent pixels. The images each have a pixel value obtained from the pixel and the α value for the pixel. Then, the HMD 1101 specifies the pixel value of the CG image and the coefficient indicating transparency from the received image, and generates a composite image using the captured image and the specified pixel value of the CG image. As described above, the wearer of the HMD 1101 can see an image obtained by combining the CG drawn by the image providing apparatus 1104 and the captured image.

  In the present embodiment, the CG rendering unit 1222 of the image providing apparatus 1104 renders CG on, for example, a blue background image, but the background image used for CG rendering is not limited thereto. For example, an image having a predetermined pattern may be used as the background. That is, the background image may be any image as long as the HMD 1101 and the image providing apparatus 1104 recognize the same background image.

(Configuration of image processing unit)
FIG. 3 is a block diagram illustrating the image processing unit 1203 in detail. As described above, the image processing unit 1203 is a functional unit that synthesizes the CG image received from the image providing apparatus 1104 and the image captured by the imaging unit 1201. The image processing unit 1203 includes, for example, a determination unit 1301, a translucent area specifying unit 1302, a separation unit 1303, and a synthesis unit 1304.

  The determination unit 1301 determines whether there is a pixel of a predetermined background image, for example, a blue back pixel, in the image received from the image providing apparatus 1104. FIG. 4 shows an example of an image received from the image providing device 1104. In FIG. 4, reference numeral 1401 denotes a background image pixel, 1402 denotes a CG pixel, and 1403 denotes a pixel that synthesizes the CG and the background image in a translucent manner. The determination unit 1301 outputs a region determined not to be a pixel of the background image to the translucent region specifying unit 1302, and outputs a region determined to be a pixel of the background image to the synthesis unit 1304.

  The translucent area specifying unit 1302 determines whether each area (pixel) of the image received from the determination unit 1301 is a CG image or a translucent area. The determination method will be described later. The translucent area specifying unit 1302 outputs the area (pixel) determined to be a CG image to the combining unit 1304 and the area (pixel) determined to be translucent to the separating unit 1303. The separation unit 1303 separates the translucent area into a background image and CG image before composition and an α value, that is, a coefficient indicating transparency, and outputs the CG image and the α value to the composition unit 1304.

  If the input region is a background region, the composition unit 1304 replaces the region with an image extracted from the captured image storage unit 1207, and if the input region is a CG image, the CG image remains. When the input area is a translucent area, the synthesizing unit 1304 synthesizes the image extracted from the captured image storage unit 1207 with the CG image and the α value obtained by the separation unit 1303 in a translucent manner. The synthesized image is sent to the display unit 1204.

  Before describing the processing in the translucent area specifying unit 1302, a translucent image synthesis method will be briefly described. The synthesis of the translucent image is generally performed according to the following equation.

  Z is the pixel value of the image after translucent synthesis, X is the pixel value of the first image (background image, for example, captured image, blue back image, specific pattern image), and Y is the second image ( For example, the pixel value of a CG image). Note that the first image is output when α = 1, and the second image is output when α = 0. That is, the smaller the α value, the stronger the pixel value of the second image in the composite image, and the greater the α value, the stronger the pixel value of the first image in the composite image. By changing the value of α between 0 and 1 in the image, the first image and the second image can be synthesized translucent or non-translucent. In addition, subscripts “red”, “green”, and “blue” represent red, green, and blue color information for each pixel, respectively. FIG. 5 is a diagram showing a semi-transparent composition of a blue back image and a CG image with α = 0.5 for one pixel. The pixel value of the blue-back image is (R, G, B) = (0, 0, 255), and the pixel value of the CG image is, for example, (R, G, B) = (64, 192, 128) , Α = 0.5, the pixel value synthesized in a translucent manner is (R, G, B) = (32, 96, 192). The pixel value of the image received by the HMD 1101 from the image providing apparatus 1104 is the pixel value of the semi-transparent composite image of the blue back image and the CG image. In the example of FIG. 5, the pixel value is (R, G, B) = The image is (32, 96, 192).

(Semi-transparent area judgment method)
Next, a method for determining a translucent area or a CG image area in the translucent area specifying unit 1302 will be described. In the example of FIG. 5, the pixel values of the image received from the image providing apparatus 1104 are (R, G, B) = (32, 96, 192) as described above. Here, when α = 0.5 and the pixel value (R, G, B) = (64, 192, 128) of the CG image are obtained, it can be determined that this pixel is a translucent pixel. If the α value becomes 0, according to the above formula, the blue-back image is not reflected in the translucent image, so that this pixel can be determined to be a CG pixel.

According to the above formula, for example, there are two indeterminate values (α value and Y _red ) in the formula for obtaining red (RED), so that the pixel value and α value of the CG image are obtained from the information of one pixel. Can not. Therefore, the calculation of the pixel value of the CG image and the specification of the α value are performed also using information on surrounding pixels. The CG pixel value and α value rarely change significantly between one pixel and surrounding pixels of this pixel. Therefore, in the present embodiment, the translucent area is separated using pixel information of a predetermined area including the processing target pixel. In the present embodiment, the case where eight pixels around the processing target pixel 1602 are used as the predetermined region as shown in FIG. 6A will be described. However, the range using the surrounding pixels may be further increased or decreased. . Further, for example, as shown in FIG. 6B, among the predetermined number of surrounding pixels (for example, 24 pixels) instead of the surrounding 8 pixels, the difference from the pixel value from the processing target pixel is small (for example, 8 pixels). May be used as the predetermined area.

  FIG. 7 shows an example of determination of a semi-transparent area in the case where a CG image and a background image are semi-transparent combined with α = 0.5. FIG. 7A shows the pixel value of the CG image before performing the translucent process in the predetermined area including the processing target pixel, and FIG. 7B shows the pixel value of the background image in the predetermined area. Show. Here, a predetermined pattern image is used as a background image instead of a blue back image. FIG. 7C shows pixel values of a translucent area obtained by translucently combining a CG image and a background image with α = 0.5, and the pixel value of an image received from the image providing apparatus 1104. .

  Here, when separating the known first image (background image) and the unknown second image (CG image) from the translucent image, the indefinite value is an α value that is a coefficient indicating transparency and the second image. This is a pixel value of (CG image). However, if the unknown α value is assumed to be one value, the pixel value of the image corresponding to the second image corresponding to the α value can be inversely calculated from the above equation. That is, assuming a certain α value (0 ≦ α <1), the pixel value of the CG image is inversely calculated by the following equation.

ただし、ＹはＣＧ画素値、Ｚは合成画像の画素値、Ｘは撮像画像の画素値である。なお、α＝１の場合はＹは不定となるため、この式を用いたＣＧ画素値の算出は行わない。

However, Y is a CG pixel value, Z is a pixel value of a synthesized image, and X is a pixel value of a captured image. Note that, when α = 1, Y is indefinite, and the CG pixel value is not calculated using this equation.

  FIG. 7D shows a CG pixel value in a predetermined area calculated from the received image when the α value is assumed to be 0.7, a processing target pixel calculated based on the CG pixel value, and the surrounding area. The sum of differences from pixels is shown. That is, in (d) of FIG. 7, with respect to all the pixels in the predetermined region, α = 0.7, and the calculation result of the CG pixel value Y for each of R, G, and B by the above formula, and the calculated CG pixel value The result of accumulating the difference between the surrounding pixels and the processing target pixel is shown. Similarly, FIG. 7E shows an example when α = 0.5, and FIG. 7F shows an example when α = 0.3. Although not shown in FIG. 7, the CG pixel value and its CG pixel value are also obtained when the α value is 0.1 to 0.2, 0.4, 0.6, and 0.8 to 0.9. And the sum of the difference values based on. That is, with the α value as a parameter, the CG pixel value of a predetermined area is inversely calculated for each of the plurality of parameters, and the difference between the CG pixel value of the processing target pixel and the CG pixel value of other pixels included in the predetermined area Find the cumulative value of. Further, the change of the α value may not be in increments of 0.1, may be finer steps such as 0.01 increments, or may be wider steps such as 0.125 increments. Further, the difference may be calculated even when the α value is 0. The total difference value is calculated based on the following equation.

  Since the CG pixel value hardly changes greatly between adjacent pixels, if the sum of the differences between the CG pixel values of the processing target pixel and the surrounding pixels calculated for a certain α value is small, the target pixel is It can be determined that the translucent composition is made with the α value. That is, when the CG pixel value calculated backward using a certain α value substantially matches between the processing target pixel and the surrounding pixels, it is determined that the target pixel is a pixel that is translucently synthesized using the α value. be able to. Therefore, the separation unit 1303 obtains an α value that minimizes the sum of differences, and temporarily stores the α value. In the example of FIG. 7, when (e), that is, when the α value is 0.5, the difference between the target pixel and the surrounding pixels is the smallest. For this reason, the separation unit 1303 estimates that the background image and the CG image are combined using α = 0.5, and temporarily stores the α value as 0.5. In the present embodiment, the α value that minimizes the sum of differences is assumed to be the α value used for the translucent composition, but the present invention is not limited to this. For example, an α value having a sum of differences smaller than a predetermined value may be obtained, and the minimum or maximum α value among them may be assumed as the α value used for the translucent composition. Further, when the sum of differences is smaller than a predetermined value in a certain α value range, the α value of the median value in the range may be assumed as the α value used in the translucent composition. Alternatively, the α value used for the translucent composition may be assumed from the α value and the sum of the differences by a weighted average.

  Here, when α is sufficiently small, the pixel value of the translucent image is almost composed of the CG pixel value before the semitransparent composition. Therefore, in this embodiment, the separation unit 1303 determines that the CG pixel is not a semi-transparent region when the α value is equal to or smaller than a predetermined value. For example, when the α value is 0.1 or less, it is determined that the region is not a translucent region. Note that the predetermined value for determining whether or not the region is a translucent region may not be 0.1. For example, when the difference value is calculated with respect to the α value in increments of 0.01, the CG image may be determined when the α value is 0.09 or less. Thus, the predetermined value may be determined according to the step of the α value for calculating the difference value.

  Further, the predetermined value may be variable depending on the situation. For example, if the average of the α values temporarily stored by the separation unit 1303 is low for a certain area of the received image, it is synthesized with a low α value by making the semi-transparent area easier to detect by lowering the predetermined value. The translucent area can be detected without omission. Similarly, a semi-transparent area may be determined again by lowering a predetermined value for a non-translucent area that exists in a spot in an area determined to be translucent. As a result, it is possible to increase the probability that a pixel that is erroneously determined as a CG region in a large translucent region can be included in the translucent region. Also, if there are many areas that are determined to be semi-transparent in a certain area of the received image, an area that is not originally semi-transparent is erroneously determined as a semi-transparent area by increasing the predetermined value. It becomes possible to prevent this.

  In the above description, the processing corresponding to one pixel has been described with reference to FIG. 7, but this processing is executed for all regions that are not background regions.

(Image composition processing)
FIG. 8 is a flowchart showing the overall processing for combining the received image (CG image or translucent image) received from the image providing apparatus 1104 and the captured image in this embodiment. When the process is started, the determination unit 1301 first determines whether or not the pixel of the image received from the image providing apparatus 1104 is a known background image (for example, a blue background image) (S1801). Specifically, for example, for a pixel to be processed, it is determined whether the pixel value of the known background image and the pixel value of the received image match, and if they match, the pixel is determined to be a pixel of the background image. To do. When it is determined that the pixel to be processed is a background image (Yes in S1801), the synthesis unit 1304 acquires a captured image from the captured image storage unit 1207 (S1802), and replaces the background image with the captured image for that pixel. In step S1803, the process ends. When the determination unit 1301 determines that the processing target pixel is not a background image (No in S1801), the translucent area specifying unit 1302 determines whether the processing target pixel is a CG pixel or a semitransparent pixel (S1804, S1805). . That is, the semi-transparent area specifying process is not executed when the processing target pixel matches the pixel of the background image, and the semi-transparent area specifying process is executed only when it does not match the pixel of the background image. The flow of this determination process will be described later.

  If it is determined that the pixel to be processed is a CG pixel (Yes in S1805), the CG pixel is displayed as it is, and thus the process ends without performing a process such as replacement or composition. If it is determined that the processing target pixel is a translucent pixel (No in S1805), the translucent area specifying unit 1302 calculates the pixel value (CG pixel value before translucent synthesis) of the synthesis target image and the α value. The determination is executed (S1806). The synthesizing unit 1304 synthesizes the CG image and the captured image using the pixel value and the α value of the synthesis target image obtained in S1806 and the captured image acquired from the captured image storage unit 1207 (S1807).

(Semi-transparent area judgment process)
FIG. 9 is a flowchart showing details of the translucent area determination processing in S1804 of FIG. This process is executed by the translucent area specifying unit 1302. When the processing is started, the translucent area specifying unit 1302 determines whether calculation of CG pixel values and calculation of differences as shown in FIG. 7 have been completed for all α values to be calculated ( S1901). If there is an α value for which calculation has not been completed (No in S1901), as shown in FIG. 7, calculation of CG pixel values for the processing target pixel and surrounding pixels with respect to the α value to be calculated (S1902). ) And calculation of a cumulative value of differences of CG pixel values (S1903).

  When the calculation of the CG pixel value and the difference is completed for all the α values to be calculated (Yes in S1901), the translucent area specifying unit 1302 determines the α value and the CG when the difference value obtained in S1903 is the smallest. The pixel value is specified and temporarily stored (S1904). Then, the translucent area specifying unit 1302 determines whether the temporarily stored α value is equal to or less than a threshold value (S1905). Then, when the α value is equal to or smaller than the threshold value (Yes in S1905), the translucent area specifying unit 1302 determines the CG pixel value (S1906), and when the α value is larger than the threshold value (No in S1905), determines the translucent pixel value. (S1907).

  In the above description, it has been described that the translucent area specifying unit 1302 determines the pixel value and α value of the synthesis target image in S1806 of FIG. 8, but in S1806, the α temporarily stored in S1904 is stored. You may perform the process which only reads a value and a CG pixel value. However, the α value and the CG pixel value temporarily stored in S1904 are not necessarily used. For example, in an area determined to be translucent, a value obtained by averaging α values in surrounding semi-transparent pixels including the processing target pixel may be used as the α value for the processing target pixel. Further, a CG pixel value that can be calculated using the α value may be used as the CG pixel value of the processing target pixel. As described above, for image synthesis, the α value and the CG pixel value temporarily stored in S1904 may be used as they are, or the separately specified α value and CG pixel value may be used.

  As described above, according to the present embodiment, the semi-transparent region and the CG of the image synthesized with the background image without being received the alpha value or special data and without embedding information in the image. It is possible to discriminate the region and separate the pre-combination image. Therefore, an image including a translucent area can be transmitted to the HMD without degrading the quality of the CG image, and a high-definition CG image and a captured image can be combined translucently.

<< Embodiment 2 >>
In the first embodiment, a method has been described in which the image providing apparatus 1104 transmits an image in which CG is drawn on a background image to the HMD 1101, and the HMD 1101 generates a display image by combining the captured image and the received image. In the present embodiment, a method for changing the superimposing process according to the output destination when the HMD 1101 performs synthesis will be described.

(System configuration)
FIG. 10 is a block diagram illustrating functional configurations of the image providing apparatus 1104, the HMD 1101, and the image display apparatus 2010 included in the image processing system according to this embodiment. In the image processing system of the present embodiment, for example, an image transmission unit 2001 is added to the HMD 1101 and an image display device 2010 connected to the image transmission unit 2001 is added to the configuration of the first embodiment. In FIG. 10, one image display device 2010 is shown as an external display device. However, at least one external display device may be provided, and a plurality of display devices may be provided.

  The image display device 2010 is a display device for a person who is not wearing the HMD to see an image displayed on the HMD. Thereby, even a person who is not wearing the HMD can see the same image as the HMD experienced person. The image transmission unit 2001 of the HMD 1101 acquires an image from the image processing unit 1203 and transmits this image to the image display device 2010. The image receiving unit 2011 of the image display apparatus 2010 receives an image from the HMD 1101 and sends this image to the image presenting unit 2012. The image presentation unit 2012 displays the received image. This makes it possible to display an image even on an external display device.

(Configuration of image processing unit)
FIG. 11 is a block diagram illustrating a detailed configuration example of the image processing unit 1203 in the present embodiment. The image processing unit 1203 is configured by adding, for example, an external display combining unit 2101 as compared to the configuration of the first embodiment. The external display combining unit 2101 is a functional unit that creates an image to be displayed on the image display apparatus 2010.

  An example of an image that the HMD 1101 receives from the image providing apparatus 1104, an image that the HMD 1101 displays on the display unit 1204, and an image that the HMD 1101 transmits to the image display apparatus 2010 are illustrated in FIGS. 12A and 12B. In FIG. 12A, (1) is an image captured by the imaging unit 1201, and (2) is a CG image including a translucent area received from the image providing apparatus 1104. Character information 2221 and area 2223 in (2) of FIG. 12A indicate a semi-transparent area, and area 2222 indicates a non-translucent CG area. In (2) of FIG. 12A, the character information is made translucent, but the object to be made translucent may not be the character information, and may be a symbol or an object. The HMD 1101 according to the first embodiment synthesizes the captured image and the received image and displays them on the display unit 1204 as shown in (3) of FIG. 12A.

  Here, (4) in FIG. 12B is a diagram in which semi-transparent characters are not synthesized, (5) is a diagram in which semi-transparent characters are synthesized as non-translucent characters, and (6) is a diagram in which semi-transparent characters are not synthesized. In addition, a diagram in which a translucent CG is synthesized as a non-translucent CG is shown. Note that (5) in FIG. 12B shows an example of a diagram in which semi-transparent characters are synthesized as non-translucent characters. In the process of making non-translucent characters, the α value that is a coefficient indicating transparency is changed. May be. By changing the α value, a CG image with higher transparency can be combined with the captured image, and a non-translucent or non-translucent semi-transparent CG image can be combined with the captured image. . For example, if the HMD user wants to clearly present the character information, an image as shown in (5) is displayed on the display unit 1204. If the image display device 2010 does not want to present the character information, an image transmission is performed. Images such as (4) may be output from the parts 2001. In the CG image, when there is no non-translucent pixel around the translucent pixel, display control of a part of information can be performed by such usage. Further, for the non-background area including the translucent area, the entire area may not be displayed on the image display device 2010, for example. If it is not desired to display the translucent area on the external display device, it can be displayed as in (6). In this manner, the images as shown in (3) to (6) are switched and displayed on the display unit 1204 and the image display device 2010 according to the usage and environment.

  Hereinafter, the image displayed on the display unit 1204 of the HMD 1101 will be described as (3), and the image displayed on the image presentation unit 2012 of the image display device 2010 will be described as (6). However, the present invention is not limited to this, and FIG. 12A and FIG. Any combination of (3) to (6) may be displayed. In addition, the same image may be displayed on the display unit 1204 and the image presentation unit 2012. Which display method is used to display on which display unit may be fixedly set in the HMD 1101 or may be set by an instruction from the image providing apparatus 1104. Further, for example, in response to an instruction from the image providing apparatus 1104 or an instruction from a user interface connected to the HMD 1101 (not shown), what composite image is displayed on which display unit is dynamically displayed during the image display operation. You may switch to.

  FIG. 13 is a flowchart showing a composition process of images displayed on each display unit in the present embodiment. The flowchart of FIG. 13 is obtained by adding the process of S2301 to the flowchart of FIG. 8, and thus description of processes other than S2301 is omitted. In step S <b> 2301, the external display composition unit 2101 receives images from the separation unit 1303, the determination unit 1301, and the captured image storage unit 1207, and performs composition of images to be passed to the image display device 2010. The external display composition unit 2101 receives the CG pixel value and the α value from the determination unit 1301. For this reason, the external display combining unit 2101 can combine images in a manner different from that of the display unit 1204 for the image display apparatus.

  According to this embodiment, an image composition method can be changed according to the display part to display, and the image suitable for each display part can be displayed.

  In the above description, the HMD 1101 is used. However, the present invention is not limited to this, and any image display apparatus that combines and displays the image received from the image providing apparatus 1104 and the captured image may be used.

<< Other Embodiments >>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (14)

For a composite image obtained by combining a known first image and an unknown second image using an unknown coefficient indicating transparency, either semi-transparent or non-translucent for each pixel, An image processing apparatus that determines whether each of the included pixels is a translucent and synthesized pixel,
The second image when each of a plurality of values is used as the coefficient from the pixel value of the composite image and the pixel value of the first image of each pixel in a predetermined region including one pixel And calculating the sum of pixel value differences between the one pixel and other pixels included in the predetermined area in the image corresponding to the second image. A calculation means;
Estimating means for estimating the coefficient used to obtain the composite image in the one pixel based on the sum of the differences corresponding to each of the plurality of values of the coefficient;
When the estimated value of the coefficient is larger than a predetermined value, the one pixel is a pixel in which the pixel value of the first image and the pixel value of the second image are combined in a translucent manner. Determination means for determining;
An image processing apparatus comprising: For the one pixel, the image processing apparatus further includes a determination unit that determines whether a pixel value of the composite image matches a pixel value of the first image,
When the pixel value of the composite image does not match the pixel value of the first image in the one pixel, the calculation unit, the estimation unit, and the determination unit for the one pixel Perform processing,
The image processing apparatus according to claim 1. An imaging means for capturing an image;
For the one pixel that is determined to be a semi-transparent pixel, it is used to synthesize the captured image and a synthesis target image to be synthesized based on the estimated coefficient. Determining a value of a coefficient indicating transparency, and determining means for determining the synthesis target image based on the determined coefficient value, the first image, and the synthesized image;
The pixel having the pixel value of the composite image and the pixel value of the first image having the pixel value of the captured image has the pixel value of the captured image. A pixel value obtained by semi-transparently combining the image to be combined with the captured image using the value of the pixel, the pixel value of the composite image and the pixel value of the first image do not match, and Generating means for generating an image having a pixel value of the synthesis target image for pixels that are not determined to be semi-transparent and synthesized;
Display means for displaying the generated image;
The image processing apparatus according to claim 2, further comprising: A transmission unit configured to transmit the generated image to at least one external display device;
The image processing apparatus according to claim 3. Further comprising: transmission means for transmitting the image generated by the generating means for the external display device to at least one external display device;
The generation unit generates an image in which pixel values of the pixels determined to be semi-transparent and synthesized are different between an image transmitted to the external display device and an image displayed on the display unit.
The image processing apparatus according to claim 3. The generating means sets the pixel value of the pixel determined to be combined with the semi-transparent image to be transmitted to the external display device as the pixel value of the captured image.
The image processing apparatus according to claim 5. The generation means uses a pixel value of a pixel determined to be semi-transparent as an image to be transmitted to the external display device as a pixel value of the composition target image.
The image processing apparatus according to claim 5. The generation unit is configured to set a pixel value of a pixel determined to be semitransparent for an image to be transmitted to the external display device to a coefficient different from the coefficient used in generating an image to be displayed on the display unit. A pixel value of an image obtained by translucently synthesizing the captured image and the synthesis target image using a value;
The image processing apparatus according to claim 5. The estimating means estimates a value that minimizes the sum of the differences among a plurality of values of the coefficient as the coefficient used to obtain the composite image;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The predetermined area is an area composed of the one pixel and eight pixels surrounding the one pixel.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The predetermined area is an area composed of the one pixel and eight pixels having a pixel value with a small difference from the pixel value of the one pixel out of a predetermined number of pixels surrounding the one pixel. ,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The second image is a color graphics image, and the first image is a predetermined background image;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. For a composite image obtained by combining a known first image and an unknown second image using an unknown coefficient indicating transparency, either semi-transparent or non-translucent for each pixel, An image processing method for determining whether or not each of the included pixels is a translucent and synthesized pixel,
When the calculation unit uses each of a plurality of values as the coefficient from the pixel value of the composite image and the pixel value of the first image of each pixel in a predetermined region including one pixel, A pixel value of an image corresponding to the second image is calculated, and in the image corresponding to the second image, a difference in pixel value between the one pixel and another pixel included in the predetermined region is calculated. A calculation step for calculating the total;
An estimating unit that estimates the coefficient used to obtain the composite image in the one pixel based on a sum of the differences corresponding to each of the plurality of values of the coefficient;
When the determination means has a value of the estimated coefficient larger than a predetermined value, the pixel value of the first image and the pixel value of the second image are combined in a translucent manner for the one pixel. A determination step of determining that it is a pixel;
An image processing method comprising:   The program for functioning a computer as each means with which the image processing apparatus of any one of Claim 1 to 12 is provided.

Images