JP2020167660A - Image data transmission method, content processing device, head-mounted display, relay device, and content processing system - Google Patents

Abstract

To preferably display a composite image including a photographic image as a moving image on a head-mounted display.SOLUTION: A communication unit 258 of a content processing device 200 acquires an analysis result of a photographic image from a head-mounted display 100. A position and posture prediction unit 252 of a composite data generation unit 250 predicts the position and posture of a subject based on the analysis result. An image drawing unit 254 draws an object that should be synthesized based on the prediction. A composite information integration unit 256 integrates an image and an α value of the object on the same image plane. The composite data is transmitted to the head-mounted display 100 and is synthesized with the photographic image.SELECTED DRAWING: Figure 10

Description

The present invention relates to an image data transmission method, a content processing device, a head-mounted display, a relay device, and a content processing system used for image display.

Technologies for shooting moving images and processing them in real time to obtain some information or use them for display are used in various fields. For example, if a camera that captures a real space is provided in front of a shielded head-mounted display and the captured image is displayed as it is, the user can operate while checking the surrounding conditions. Augmented reality and mixed reality can be realized by synthesizing virtual objects with captured images and displaying them.

With technology that synthesizes separately generated images such as virtual objects with captured images and displays them in real time, the more high-quality image expression is pursued, the more data that must be transmitted from shooting to display, image analysis, etc. The processing load increases. As a result, the consumption of resources such as power consumption, required memory capacity, and CPU time increases, and the movement of the user and the movement on the display are time-shifted, which makes the user feel uncomfortable and in some cases. It can also cause poor physical condition such as image sickness.

Further, according to the embodiment in which an image is generated by an external device and transmitted to the head-mounted display, a high-quality image can be displayed without increasing the load on the head-mounted display itself, but for transmission of large-sized data. Requires wired communication, which limits the user's range of motion.

The present invention has been made in view of these problems, and an object of the present invention is to produce a high-quality image while suppressing a delay time from shooting to display and resource consumption in a technique for displaying a composite image including a shot image as a moving image. The purpose is to provide technology that can be displayed. Another object of the present invention is to apply an image display technique capable of supporting various communication methods between a head-mounted display and an external device.

In order to solve the above problems, an aspect of the present invention relates to an image data transmission method. In this image data transmission method, the image generator generates an image to be combined with the display image, an α value indicating the transparency of the pixels of the image to be combined, and data of the image to be combined and the α value. It is characterized by including a step of generating composite data represented on one image plane and a step of transmitting the composite data to an apparatus for generating a display image.

Another aspect of the present invention relates to a content processing device. This content processing device represents an image drawing unit that generates an image to be combined with a display image, an image to be combined, and α value data representing the transparency of the pixels of the image to be combined in one image plane. It is characterized by including a synthetic information integration unit that generates synthetic data and a communication unit that outputs synthetic data.

Yet another aspect of the present invention relates to a head-mounted display. This head mount display is a composite data in which a camera that captures a real space, an image to be combined with a display image, and α value data representing the transparency of the pixels of the image to be combined are represented on one image plane. Is received from an external device, and based on the α value, an integrated circuit for image processing that synthesizes the image to be combined with the image taken by the camera to generate a display image, and a display panel that outputs the display image. Characterized by having

Yet another aspect of the present invention relates to a relay device. This relay device combines the image to be combined with the image to be combined and the data for composition in which the α value data representing the transparency of the pixels of the image to be combined is represented on one image plane. The data separation unit that separates the value data, the compression coding unit that compresses and encodes the image to be combined and the α value data by different methods, and the composition data are acquired from the device that generates the image and the compression code is obtained. It is characterized by including a communication unit that transmits the converted data to a device that generates a display image.

Yet another aspect of the present invention relates to a content processing system. This content processing system includes a display device and a content processing device that generates an image to be displayed on the display device, and the content processing device includes an image to be combined with a display image and pixels of the image to be combined. The display device includes a compositing data generation unit that generates compositing data that represents α value data representing transparency on one image plane, and a communication unit that outputs compositing data, and the display device captures a real space. It is equipped with an integrated circuit for image processing that synthesizes an image to be combined with an image taken by the camera based on the α value of the data for composition to generate a display image, and a display panel that outputs the display image. It is characterized by that.

It should be noted that any combination of the above components and the conversion of the expression of the present invention between methods, devices, systems, computer programs, data structures, recording media and the like are also effective as aspects of the present invention.

According to the present invention, in a technique for displaying a composite image including a captured image as a moving image, a high-quality image can be displayed while suppressing a delay time from shooting to display and resource consumption. In addition, it can support various communication methods between the head-mounted display and an external device.

It is a figure which shows the appearance example of the head-mounted display of this embodiment. It is a figure which shows the configuration example of the content processing system of this embodiment. It is a figure which shows typically the path of the data in the content processing system of this embodiment. It is a figure for demonstrating the process of generating the display image from the photographed image in the integrated circuit for image processing of this embodiment. It is a figure for demonstrating the process of generating the display image by synthesizing the virtual object transmitted from the content processing apparatus with the photographed image in the integrated circuit for image processing of this embodiment. It is a figure for demonstrating the content of the data transmitted by the content processing apparatus in order for the integrated circuit for image processing to synthesize an image in this embodiment. It is a figure which shows the variation of the system configuration for transmitting the data for synthesis from the content processing apparatus to the head-mounted display in this embodiment. It is a figure which shows the circuit structure of the integrated circuit for image processing of this embodiment. It is a figure which shows the internal circuit structure of the content processing apparatus of this embodiment. It is a figure which shows the structure of the functional block of the content processing apparatus in this embodiment. It is a figure which shows the structure of the functional block of the relay device in this embodiment. It is a figure which shows the structure of the functional block of the image processing apparatus built in the head-mounted display in this embodiment. It is a figure which shows an example of the structure of the image which the content processing apparatus of this embodiment integrates a graphic image and an α image. It is a figure which illustrates the data structure of the pixel value of an α image which the content processing apparatus of this embodiment integrates into a graphics image. In this embodiment, it is a figure which shows the process procedure at the time of embedding α image data in the area where a graphics image is not represented, and transmitting. In this embodiment, it is a figure which shows the process procedure at the time of reducing the α image and the graphics image in the vertical direction, connecting them up and down, and transmitting them. In this embodiment, it is a figure which shows the process procedure at the time of reducing a graphics image and an α image in the lateral direction, connecting them to the left and right, and transmitting them.

FIG. 1 shows an example of the appearance of the head-mounted display 100. In this example, the head-mounted display 100 is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that goes around the head and realizes fixing of the device when the user wears it. The output mechanism 102 includes a housing 108 having a shape that covers the left and right eyes when the head-mounted display 100 is worn by the user, and includes a display panel inside so as to face the eyes when worn.

Further, the inside of the housing 108 is provided with an eyepiece located between the display panel and the user's eyes when the head-mounted display 100 is attached to magnify the image. The head-mounted display 100 may further include a speaker or earphone at a position corresponding to the user's ear when worn. Further, the head-mounted display 100 has a built-in motion sensor, and may detect the translational motion and the rotational motion of the head of the user wearing the head-mounted display 100, and eventually the position and posture at each time.

The head-mounted display 100 further includes a stereo camera 110 on the front surface of the housing 108, a monocular camera 111 with a wide viewing angle in the center, and four cameras 112 with a wide viewing angle at the four corners of the upper left, upper right, lower left, and lower right. Take a video of the real space in the direction corresponding to the direction of the face. The present embodiment provides a mode in which an image taken by the stereo camera 110 is immediately displayed to show the state of the real space in the direction in which the user is facing. Hereinafter, such a mode is referred to as a "see-through mode". The head-mounted display 100 basically shifts to the see-through mode during the period when the image of the content is not displayed.

By automatically shifting to the see-through mode of the head-mounted display 100, the user can check the surrounding situation without removing the head-mounted display 100 before the content starts, after the content ends, or when the content is interrupted. In addition to this, the transition timing to the see-through mode may be when the user explicitly performs the transition operation. As a result, even while viewing the content, the display can be temporarily switched to the image in the real space at any timing, and necessary work such as finding the controller and picking it up can be performed.

At least one of the images captured by the stereo camera 110, the monocular camera 111, and the four cameras 112 can also be used as an image of the content. For example, augmented reality (AR: Augmented Reality) and mixed reality (MR) can be created by synthesizing virtual objects with captured images and displaying them in positions, postures, and movements that correspond to the actual space in the image. realizable. In this way, regardless of whether or not the captured image is included in the display, the position, posture, and movement of the object to be drawn can be determined by using the analysis result of the captured image.

For example, the corresponding points may be extracted by performing stereo matching on the captured image, and the distance of the subject may be acquired by the principle of triangulation. Alternatively, the head-mounted display 100 with respect to the surrounding space, and thus the position and posture of the user's head may be acquired by SLAM (Simultaneous Localization and Mapping). It can also perform object recognition and object depth measurement. By these processes, the virtual world can be drawn and displayed in the field of view corresponding to the position of the user's viewpoint and the direction of the line of sight.

The head-mounted display 100 of the present embodiment is not limited to the one shown in the figure as long as it is provided with a camera that captures a real space with a field of view corresponding to the position and orientation of the user's face. Further, if the images of the field of view of the left eye and the field of view of the right eye are simulated in the see-through mode, a monocular camera can be used instead of the stereo camera 110.

FIG. 2 shows a configuration example of the content processing system according to the present embodiment. The head-mounted display 100 is connected to the content processing device 200 by an interface 300 that connects peripheral devices such as wireless communication or USB Type-C. A flat plate display 302 is connected to the content processing device 200. The content processing device 200 may be further connected to the server via a network. In that case, the server may provide the content processing device 200 with an online application such as a game in which a plurality of users can participate via a network.

The content processing device 200 basically processes a content program, generates a display image, and transmits the display image to the head-mounted display 100 or the flat-plate display 302. In a certain aspect, the content processing device 200 specifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head-mounted display 100, and generates a display image of the corresponding visual field at a predetermined rate. To do.

The head-mounted display 100 receives the data of the display image and displays it as an image of the content. To this extent, the purpose of displaying the image is not particularly limited. For example, the content processing device 200 may generate a virtual world, which is the stage of the game, as a display image while advancing the electronic game, or may use a still image or a still image for viewing or providing information regardless of whether the virtual world or the real world is used. A moving image may be displayed.

The distance between the content processing device 200 and the head-mounted display 100 and the communication method of the interface 300 are not limited. For example, the content processing device 200 may be a game device owned by an individual, a server of a company or the like that provides various distribution services such as a cloud game, or a home server that transmits data to an arbitrary terminal. Therefore, communication between the content processing device 200 and the head mount display 100 is in addition to the above-mentioned examples, in public networks such as the Internet, LAN (Local Area Network), mobile phone carrier networks, Wi-Fi spots in the city, and homes. It may be realized via an arbitrary network or access point such as a Wi-Fi access point.

FIG. 3 schematically shows a data path in the content processing system of the present embodiment. The head-mounted display 100 includes a stereo camera 110 and a display panel 122 as described above. However, as described above, the camera is not limited to the stereo camera 110, and may be any or a combination of the monocular camera 111 and the four cameras 112. The same applies to the following description. The display panel 122 is a panel having a general display mechanism such as a liquid crystal display or an organic EL display, and displays an image in front of a user wearing the head-mounted display 100. Further, the head-mounted display 100 includes an integrated circuit 120 for image processing inside.

The image processing integrated circuit 120 is, for example, a system-on-chip equipped with various functional modules including a CPU. In addition to this, the head mount display 100 includes motion sensors such as a gyro sensor, an acceleration sensor, and an angular acceleration sensor, a main memory such as a DRAM (Dynamic Random Access Memory), an audio circuit that allows the user to hear voice, and peripheral devices. Peripheral device interface circuits and the like for connecting the devices may be provided, but the illustrations are omitted here.

When augmented reality or mixed reality is realized by a shield-type head-mounted display, generally, an image taken by a stereo camera 110 or the like is taken into a subject that processes the content, and then combined with a virtual object to generate a display image. Since the main body that processes the content in the illustrated system is the content processing device 200, as shown by the arrow B, the image taken by the stereo camera 110 is once transmitted to the content processing device 200 via the image processing integrated circuit 120. Will be done.

Then, the virtual object is synthesized and returned to the head-mounted display 100, and is displayed on the display panel 122. On the other hand, in the present embodiment, as shown by arrow A, a data path for the captured image is provided. For example, in the see-through mode, the image taken by the stereo camera 110 is appropriately processed by the image processing integrated circuit 120 and displayed as it is on the display panel 122. At this time, the image processing integrated circuit 120 only performs the process of correcting the captured image into a format suitable for display.

Alternatively, in the image processing integrated circuit 120, the image generated by the content processing device 200 and the captured image are combined and displayed on the display panel 122. In this way, it is sufficient to transmit only the information related to the real space acquired from the captured image to the content processing device 200 from the head-mounted display 100 instead of the data of the captured image. Further, only the image data to be combined needs to be transmitted from the content processing device 200 to the head-mounted display 100.

According to the path of arrow A, the data transmission path is significantly shortened as compared with arrow B. Further, as described above, the size of data to be transmitted between the head-mounted display 100 and the content processing device 200 can be reduced. As a result, the time from image capture to display can be shortened, and the power consumption required for transmission can be reduced.

FIG. 4 is a diagram for explaining a process of generating a display image from a captured image in the image processing integrated circuit 120. In real space, suppose a table on which objects are placed is in front of the user. By photographing the stereo camera 110, the photographed image 16a of the left viewpoint and the photographed image 16b of the right viewpoint are acquired. Due to the distance between both viewpoints of the stereo camera 110, parallax occurs in the images of the same subject in the captured images 16a and 16b.

In addition, the lens of the camera causes distortion in the image of the subject. Generally, such lens distortion is corrected to generate a distortion-free left-viewpoint image 18a and a right-viewpoint image 18b (S10). Here, assuming that the pixels of the position coordinates (x, y) in the original images 16a and 16b are corrected to the position coordinates (x + Δx, y + Δy) in the corrected images 18a and 18b, the displacement vector (Δx, Δy) is It can be expressed by the following general formula.

Here, r is the distance from the optical axis of the lens to the target pixel in the image plane, and (Cx, Cy) is the position of the optical axis of the lens. Further, k ₁ , k ₂ , k ₃ , ... Are lens distortion coefficients and depend on the lens design. The upper limit of the order is not particularly limited. It should be noted that the equation used for correcting the lens distortion in the present embodiment is not intended to be limited to Equation 1. When displaying on a flat plate display or performing image analysis, a general image corrected in this way is used. On the other hand, in the head-mounted display 100, in order for the images 18a and 18b without distortion to be visually recognized when viewed through the eyepiece, it is necessary to apply distortion opposite to the distortion caused by the eyepiece.

For example, in the case of a lens in which the four sides of the image appear to be recessed like a pincushion, the image is curved in a barrel shape. Therefore, the undistorted images 18a and 18b are distorted so as to correspond to the eyepieces, and are connected to the left and right according to the size of the display panel 122 to generate the final display image 22 (S12). The relationship between the image of the subject in the left and right regions of the display image 22 and the image of the subject in the images 18a and 18b without distortion before correction is equivalent to the relationship between the image having lens distortion of the camera and the image corrected for distortion. ..

Therefore, a distorted image in the display image 22 can be generated by the reciprocal vector of the displacement vector (Δx, Δy) of the equation 1. However, as a matter of course, the variable related to the lens is the value of the eyepiece. The image processing integrated circuit 120 of the present embodiment completes the removal and addition of distortion based on such two lenses in a single calculation (S14). Specifically, a displacement vector map is created in which the displacement vector indicating the position of the pixels on the original captured images 16a and 16b is displaced on the display image 22 by the correction is represented on the image plane.

If the displacement vector for removing the distortion caused by the camera lens is (Δx, Δy) and the displacement vector for adding the distortion for the eyepiece is (-Δx', −Δy'), the displacement vector maps will be displayed respectively. The displacement vector held at the position is (Δx−Δx', Δy−Δy'). Note that the displacement vector only defines the direction of pixel displacement and the amount of displacement, so if those parameters can be determined in advance, various corrections and combinations are possible, not limited to corrections caused by lens distortion. It can be easily realized with the same configuration.

When the display image 22 is generated, the pixels at the positions of the captured images 16a and 16b are moved by the displacement vector with reference to the displacement vector map. The captured images 16a and 16b may be corrected to generate display images for the left eye and the right eye, respectively, and may be connected later to generate the display image 22. Although the captured images 16a and 16b and the displayed image 22 are displaced by the amount of distortion, there is no significant change in the position or shape in which the image appears. Therefore, the pixel values of the captured images are acquired in order from the line above the image plane. In parallel with, the pixel value can be acquired and corrected. Then, in order from the upper stage, the display with low delay can be realized by performing the processing in the subsequent stage in parallel with the correction processing.

FIG. 5 is a diagram for explaining a process of generating a display image by synthesizing a virtual object transmitted from the content processing device 200 with a captured image in the image processing integrated circuit 120. The image 26 in the upper right of the figure is an image obtained by correcting the captured image as described in FIG. However, in this embodiment, it is not displayed as it is, and it is combined with the virtual object image 24 transmitted from the content processing device 200 to obtain the final display image 28. In this example, a cat object is synthesized.

As shown in the figure, the content processing device 200 generates an image 24 in which a cat object is drawn at an appropriate position to be fused with the captured image. At this time, images with parallax are generated for the left eye and the right eye, and distortion is applied based on the eyepiece of the head-mounted display 100 as shown in S12 of FIG. The content processing device 200 transmits an image 24 formed by connecting the distorted left and right images to the head-mounted display 100.

The image processing integrated circuit 120 of the head-mounted display 100 synthesizes the captured image 26 by fitting the cat object in the image 24 transmitted from the content processing device 200 into the image 26, and displays the display image 28. Generate. By drawing the object at an appropriate position in the image 24, for example, an image in which a cat object is standing on a table which is a real object is displayed. By viewing the display image 28 through the eyepiece, the user can see the image like the image 29 in three dimensions.

The generation source and transmission source of image data to be combined such as the image 24 of the virtual object are not limited to the content processing device 200. For example, a server connected to the content processing device 200 or the head-mounted display 100 via a network may be used as a generator or a source, or a module different from the image processing integrated circuit 120 built in the head-mounted display 100 is generated. It may be the source or source. These devices, including the content processing device 200, can also be regarded as an "image generation device". Further, a device that performs synthesis processing and generates a display device may be provided separately from the head-mounted display 100.

FIG. 6 is a diagram for explaining the content of data transmitted by the content processing device 200 in order for the image processing integrated circuit 120 to synthesize images. (A) is composed of an image (hereinafter referred to as "graphics image") 50 representing a virtual object to be synthesized in a display format, and an α image 52 representing an α value representing the transparency of the graphics image 50 on an image plane. It is data. Here, the α value is a general parameter in which 0 is transparent, 1 is opaque, and the intermediate value is semi-transparent with a density corresponding to the numerical value.

When only the cat object is opaquely combined in the illustrated example, an α image is generated in which the α value of the area of the cat object is 1 and the α value of the other areas is 0. The image processing integrated circuit 120 synthesizes an image obtained by correcting a captured image and a graphics image 50 transmitted from the content processing device 200 by the following calculation to generate a display image.
F _out = (1-α) F _i + α F _o

Here F _i, F _o, respectively, corrected pixel values at the same position in the captured image and the graphics image 50, alpha is alpha values of the same position in the alpha image, the F _out is the pixel value at the same position in the display image .. Actually, the above calculation is performed for each of the three channel images of red, green, and blue.

(B) is data consisting of an image 54 in which a region other than the virtual object to be combined is filled with a predetermined color such as green in the graphics image. In this case, the image processing integrated circuit 120 targets only the region of the image 54 whose pixel value is other than the predetermined color as the synthesis target, and replaces the pixels of the captured image. As a result, in the illustrated example, only the area of the cat object is replaced with the pixels of the captured image, and the other areas generate a display image in which the captured image remains. Such a synthesis method is known as chroma key synthesis.

The content processing device 200 acquires information related to the position and posture of the subject in the captured image from the head-mounted display 100, and draws a virtual object based on the positional relationship with the subject to generate a graphics image. At the same time, the α image 52 is generated, and the area other than the virtual object is filled with a predetermined color to generate the information necessary for the compositing process (hereinafter, referred to as “composite information”). By transmitting these to the head-mounted display 100 and synthesizing them, the amount of data to be transmitted can be reduced as a whole.

FIG. 7 shows a variation of the system configuration for transmitting data for synthesis from the content processing device 200 to the head-mounted display 100. (A) shows a case where the content processing device 200 and the head-mounted display 100 are connected by wire according to a standard such as DisplayPort. In (b), a relay device 310 is provided between the content processing device 200 and the head-mounted display 100, the content processing device 200 and the relay device 310 are connected by wire, and the relay device 310 and the head-mounted display 100 are Wi-Fi (registered). A case of wireless connection is shown by means of a trademark) or the like.

In the case of the configuration (a), since the cable is connected to the head-mounted display 100, if the content processing device 200 is a stationary type, the movement of the user can be hindered, but a relatively high bit rate can be secured. In the case of the configuration (b), in order to transmit data corresponding to the frame rate in wireless communication, it is necessary to increase the data compression rate as compared with the case of wired communication, but the movable range of the user can be expanded. In the present embodiment, by making it possible to support these system configurations, it is possible to perform optimization according to the communication environment and the required processing performance.

FIG. 8 shows the circuit configuration of the image processing integrated circuit 120 of the present embodiment. However, only the configuration according to this embodiment is shown, and the others are omitted. The image processing integrated circuit 120 includes an input / output interface 30, a CPU 32, a signal processing circuit 42, an image correction circuit 34, an image analysis circuit 46, a decoding circuit 48, a synthesis circuit 36, and a display controller 44.

The input / output interface 30 establishes communication with the content processing device 200 by wired communication and the relay device 310 by wireless communication, and realizes transmission / reception of data with either of them. In the present embodiment, the input / output interface transmits an image analysis result, a motion sensor measurement value, and the like to the content processing device 200. At this time as well, the relay device 310 may be relayed. Further, the input / output interface 30 receives the graphics image and composite information data generated by the content processing device 200 from the content processing device 200 or the relay device 310.

The CPU 32 is a main processor that processes and outputs signals such as image signals and sensor signals, commands and data, and controls other circuits. The signal processing circuit 42 acquires captured image data from the left and right image sensors of the stereo camera 110 at a predetermined frame rate, and performs necessary processing such as demosaic processing on each of them. The signal processing circuit 42 supplies data to the image correction circuit 34 and the image analysis circuit 46 in the order of the pixel sequences in which the pixel values are determined.

As described above, the image correction circuit 34 corrects each pixel in the captured image by displacing it by the displacement vector. The target for setting the displacement vector in the displacement vector map may be all pixels in the captured image plane, or may be only discrete pixels at predetermined intervals. In the latter case, the image correction circuit 34 first obtains the displacement destinations of the pixels for which the displacement vector is set, and obtains the displacement destinations of the remaining pixels by interpolation based on the positional relationship with those pixels.

When correcting chromatic aberration, the displacement vectors are different for each of the primary colors of red, green, and blue, so three displacement vector maps are prepared. Further, the image correction circuit 34 determines the pixel value by appropriately interpolating the surrounding pixel values for the pixel whose value is not determined by the displacement of the pixel in the display image. The image correction circuit 34 stores the pixel values determined in this way in the buffer memory. Then, in the see-through mode, data is output to the display controller 44 together with the determination of the pixel value in order from the line on the image plane. At the time of image composition, data is output to the composition circuit 36 in the same manner.

The image analysis circuit 46 acquires predetermined information by analyzing the captured image. For example, the distance between the subjects is obtained by stereo matching using the left and right captured images, and a depth map is generated in which the distance is expressed as a pixel value on the image plane. The position and orientation of the head-mounted display may be acquired by SLAM. In addition, it is understood by those skilled in the art that various things can be considered as the content of image analysis. The image analysis circuit 46 sequentially transmits the acquired information to the content processing device 200 via the input / output interface 30.

The decoding circuit 48 decodes and decompresses the data for synthesis received by the input / output interface 30. As described above, since the communication band changes depending on whether the communication with the other device is wired or wireless, in the present embodiment, the structure and compression method of the data for synthesis are switched accordingly. Therefore, the decoding circuit 48 appropriately selects a method suitable for the received data and performs decoding and decompression. The decoding circuit 48 sequentially supplies the decoded and decompressed data to the synthesis circuit 36.

As shown in FIG. 5, the synthesis circuit 36 combines the captured image supplied from the image correction circuit 34 and the graphics image supplied from the decoding circuit 48 into a display image. As described above, either alpha blending or chroma key synthesis may be adopted for the synthesis. When data of an image in which a graphics image and an α image are integrated is transmitted from the content processing device 200, the synthesis circuit 36 separates them, restores the graphics image and the α image, and then performs alpha blending.

When the data of the graphics image and the α image reduced in the predetermined direction are transmitted from the relay device 310, the synthesis circuit 36 enlarges and restores the images in the predetermined direction and then performs alpha blending. The image correction circuit 34 corrects chromatic aberration on the displayed image after synthesis, if necessary. Then, the synthesis circuit 36 or the image correction circuit 34 outputs data to the display controller 44 together with the determination of the pixel values in order from the line on the image plane.

FIG. 9 shows the internal circuit configuration of the content processing device 200. The content processing device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and a main memory 226. Each of these parts is connected to each other via a bus 230. An input / output interface 228 is further connected to the bus 230.

The input / output interface 228 stores data to a peripheral device interface such as USB or PCIe, a communication unit 232 composed of a wired or wireless LAN network interface, a storage unit 234 such as a hard disk drive or non-volatile memory, and a head mount display 100. An output unit 236 for output, an input unit 238 for inputting data from the head mount display 100, and a recording medium drive unit 240 for driving a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory are connected.

The CPU 222 controls the entire content processing device 200 by executing the operating system stored in the storage unit 234. The CPU 222 also executes various programs read from the removable recording medium, loaded into the main memory 226, or downloaded via the communication unit 232. The GPU 224 has a geometry engine function and a rendering processor function, performs drawing processing according to a drawing instruction from the CPU 222, and outputs the drawing process to the output unit 236. The main memory 226 is composed of a RAM (Random Access Memory) and stores programs and data required for processing.

FIG. 10 shows the configuration of the functional block of the content processing device 200 according to the present embodiment. The functional blocks shown in the figure and FIGS. 11 and 12 described later can be realized by a configuration of a CPU, GPU, various memories, etc. in terms of hardware, and a data input function loaded into the memory from a recording medium or the like in terms of software. , Data retention function, image processing function, communication function and other functions are realized by the program. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various ways by hardware only, software only, or a combination thereof, and is not limited to any of them.

The content processing device 200 includes a communication unit 258 that transmits / receives data to / from the head-mounted display 100 or the relay device 310, a synthesis data generation unit 250 that generates data for synthesis, and a compression that compresses and encodes the generated data. A coding unit 260 is provided. The communication unit 258 establishes communication with the head-mounted display 100 or the relay device 310 by wire, and receives the analysis result of the captured image performed by the head-mounted display 100. The communication unit 258 also transmits the synthesis data generated by the synthesis data generation unit 250 to the head-mounted display 100 or the relay device 310.

The composition data generation unit 250 includes a position / orientation prediction unit 252, an image drawing unit 254, and a composition information integration unit 256. The position / orientation prediction unit 252 predicts the position and orientation of the subject after a predetermined time based on the analysis result of the captured image performed by the head-mounted display 100. In the present embodiment, since the captured image is processed on the head-mounted display 100 side, it can be displayed with low delay. On the other hand, it takes a certain amount of time for the content processing device 200 to generate the corresponding graphics image by transmitting the analysis result of the captured image between the devices.

Therefore, when the graphics image is combined on the head-mounted display 100, the frame of the captured image of the composition destination is a frame after the frame that is the source of the analysis. Therefore, the time difference between the frame used for the image analysis and the frame of the synthesis destination is calculated in advance, and the position / orientation prediction unit 252 predicts the position and orientation of the subject after the time difference. By synthesizing the graphics image generated based on the prediction result with the captured image on the head-mounted display 100, it is possible to generate a display image with little deviation between the graphics and the captured image.

The position and orientation of the subject may be relative to the imaging surface of the stereo camera 110. Therefore, the information used for the prediction is not limited to the analysis result of the captured image, but may be the measured value of the motion sensor built in the head-mounted display 100, or may be combined as appropriate. Further, any of the general techniques may be adopted for predicting the position and posture of the subject.

The image drawing unit 254 generates a graphics image corresponding to the predicted position and posture information of the subject. As described above, since the purpose of image display is not particularly limited, the image drawing unit 254 advances, for example, an electronic game in parallel, and draws a virtual object based on the situation and the predicted position and posture of the subject. However, the processing performed by the image drawing unit 254 is not limited to drawing computer graphics. For example, a still image or a moving image acquired in advance may be played back or cut out to form an image to be combined. The image drawing unit 254 also generates composite information at the same time as generating the graphics image.

The composite information integration unit 256 integrates the graphics image and the composite information into composite data. When the composite information is an α image, the composite information integration unit 256 integrates the graphics image and the α image by representing them in one image plane. In the case of chroma key composition, filling an area other than the drawn object with a predetermined color is an integration process in the composition information integration unit 256.

In any case, the composite information integration unit 256 includes the composite information in the graphics image so that it can be handled as image data. The compression coding unit 260 compresses and encodes the image data for composition in which the graphics image and the composition information are integrated by a predetermined method. By realizing transmission from the content processing device 200 to the head-mounted display 100 or the relay device 310 by wired communication, transmission is possible even with lossless compression having a relatively low compression rate. Huffman coding and run-length coding can be used as examples of lossless compression. Further, the compression coding unit 260 may use lossy compression of image data for compositing even in wired communication.

FIG. 11 shows the configuration of the functional block of the relay device 310 according to the present embodiment. The relay device 310 is a communication unit 312 that establishes communication with the content processing device 200 and the head-mounted display 100 to transmit and receive data, and a data separation unit that separates data for synthesis transmitted from the content processing device 200 as necessary. 314, a compression coding unit 316 that appropriately recompresses the data for synthesis is provided.

The communication unit 312 establishes communication with the content processing device 200 by wire and receives data for synthesis. As described above, this data is a combination of graphics images and composite information. The communication unit 312 also establishes wireless communication with the head-mounted display 100 and transmits appropriately recompressed data for synthesis to the head-mounted display 100.

First, the data separation unit 314 decodes and decompresses the data for synthesis transmitted from the content processing device 200. By transmitting the losslessly compressed data from the content processing device 200, an image in which the original graphics image and the composite information are integrated is restored. When the composite information is an α image, the data separation unit 314 separates the integrated image into a graphics image and an α image.

On the other hand, when the image for chroma key composition is transmitted, the data separation unit 314 can omit the separation process. Alternatively, the data separation unit 314 generates an α image in which the pixel value of the area where the object is drawn is 1 and the pixel value of the other area filled with a predetermined color is 0 in the image for chroma key composition. The separation process may be carried out at. In this case, the subsequent processing is the same as when the α image is transmitted.

The compression coding unit 316 includes a first compression unit 318 and a second compression unit 320. Of the data separated by the data separation unit 314, the first compression unit 318 compresses and encodes the graphics image, and the second compression unit 320 compresses and encodes the α image. This α image may be generated from an image for chroma key composition. The first compression unit 318 and the second compression unit 320 have different compression coding methods. Preferably, the first compression unit 318 performs lossy compression having a higher compression rate than the second compression unit 320, and the second compression unit 320 performs lossless compression.

The relay device 310 plays a role of releasing the head-mounted display 100 from the communication cable. Therefore, it is necessary to compress the transmitted data as much as possible, but if the α image is irreversibly compressed in the same way as the graphics image, errors may be included in the most important contours, which may adversely affect the composite result. .. Therefore, the data size is reduced as a whole by once integrating and separating the graphics image and the α image transmitted by wire and performing appropriate coding processing on each. As a result, data transmission with less delay is realized even by wireless communication.

When an image for chroma key composition is transmitted from the content processing device 200 and the data separation unit 314 does not generate the corresponding α image, the entire image may be compressed and encoded by the first compression unit 318 at a high compression rate. However, in this case, the image decoded by the head-mounted display 100 may contain an error. As a result, it is conceivable that the pixel value of the region filled with the predetermined color changes by a minute amount, and the pixel of the captured image is replaced with the pixel value at the time of compositing.

Therefore, the synthesis circuit 36 of the head-mounted display 100 gives a width to the pixel values as a reference for determining that the area is not to be combined (the region that does not replace the pixel values). For example, if the pixel value of the fill color is (Cr, Cg, Cb) in the order of red, green, and blue, the captured image has a pixel value in the range of (Cr ± Δr, Cg ± Δg, Cb ± Δb). Do not synthesize to. Here, the optimum value of the pixel value margin (Δr, Δg, Δb) is set by an experiment or the like. As a result, even if the image for chroma key composition is compressed and encoded at a relatively high compression rate, deterioration of the accuracy of the composition process can be suppressed.

FIG. 12 shows the configuration of the functional block of the image processing device 128 built in the head-mounted display. As described above, this functional block can be realized by the configuration of the integrated circuit 120 for image processing shown in FIG. 8 in terms of hardware, and the data input function loaded from a recording medium or the like into the main memory or the like in terms of software. , Data retention function, image processing function, communication function and other functions are realized by the program.

In this example, the image processing device 128 includes a signal processing unit 150, an image analysis unit 152, a first correction unit 156, a signal processing unit 158, a synthesis unit 160, a second correction unit 162, and an image display control unit 164. The signal processing unit 150 is realized by the signal processing circuit 42 of FIG. 8, acquires data of a captured image from the image sensor of the stereo camera 110, and performs necessary processing. The image analysis unit 152 is realized by the CPU 32, the image analysis circuit 46, and the input / output interface 30 of FIG. 8, analyzes the captured image, acquires predetermined information, and transmits it to the content processing device 200.

For example, the distance between the subjects is obtained by stereo matching using the left and right captured images, and a depth map is generated in which the distance is expressed as a pixel value on the image plane. The position and orientation of the head-mounted display may be acquired by SLAM. In addition, it is understood by those skilled in the art that various things can be considered as the content of image analysis. However, depending on the case, the captured image data itself processed by the signal processing unit 150 may be transmitted to the content processing device 200.

In this case, the signal processing unit 150 includes the input / output interface 30 of FIG. Further, the analysis result by the image analysis unit 152 and the measurement value of the motion sensor (not shown) built in the head-mounted display 100 may be used for the image deformation processing. That is, the movement of the user's line of sight during the time required for processing inside the head-mounted display 100 and data transfer with the content processing device 200 may be specified based on these parameters and dynamically reflected in the displacement vector map.

The signal processing unit 150 may further perform super-resolution processing for increasing the definition of the captured image by a predetermined method. For example, the captured image is sharpened by synthesizing an image shifted in the horizontal and vertical directions of the image plane by a width smaller than one pixel and an image before shifting. In addition, various methods have been proposed for super-resolution, and any of them may be adopted.

The first correction unit 156 is realized by the CPU 32 and the image correction circuit 34 of FIG. 8 and corrects the captured image as shown in S14 of FIG. 4 to generate a display image having distortion for the eyepiece. However, when synthesizing the images transmitted from the content processing device 200, the first correction unit 156 does not perform chromatic aberration correction. That is, the same distortion is given to all primary colors. Considering the characteristics of the human eye looking at the display panel, all the red, green, and blue images are corrected using the displacement vector map generated for green. In addition, when the RAW image acquired by the image sensor is a Bayer array, green having the highest pixel density can be used.

In the case of the see-through mode in which the captured image is displayed without being combined with another image, the first correction unit 156 may generate a display image in which the chromatic aberration is corrected at once as described above. That is, the captured images of each color are corrected by using the displacement vector maps prepared for each of red, green, and blue. The signal processing unit 158 is realized by the CPU 32, the input / output interface 30, and the decoding circuit 48 of FIG. 8, and decodes and decompresses the data transmitted from the content processing device 200 or the relay device 310.

The compositing unit 160 is realized by the CPU 32 and the compositing circuit 36 of FIG. 8, and synthesizes a graphics image transmitted from the content processing device 200 or the like with the photographed image corrected by the first correction unit 156. The compositing unit 160 separates the graphics image and the α image, and enlarges and restores the reduced graphics image and the α image in a predetermined direction, if necessary.

The second correction unit 162 is realized by the CPU 32 and the image correction circuit 34 of FIG. 8 and corrects the image input from the synthesis unit 160. However, the second correction unit 162 performs only the correction that has not been performed, specifically the correction of chromatic aberration, among the corrections to be made for the displayed image. In the prior art, when a display image that has been composited by the content processing device 200 is transmitted to the head-mounted display 100 for display, a composite image without distortion is generated, and then correction for an eyepiece and chromatic aberration correction are performed. Is common. On the other hand, in the present embodiment, since the data paths of the captured image and the image to be combined with the captured image are different, the correction process is separated into two stages.

That is, the image transmitted from the content processing device 200 and the captured image are given a common distortion corresponding to the eyepiece, and are corrected for each color after composition. The second correction unit 162 further performs necessary corrections on the red and blue images of the combined image to complete the image. Color bleeding is performed by first correcting green, which is the wavelength band with the highest human visibility, and then scaling, super-resolution, and compositing, and then correcting red and blue aberrations. And contour abnormalities are less visible. However, the purpose is not to limit the order of colors used for correction. By leaving the correction of chromatic aberration after synthesis, the boundary line of synthesis can be clearly defined.

That is, when the image is combined after the chromatic aberration is corrected, the boundary line set in the chroma key image or the α image contains an error depending on the primary color, and the contour after the composition causes color bleeding. A display image without bleeding on the contour can be generated by shifting the pixels by a small amount by chromatic aberration correction after synthesizing without color shift. Filter processing such as bilinear or trilinear is generally used in processing such as image scaling, super-resolution, and composition. If these filtering processes are performed after the chromatic aberration is corrected, the result of the chromatic aberration correction is destroyed at a micro level, and color bleeding and abnormal contours occur at the time of display. By setting the processing of the second correction unit 162 immediately before the display, such a problem can be avoided. The image display control unit 164 is realized by the display controller 44 of FIG. 5, and the display images thus generated are sequentially output to the display panel 122.

The compression coding unit 260 of the content processing device 200, the data separation unit 314 and the compression coding unit 316 of the relay device 310, and the signal processing unit 158 of the image processing device 128 are used for each unit region formed by dividing the image plane. Compression coding, decoding / decompression, and motion compensation may be performed. Here, the unit area is divided in the horizontal direction for each predetermined number of lines, for example, one line or two lines of pixels, or divided in both vertical and horizontal directions such as 16 × 16 pixels and 64 × 64 pixels. It is a rectangular area formed by.

At this time, each of the above functional blocks starts the compression coding process and the decoding / decompression process each time the data to be processed for the unit area is acquired, and outputs the processed data for each unit area. A series of data with low delay by performing input / output control by a functional block related to a series of processing including compression coding and decoding / decompression in units of pixel data in a unit area that is less than the total number of pixels of the display image. Can be processed and transferred.

FIG. 13 illustrates an image configuration in which the content processing device 200 integrates a graphic image and an α image. In the configuration of (a), the α image data is embedded in the area 58 other than the range 56 in which the graphics image is represented in the image plane for one frame. That is, as described above, since the image displayed on the head-mounted display 100 is distorted due to the eyepiece, the range 56 of the graphics image is not rectangular.

On the other hand, since the data to be transmitted is premised on a rectangular plane in which the widths in the horizontal direction and the vertical direction are defined, an unused area 58 is generated between the data and the range 56 of the graphics image. Therefore, the synthetic information integration unit 256 of the content processing device 200 embeds an α value in the area 58. For example, as shown enlarged to the right, the pixel strings in the raster order of the α image are arranged from left to right and from top to bottom so as to fill the area 58.

In the configuration of (b), the α image and the graphics image are reduced in the vertical direction and connected vertically to form one image plane. In the configuration of (c), the graphics image and the α image are reduced in the horizontal direction and connected to the left and right to form one image plane. In the configurations of (b) and (c), the reduction ratios of the graphics image and the α image may be the same or different. In any case, according to these configurations, the α image can be easily transmitted even by a standard that does not support the α value channel. In the configuration of (b) and (c), the graphics image does not have to be a distorted image as shown in the figure. That is, the image may be a distortion-free image generated on the assumption that it is displayed on a flat plate display.

In these configurations, the resolution of the α image may be lower than the resolution of the graphics image. For example, in the configuration (a), the area of the region 58 may not be sufficient depending on the degree of distortion of the eyepiece. Therefore, the α image is reduced to 1/2 or 1/4 in both vertical and horizontal directions so that it can be accommodated in one frame of the image plane together with the graphics image. Alternatively, for the α image, only the data in the predetermined area including the drawn object may be the transmission target.

That is, it is clear that the α value is important at the time of compositing in the region of the object to be composited and the vicinity of the contour thereof, and the region far from the object remains as the captured image. Therefore, as shown in (a), only the α image of the region 60 in the predetermined range including the object is cut out and used as the embedding target. Here, the area of a predetermined range is, for example, an circumscribing rectangle of a virtual object or an area to which a margin area having a predetermined width is added. In this case, information on the position and size of the area 60 is also embedded at the same time. When drawing a plurality of virtual objects, naturally, the area of the α image to be cut is also a plurality.

FIG. 14 illustrates a data structure of pixel values of an α image that the content processing apparatus 200 integrates into a graphics image. Here, it is assumed that each pixel value of the graphics image is represented by 24 bits (the brightness of red (R), green (G), and blue (B) is 8 bits each). The rectangle indicated by "A" in the figure is the bit width representing one α value. In the data structure of (a), the α value is set to 1 bit, and the α value for 8 × 3 = 24 pixels is set to the data of 1 pixel on the image. Similarly, in the data structures of (b), (c), and (d), the α values are set to 2 bits, 4 bits, and 8 bits, respectively, so that the α values for 12 pixels, 6 pixels, and 3 pixels can be obtained. , It is the data of one pixel on the image.

On the other hand, in the data structures (e), (f), and (g), each color is associated with an α value for one pixel. That is, the number of bits representing the α value is different from 1 bit, 2 bits, and 4 bits, but in any structure, the α value for 1 × 3 = 3 pixels is used as the data of 1 pixel on the image. When translucent composition is not performed, the α value is 0 or 1, so the data structure of (a) or (e) is sufficient. In addition, an appropriate data structure is selected in consideration of the gradation, resolution, area of the transmission target, etc. required for transparency.

In addition, the actual display image can be visually matched by performing chromatic aberration correction that gives a slight amount of misalignment to the image of the object for the images of the red, green, and blue channels. Need to be. When performing such chromatic aberration correction on the image generated by the content processing device 200, it is necessary to generate an α image for each color as the image is displaced. In this case, the α image itself will have red, green, and blue channels. Therefore, in the illustrated data structure, the red α value, the green α value, and the blue α value are stored in each of the red, green, and blue channels, respectively.

As a result, the data size of the α value that can be stored in one pixel on the image is 1/3 of the case where the α value common to all colors is set as described above. However, it is also conceivable that the content processing device 200 does not correct the chromatic aberration, but the head-mounted display 100 synthesizes the content and then corrects the chromatic aberration. In this case, since the α value common to all colors can be set, more information can be stored in one pixel.

FIG. 15 shows a processing procedure in the case of embedding α image data in an area where a graphics image is not displayed and transmitting the data, as shown in FIG. 13A. When the transmission data has such a configuration, the extraction of the α image becomes relatively complicated. Therefore, as shown in FIG. 7A, the transmission data is directly transmitted from the content processing device 200 to the head-mounted display 100 by wire. Is desirable. As a result, the data can be losslessly compressed and highly accurate synthesis can be realized.

When the compositing unit 160 of the head-mounted display 100 acquires the data of the image 70 in which the α image is embedded in this way, it separates the data from the graphics image. Specifically, the pixel values in the range representing the α value are extracted while scanning the pixel sequence from left to right in order from the row above the image plane. Here, in the region representing the α value, as shown in FIG. 14, a plurality of α values are stored in one pixel. Therefore, the compositing unit 160 decomposes them into one pixel and arranges them on the plane of the α image 74 in raster order.

For example, in the y-th row shown in the figure, the pixel values from the leftmost pixel to x0 are read, then the pixel values from x1 to x2 and from x3 to the right end are read out. In this way, the range of pixels representing the α value in the image plane is derived line by line based on the design of the eyepiece lens and shared with the content processing device 200. A map showing the range of pixels in which the α value is embedded on an image plane may be created and shared between the content processing device 200 and the head-mounted display 100. As a result, the graphics image 72 and the α image 74 are separated. Therefore, as shown in FIG. 5, the compositing unit 160 generates a display image by combining the graphics image with the captured image using the α value. When the graphics image 72 and the captured image are not corrected for chromatic aberration, the compositing unit 160 performs chromatic aberration correction on the combined image. The same applies to the aspects described below.

As shown in FIG. 13B, FIG. 16 shows a processing procedure in the case where the α image and the graphics image are reduced in the vertical direction and connected vertically to be transmitted. In this case, as shown in FIG. 7A, the content processing device 200 may directly transmit to the head-mounted display 100 by wire, or as shown in FIG. 7B, once transmit to the relay device 310. Then, it may be transmitted wirelessly from the relay device 310 to the head-mounted display 100. FIG. 16 shows the latter case.

When the data separation unit 314 of the relay device 310 acquires the data of the image 76 from the content processing device 200, it separates the data into the α image 80 and the graphics image 78. Therefore, in the data separation unit 314, the position of the boundary line between the α image and the graphics image, which is determined by the reduction ratio in the vertical direction, is set in advance. The separated α image 80 and the graphics image 78 are compressed and coded by the compression coding unit 316 while being reduced in the vertical direction. As described above, the graphics image 78 selects a high compression rate, and the α image 80 selects a low compression rate coding method.

Then, the communication unit 312 wirelessly transmits the two types of compression-encoded data to the head-mounted display 100. However, in reality, the relay device 310 acquires in order from the upper line of the image 76, compresses and encodes it in order from the upper line in parallel with the acquisition, and transmits it to the head-mounted display 100. Therefore, in the illustrated example, the data of the α image 80 is first transmitted, and then the data of the graphics image 78 is transmitted.

The signal processing unit 158 of the head-mounted display 100 decodes and decompresses the two types of data by a method corresponding to each. The compositing unit 160 acquires the α image 80 and the graphics image 78 after decoding and decompression, and enlarges them in the vertical direction to obtain the α image 84 and the graphics image 82 of the original size. When a plurality of α values are stored in one pixel in the transmitted data, the compositing unit 160 appropriately decomposes the α values to generate the α image 84. Then, as shown in FIG. 5, the compositing unit 160 generates a display image by compositing the graphics image 82 with the captured image using the α image 84.

FIG. 17 shows a processing procedure in the case where the graphics image and the α image are reduced in the horizontal direction and connected to the left and right, respectively, as shown in FIG. 13 (c). In this case as well, as shown in FIG. 7A, the content processing device 200 may directly transmit to the head-mounted display 100 by wire, or as shown in FIG. 7B, once transmit to the relay device 310. Then, it may be transmitted wirelessly from the relay device 310 to the head-mounted display 100. FIG. 17 shows the latter case.

When the data separation unit 314 of the relay device 310 acquires the data of the image 86 from the content processing device 200, it separates the data into the graphics image 88 and the α image 90. Therefore, in the data separation unit 314, the position of the boundary line between the graphics image and the α image, which is determined by the horizontal reduction ratio, is set in advance. The separated graphics image 88 and α image 90 are compressed and coded by the compression coding unit 316 while being reduced in the horizontal direction. As described above, the graphics image 88 has a high compression ratio, and the α image 90 has a low compression ratio.

Then, the communication unit 312 wirelessly transmits the two types of compression-encoded data to the head-mounted display 100. However, in reality, the relay device 310 acquires in order from the upper line of the image 86, compresses and encodes it in order from the upper line in parallel with the acquisition, and transmits it to the head-mounted display. Therefore, in the illustrated example, one line of graphics image data and one line of α image data are transmitted alternately.

The signal processing unit 158 of the head-mounted display 100 decodes and decompresses the two types of data by a method corresponding to each. The compositing unit 160 acquires the graphics image 88 and the α image 90 after decoding and decompression, and enlarges them in the lateral direction to obtain the graphics image 92 and the α image 94 of the original size. When a plurality of α values are stored in one pixel in the transmitted data, the compositing unit 160 appropriately decomposes them to generate an α image 94. Then, as shown in FIG. 5, the compositing unit 160 generates a display image by compositing the graphics image 92 with the captured image using the α image 94.

In order to properly realize stereoscopic vision in the head-mounted display 100, it is important that the displayed image is accurately given binocular parallax. As shown in FIG. 16, when a vertically reduced image is connected and transmitted, the resolution in the horizontal direction is maintained, which is advantageous in stereoscopic viewing. On the other hand, in the aspect of FIG. 16, as described above, the transmission of the graphics image is started after the α image of the entire region is transmitted. Therefore, also in the head-mounted display 100, the compositing unit 160 has no choice but to start compositing the graphics images to be acquired after acquiring the α image of the entire region, which causes a delay in the period for acquiring the α image.

By transmitting the α image having a small data size first, the delay time can be shortened as compared with the case where the graphics image is transmitted first, but in any case, there is a waiting time until both data are collected. Based on this, when the graphics image and the α image are reduced in the vertical direction, the content processing device 200 may transmit the graphics image and the α image by alternately connecting one line or a plurality of lines. By doing so, the waiting time until both data are collected can be further shortened.

As shown in FIG. 17, when a horizontally reduced image is connected and transmitted, an error may be included in the binocular parallax due to a decrease in the horizontal resolution, which may affect stereoscopic vision. is there. On the other hand, as described above, since the graphics image and α image data are included in one line, the composition of each line can be efficiently advanced and the delay time can be minimized. As described above, based on the characteristics of the configurations shown in FIGS. 16 and 17, an appropriate configuration is selected according to a balance such as required stereoscopic accuracy and allowable delay time.

According to the present embodiment described above, in the technique of displaying the image of the content on the head-mounted display provided with the camera, the captured image is stored in the head-mounted display separately from the path for displaying the image transmitted from the content processing device. Provide a route to process and display with. When realizing augmented reality or mixed reality, only the analysis result of the captured image is sent from the head-mounted display to the content processing device, and the graphics image drawn by the content processing device is combined with the captured image on the head-mounted display. To do.

As a result, while the content processing device can draw a high-definition image, the size of data to be transmitted to and from the head-mounted display can be reduced, and the time and power consumption required for each processing and data transmission can be reduced. In addition, when the α image required for compositing is transmitted together with the graphics image, by integrating them into one image data, it is possible to transmit even a communication standard that does not support data transmission of the α image. .. At this time, by embedding the α image in the gap region generated by the distortion given to the graphics image in the image plane to be transmitted and transmitting the image by wire communication, the graphics image can be synthesized without deterioration.

Alternatively, the graphics image and the α image are reduced in the vertical direction or the horizontal direction and connected to be transmitted as one image data. In this case, the two are once separated in the relay device, the graphics image is compressed and encoded at a high compression rate, and the α image is losslessly compressed and transmitted to the head-mounted display. As a result, the size of data to be transmitted can be reduced while suppressing the influence on the accuracy at the time of synthesis. As a result, data transmission to the head-mounted display can be realized wirelessly, and the movable range of the user can be expanded.

The present invention has been described above based on the embodiments. Embodiments are examples, and it is understood by those skilled in the art that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. ..

30 Input / output interface, 32 CPU, 34 Image correction circuit, 36 Synthesis circuit, 42 Signal processing circuit, 44 Display controller, 46 Image analysis circuit, 48 Decoding circuit, 100 Head mount display, 110 Stereo camera, 120 Integrated circuit for image processing , 122 Display panel, 128 Image processing device, 150 Signal processing unit, 152 Image analysis unit, 156 First correction unit, 158 Signal processing unit, 160 Synthesis unit, 162 Second correction unit, 164 Image display control unit, 200 Content processing Device, 222 CPU, 224 GPU, 226 main memory, 232 communication unit, 234 storage unit, 250 data generation unit for synthesis, 252 position / orientation prediction unit, 254 image drawing unit, 256 composition information integration unit, 258 communication unit, 260 compression Encoding unit, 310 relay device, 312 communication unit, 314 data separation unit, 316 compression coding unit, 318 first compression unit, 320 second compression unit.

Claims (27)

The image generator
An image to be combined with the display image, a step of generating an α value representing the transparency of the pixels of the image to be combined, and
A step of generating composite data in which the image to be composited and the α value data are represented on one image plane, and
The step of transmitting the composite data to the device that generates the display image, and
An image data transmission method comprising. The step of generating the compositing data adds a distortion to be given to the display image viewed through the eyepiece to the compositing image, and the distortion does not represent the compositing image in the image plane. The image data transmission method according to claim 1, wherein the α value is embedded in the region. Claim 1 is characterized in that the step of generating the composite data is represented on the one image plane by connecting the image to be composited and the image representing the α value in a predetermined direction. The image data transmission method described in. In the step of generating the compositing data, the image to be composited and the image representing the α value are reduced in the vertical direction of the image plane, and are alternately connected every predetermined number of rows. The image data transmission method according to claim 1, wherein the image data is represented on one image plane. The image data transmission method according to claim 3 or 4, wherein the image to be combined is a distortion-free image generated on the premise that the image is displayed on a flat plate display. The image data transmission method according to any one of claims 1 to 5, wherein the step of generating the synthesis data is a step of associating one pixel with α values of a plurality of pixels in the synthesis data. The sixth aspect of claim 6, wherein the step of generating the composite data stores the α value of one or a plurality of pixels for each storage area allocated to the plurality of primary colors of the one pixel. Image data transmission method. The step of generating the composite data is characterized in that the α value of one or a plurality of pixels set for each primary color is stored for each storage area assigned to the plurality of primary colors of the one pixel. The image data transmission method according to claim 6. The step of generating the composite data is characterized in that the α value of one or a plurality of pixels, which is commonly set for all the primary colors, is stored for each storage area allocated to the plurality of primary colors of the one pixel. The image data transmission method according to claim 6. The relay device
A step of acquiring the composite data and separating the image to be composited from the α value data,
The step of compressing and coding the image to be combined and the α value data by different methods,
A step of transmitting the compressed coded data to the device for generating the display image, and
The image data transmission method according to any one of claims 1 to 9, further comprising. The image data transmission method according to claim 10, wherein the compression coding step irreversibly compresses the image to be synthesized and losslessly compresses the α value data. The device for generating the display image further includes a step of acquiring information on the position and posture of the subject.
The step of generating the synthesis data is
Based on the position and posture information transmitted from the device that generates the display image, the position and posture of the subject after a predetermined time are predicted, and then the image to be combined is generated so as to correspond to the result. The image data transmission method according to any one of claims 1 to 11. Claim 1 is characterized in that the step of generating the composite data represents an α value on the image plane based on the information related to the range representing the α value shared with the apparatus for generating the display image. The image data transmission method according to any one of 12 to 12. The image data transmission method according to claim 13, wherein the step of generating the synthesis data is based on a map in which the range representing the α value is represented on the image plane, and the α value is represented on the image plane. .. The image data transmission method according to any one of claims 1 to 14, wherein the step of generating the composite data represents an α value generated at a resolution smaller than the image to be composited on the image plane. .. Any of claims 1 to 14, wherein the step of generating the compositing data represents an α value of a region of a predetermined range including an image represented by the compositing in the display image on the image plane. The image data transmission method described in. In the step of generating the composite data, instead of displaying the α value data on the one image plane, the composite data in which a predetermined pixel value is given to a region other than the image to be represented by the composite is generated.
In the transmission step, the synthesis data is irreversibly compressed and transmitted.
The device that generates the display image
The step of expanding the synthetic data and
A step of displaying a region other than a pixel region having a value in a predetermined range from the predetermined pixel value in the display image in the composite data,
The image data transmission method according to any one of claims 1 to 16, wherein the image data transmission method comprises. An image drawing unit that generates an image to be combined with the display image,
A composite information integration unit that generates composite data in which the image to be composited and the α value data representing the transparency of the pixels of the image to be composited are represented on one image plane.
The communication unit that outputs the synthesis data and
A content processing device characterized by being equipped with. The content processing device according to claim 18, wherein the image drawing unit adds distortion to be given to the display image viewed through the eyepiece to the image to be synthesized. A camera that shoots real space,
The composition data, which represents the image to be combined with the display image and the α value data representing the transparency of the pixels of the image to be combined on one image plane, is received from an external device and converted to the α value. Based on this, an integrated circuit for image processing that synthesizes the image to be combined with the image taken by the camera to generate a display image, and
A display panel that outputs the display image and
A head-mounted display featuring a head-mounted display. The integrated circuit for image processing adds distortion to be given to the display image viewed through the eyepiece to the captured image, and is transmitted from the external device and should be synthesized with the distortion added. The head-mounted display according to claim 20, wherein the images are combined. The head-mounted display according to claim 20 or 21, wherein the integrated circuit for image processing analyzes the captured image to acquire information on the position and posture of the subject and transmits the information to the external device. The composition data in which the image to be combined with the display image and the α value data representing the transparency of the pixels of the image to be combined are represented on one image plane is converted into the image to be combined and the α value data. The data separator to be separated and
A compression coding unit that compresses and encodes the image to be synthesized and the α value data by different methods.
A communication unit that acquires the composite data from an image generating device and transmits the compressed coded data to the display image generating device.
A relay device characterized by being equipped with. Includes a display device and a content processing device that generates an image to be displayed on the display device.
The content processing device is
A compositing data generation unit that generates compositing data in which an image to be composited with a display image and α value data representing the transparency of pixels of the image to be composited are represented on one image plane.
The communication unit that outputs the synthesis data and
With
The display device is
A camera that shoots real space,
An integrated circuit for image processing that generates a display image by synthesizing the image to be combined with the image taken by the camera based on the α value of the composition data.
A display panel that outputs the display image and
A content processing system characterized by being equipped with. A function to generate an image to be combined with the display image,
A function of generating composite data in which the image to be composited and α value data representing the transparency of the pixels of the image to be composited are represented on one image plane.
The function to output the synthesis data and
A computer program characterized by realizing a computer. A function to acquire composite data in which an image to be composited with a display image and α value data representing the transparency of the pixels of the image to be composited are represented on one image plane.
The function of separating the image to be combined and the α value data, and
The function of compressing and coding the image to be combined and the α value data by different methods,
A function of transmitting compressed coded data to a device that generates the display image, and
A computer program characterized by realizing a computer. A function to acquire composite data in which an image to be composited with a display image and α value data representing the transparency of the pixels of the image to be composited are represented on one image plane.
The function to restore the image to be combined and the α value data,
Based on the α value, the function to synthesize the image to be combined with the captured image and output it to the display panel,
A computer program characterized by realizing a computer.