JP2020167659A - Image processing apparatus, head-mounted display, and image display method - Google Patents

Abstract

To appropriately achieve both appreciation of an augmented reality content and checking of the surrounding state using a closed-type head-mounted display.SOLUTION: In an image processing apparatus 128 of a head-mounted display, a signal processing unit 150 acquires a photographic image from a stereo camera 110. A first correction unit 156 executes part of correction required for the photographic image. A composition unit 160 composites an image from a content processing apparatus 200 with the photographic image. A second correction unit 162 executes the remaining correction required for the photographic image and outputs the corrected image to a display panel 122.SELECTED DRAWING: Figure 10

Description

The present invention relates to a head-mounted display that displays an image in front of the user's eyes, an image processing device that processes the displayed image, and an image display method performed therein.

Image display systems that allow users to view the target space from a free viewpoint are widespread. For example, a system has been developed in which a panoramic image is displayed on a head-mounted display and an image corresponding to the line-of-sight direction of a user wearing the head-mounted display is displayed. By using a head-mounted display, it is possible to enhance the immersive feeling in the image and improve the operability of applications such as games. In addition, a walk-through system has also been developed that allows a user wearing a head-mounted display to virtually walk around in the space displayed as an image by physically moving.

There are two types of head-mounted displays: a shield type that covers the user's field of view to block light from the outside world, and an optical transmission type that takes in light from the outside world and makes it possible to see the surroundings. When realizing augmented reality (AR: Augmented Reality) or mixed reality (MR: Mixed Reality) that fuses real space and virtual objects (virtual space) with a head-mounted display, the optical transmission type is the superimposed display. It is excellent in terms of ease of display, delay time to display, and low system load (power consumption) required for drawing. On the other hand, the shielded head-mounted display can completely block the external field of view, so that an immersive virtual reality can be realized.

In a shielded head-mounted display, basically, only the light emitted from the display panel is a visual stimulus. Therefore, for example, if there is a period from when the head-mounted display is attached until the image of the content is displayed, or after the display is finished, the viewer naturally cannot see anything. As a result, there is a risk of tripping or bumping into surrounding objects during such periods. Also, if you want to see the surrounding situation such as picking up a controller placed nearby while the image of the virtual world is displayed, you need to remove the head-mounted display each time.

When realizing augmented reality or mixed reality with a shielded head-mounted display, it is conceivable to provide a camera in front of the head-mounted display and combine the captured image with a separately generated image such as a virtual object. However, as various processes increase, power consumption increases, and there are cases where a camera and a device for generating contents such as virtual objects cannot be integrally provided. As a result, a delay is likely to occur from shooting to display, which tends to give the user a sense of discomfort.

The present invention has been made in view of these problems, and an object of the present invention is to be able to visually recognize the surrounding situation without discomfort while wearing a shielded head-mounted display, and to appreciate contents in which virtual objects and the like are synthesized. The purpose is to provide technology that can be compatible with each other appropriately.

In order to solve the above problems, an aspect of the present invention relates to an image processing apparatus. This image processing device has a signal processing unit that acquires data of a captured image, a correction unit that corrects the captured image into an image suitable for display, and a composite image transmitted from a device that is not integrally provided. It includes a compositing unit that synthesizes the captured image and an image display control unit that displays the composited image on the display panel, and the correction unit is a process that corrects the image to be suitable for the display before the compositing unit. It is characterized in that a part of the correction is performed on the captured image of the above, and the remaining correction is performed on the combined image.

Another aspect of the invention relates to a head-mounted display. The head-mounted display is characterized by including the above-mentioned image processing device, an imaging device for supplying a captured image to the signal processing unit thereof, and a display panel.

Yet another aspect of the present invention relates to an image display method. This image display method is integrally provided with a step in which the image processing device acquires data of the captured image and a step of performing some corrections in the process of correcting the captured image to an image suitable for display. Performs the remaining corrections in the step of synthesizing the composite image transmitted from the non-existing device into the captured image with some corrections and the process of correcting the composited image to an image suitable for display. It is characterized by including a step of displaying a display image and a step of displaying the display image on a display panel.

It should be noted that any combination of the above components and the conversion of the expression of the present invention between methods, devices, systems, computer programs, data structures, recording media and the like are also effective as aspects of the present invention.

According to the present invention, in a shielded head-mounted display, it is possible to achieve both visual recognition of the surrounding situation without discomfort and viewing of content in which a virtual object or the like is synthesized, with low delay and low power consumption.

It is a figure which shows the appearance example of the head-mounted display of this embodiment. It is a figure which shows the configuration example of the content processing system of this embodiment. It is a figure which shows typically the path of the data in the content processing system of this embodiment. It is a figure for demonstrating the process of generating the display image from the photographed image in the integrated circuit for image processing of this embodiment. It is a figure which shows the circuit structure of the integrated circuit for image processing of this embodiment. It is a figure for demonstrating the procedure of display processing in the see-through mode of this embodiment. It is a figure for demonstrating the significance of this Embodiment in the time until the image without distortion is processed and displayed. It is a figure for demonstrating an example of the processing procedure which the correction circuit of this embodiment corrects a photographed image. It is a figure for demonstrating the capacity of the buffer memory required for the correction process in this embodiment. It is a figure which shows the structure of the functional block of the content processing system when the integrated circuit for image processing of this embodiment is used in the display mode other than the see-through mode. In this embodiment, it is a figure for demonstrating the example of the element included in the displacement vector as chromatic aberration correction. It is a figure which shows typically the data stored in the displacement vector map memory of this embodiment. It is a flowchart which shows the processing procedure when the head-mounted display of this embodiment displays a photographed image and the like.

FIG. 1 shows an example of the appearance of the head-mounted display 100. In this example, the head-mounted display 100 is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that goes around the head and realizes fixing of the device when the user wears it. The output mechanism 102 includes a housing 108 having a shape that covers the left and right eyes when the head-mounted display 100 is worn by the user, and includes a display panel inside so as to face the eyes when worn.

Further, the inside of the housing 108 is provided with an eyepiece located between the display panel and the user's eyes when the head-mounted display 100 is attached to magnify the image. Further, the head-mounted display 100 may further include a speaker or earphone at a position corresponding to the user's ear when worn. Further, the head-mounted display 100 has a built-in motion sensor, and may detect the translational motion and the rotational motion of the head of the user wearing the head-mounted display 100, and eventually the position and posture at each time.

The head-mounted display 100 further includes a stereo camera 110 on the front surface of the housing 108, a monocular camera 111 with a wide viewing angle in the center, and four cameras 112 with a wide viewing angle at the four corners of the upper left, upper right, lower left, and lower right. Take a video of the real space in the direction corresponding to the direction of the face. The present embodiment provides a mode in which an image taken by the stereo camera 110 is immediately displayed to show the state of the real space in the direction in which the user is facing. Hereinafter, such a mode is referred to as a "see-through mode". The head-mounted display 100 basically shifts to the see-through mode during the period when the image of the content is not displayed.

By automatically shifting to the see-through mode of the head-mounted display 100, the user can check the surrounding situation without removing the head-mounted display 100 before the content starts, after the content ends, or when the content is interrupted. In addition to this, the transition timing to the see-through mode may be when the user explicitly performs the transition operation. As a result, even while viewing the content, the display can be temporarily switched to the image in the real space at any timing, and necessary work such as finding the controller and picking it up can be performed.

At least one of the images captured by the stereo camera 110, the monocular camera 111, and the four cameras 112 can also be used as an image of the content. For example, augmented reality (AR: Augmented Reality) and mixed reality (MR) can be created by synthesizing virtual objects with captured images and displaying them in positions, postures, and movements that correspond to the actual space in the image. realizable. In this way, regardless of whether or not the captured image is included in the display, the position, posture, and movement of the object to be drawn can be determined by using the analysis result of the captured image.

For example, the corresponding points may be extracted by performing stereo matching on the captured image, and the distance of the subject may be acquired by the principle of triangulation. Alternatively, the head-mounted display 100 with respect to the surrounding space, and thus the position and posture of the user's head may be acquired by SLAM (Simultaneous Localization and Mapping). It can also perform object recognition and object depth measurement. By these processes, the virtual world can be drawn and displayed in the field of view corresponding to the position of the user's viewpoint and the direction of the line of sight.

The head-mounted display 100 of the present embodiment is a shield-type head-mounted display that blocks the user's view, and is provided with a camera that captures a real space with a field of view corresponding to the position and orientation of the user's face. , The actual shape is not limited to the one shown in the figure. Further, if the images of the field of view of the left eye and the field of view of the right eye are simulated in the see-through mode, a monocular camera or four cameras 112 can be used instead of the stereo camera 110.

FIG. 2 shows a configuration example of the content processing system according to the present embodiment. The head-mounted display 100 is connected to the content processing device 200 by an interface 300 that connects peripheral devices such as wireless communication or USB Type-C. A flat plate display 302 is connected to the content processing device 200. The content processing device 200 may be further connected to the server via a network. In that case, the server may provide the content processing device 200 with an online application such as a game in which a plurality of users can participate via a network.

The content processing device 200 basically processes a content program, generates a display image, and transmits the display image to the head-mounted display 100 or the flat-plate display 302. In a certain aspect, the content processing device 200 specifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head-mounted display 100, and generates a display image of the corresponding visual field at a predetermined rate. To do.

The head-mounted display 100 receives the data of the display image and displays it as an image of the content. To this extent, the purpose of displaying the image is not particularly limited. For example, the content processing device 200 may generate a virtual world, which is the stage of the game, as a display image while advancing the electronic game, or may use a still image or a still image for viewing or providing information regardless of whether the virtual world or the real world is used. A moving image may be displayed.

The distance between the content processing device 200 and the head-mounted display 100 and the communication method of the interface 300 are not limited. For example, the content processing device 200 may be a game device owned by an individual, a server of a company or the like that provides various distribution services such as a cloud game, or a home server that transmits data to an arbitrary terminal. Therefore, communication between the content processing device 200 and the head mount display 100 is in addition to the above-mentioned examples, in public networks such as the Internet, LAN (Local Area Network), mobile phone carrier networks, Wi-Fi spots in the city, and homes. It may be realized via an arbitrary network or access point such as a Wi-Fi access point.

FIG. 3 schematically shows a data path in the content processing system of the present embodiment. The head-mounted display 100 includes a stereo camera 110 and a display panel 122 as described above. However, as described above, the camera is not limited to the stereo camera 110, and may be any or a combination of the monocular camera 111 and the four cameras 112. The same applies to the following description. The display panel 122 is a panel having a general display mechanism such as a liquid crystal display or an organic EL display, and displays an image in front of a user wearing the head-mounted display 100. Further, an integrated circuit 120 for image processing is provided inside.

The image processing integrated circuit 120 is, for example, a system-on-chip equipped with various functional modules including a CPU. In addition to this, the head mount display 100 includes motion sensors such as a gyro sensor, an acceleration sensor, and an angular acceleration sensor, a main memory such as a DRAM (Dynamic Random Access Memory), an audio circuit that allows the user to hear voice, and peripheral devices. Peripheral device interface circuits and the like for connecting the devices may be provided, but the illustrations are omitted here.

When augmented reality or mixed reality is realized by a shield-type head-mounted display, generally, an image taken by a stereo camera 110 or the like is taken into a subject that processes the content, and then combined with a virtual object to generate a display image. Since the main body that processes the content in the illustrated system is the content processing device 200, as shown by the arrow B, the image taken by the stereo camera 110 is once transmitted to the content processing device 200 via the image processing integrated circuit 120. Will be done.

Then, the virtual object is synthesized and returned to the head-mounted display 100, and is displayed on the display panel 122. On the other hand, in the present embodiment, as a see-through mode, a data path different from the content processing is provided. That is, as shown by the arrow A, the image taken by the stereo camera 110 is appropriately processed by the image processing integrated circuit 120 and displayed as it is on the display panel 122. At this time, the image processing integrated circuit 120 only performs the process of correcting the captured image into a format suitable for display.

According to the path of the arrow A, since the data transmission path is remarkably shortened as compared with the arrow B, the time from image capture to display can be shortened, and the power consumption required for transmission can be reduced. Further, in the present embodiment, the correction processing in the image processing integrated circuit 120 is performed in parallel with the shooting without waiting for the shooting of one frame in the stereo camera 110, and is sequentially output to the display panel 122.

With these configurations, it is possible to instantly display a captured image corresponding to the orientation of the user's face, and to create a state similar to looking at the surroundings without going through the display. The path of the arrow A is not limited to the see-through mode, and can be used when synthesizing the captured image with the image generated by the content processing device 200. That is, only the image data to be combined is transmitted from the content processing device 200, combined with the captured image in the image processing integrated circuit 120 of the head-mounted display 100, and then output to the display panel 122.

In this way, it is sufficient to transmit only the information related to the real space acquired from the captured image to the content processing device 200 from the head-mounted display 100 instead of the data of the captured image. As a result, the time and power consumption required for data transmission can be reduced as compared with the case where the captured image data itself is transmitted and synthesized by the content processing device 200.

FIG. 4 is a diagram for explaining a process of generating a display image from a captured image in the image processing integrated circuit 120. In real space, suppose a table on which objects are placed is in front of the user. By photographing the stereo camera 110, the photographed image 16a of the left viewpoint and the photographed image 16b of the right viewpoint are acquired. Due to the parallax of the stereo camera 110, the captured images 16a and 16b are displaced in the horizontal direction at the positions of the images of the same subject.

In addition, the lens of the camera causes distortion in the image of the subject. Generally, such lens distortion is corrected to generate a distortion-free left-viewpoint image 18a and a right-viewpoint image 18b (S10). Here, assuming that the pixels of the position coordinates (x, y) in the original images 16a and 16b are corrected to the position coordinates (x + Δx, y + Δy) in the corrected images 18a and 18b, the displacement vector (Δx, Δy) is It can be expressed by the following general formula.

Here, r is the distance from the optical axis of the lens to the target pixel in the image plane, and (Cx, Cy) is the position of the optical axis of the lens. Further, k ₁ , k ₂ , k ₃ , ... Are lens distortion coefficients and depend on the lens design. The upper limit of the order is not particularly limited. It should be noted that the expression used for the correction in the present embodiment is not intended to be limited to the expression 1. When displaying on a flat plate display or performing image analysis, a general image corrected in this way is used. On the other hand, in the head-mounted display 100, in order for the images 18a and 18b without distortion to be visually recognized when viewed through the eyepiece, it is necessary to apply distortion opposite to the distortion caused by the eyepiece.

For example, in the case of a lens in which the four sides of the image appear to be recessed like a pincushion, the image is curved in a barrel shape. Therefore, the undistorted images 18a and 18b are distorted so as to correspond to the eyepieces, and are connected to the left and right according to the size of the display panel 122 to generate the final display image 22 (S12). The relationship between the image of the subject in the left and right regions of the display image 22 and the image of the subject in the images 18a and 18b without distortion before correction is equivalent to the relationship between the image having lens distortion of the camera and the image corrected for distortion. ..

Therefore, a distorted image in the display image 22 can be generated by the reciprocal vector of the displacement vector (Δx, Δy) of the equation 1. However, as a matter of course, the variable related to the lens is the value of the eyepiece. The image processing integrated circuit 120 of the present embodiment completes the removal and addition of distortion based on such two lenses in a single calculation (S14). Specifically, a displacement vector map is created in which the displacement vector indicating the position of the pixels on the original captured images 16a and 16b is displaced on the display image 22 by the correction is represented on the image plane.

If the displacement vector for removing the distortion caused by the camera lens is (Δx, Δy) and the displacement vector for adding the distortion for the eyepiece is (-Δx', −Δy'), the displacement vector maps will be displayed respectively. The displacement vector held at the position is (Δx−Δx', Δy−Δy'). Note that the displacement vector only defines the direction of pixel displacement and the amount of displacement, so if those parameters can be determined in advance, various corrections and combinations are possible, not limited to corrections caused by lens distortion. It can be easily realized with the same configuration.

For example, correction of scaling the captured images 16a and 16b to match the size of the display panel 122 and correction of chromatic aberration based on the color arrangement of the light emitting elements on the display panel 122 may be included in the elements of the displacement vector. In this case as well, the final displacement vector map can be generated by obtaining the displacement vectors in each correction for the position of the image plane and summing them. As a result, a plurality of corrections can be applied in one process. When the display image 22 is generated, the pixels at the positions of the captured images 16a and 16b are moved by the displacement vector with reference to the displacement vector map.

Although the captured images 16a and 16b and the displayed image 22 are displaced by the amount of distortion, there is no significant change in the position or shape in which the image appears. Therefore, the pixel values of the captured images are acquired in order from the line above the image plane. In parallel with, the pixel value can be acquired and corrected. Then, by outputting to the display panel 122 in parallel with the correction process in order from the upper row, it is possible to realize a display with low delay.

However, instead of the displacement vector map described above, a conversion formula for deriving the positional relationship of the corresponding pixels in the distorted image and the captured image may be set. Further, the factor that determines the pixel value of the display image is not limited to the displacement of the pixel depending on the presence or absence of distortion. For example, the pixel value is determined by appropriately combining the following parameters.
1. 1. The user's posture and direction based on the output value of the motion sensor (not shown) and the calculation result of SLAM. User-specific distance between the left and right pupils (distance between eyes)
3. 3. A parameter determined as a result of adjusting the mounting mechanism portion 104 (mounting band 106) of the head-mounted display 100 based on the relationship between the user's head and eyes.

The interpupillary distance of 2 above is obtained as follows. That is, when the head-mounted display 100 has a built-in stereo camera for tracking the line of sight, the pupil of the user wearing the head-mounted display 100 is photographed by the stereo camera for tracking the line of sight. Alternatively, the user points the stereo camera 110 or the like provided on the front surface of the head-mounted display 100 toward his / her face to take a picture of the face with open eyes. Alternatively, a camera (not shown) outside the content processing system is aimed at the user to take a picture of a face with open eyes. The image captured in this way is processed by the pupillary image recognition software that runs on the content processing system, and the interpupillary distance is automatically measured and recorded.

When the distance between the cameras of the stereo camera for tracking the line of sight or the stereo camera 110 is used, triangulation is performed. Alternatively, the captured image is displayed on the flat plate display 302 by the content processing system, and the user specifies the positions of the left and right pupils, so that the content processing device 200 calculates the distance between the left and right pupils based on the designation. Record. The user may directly register his interpupillary distance. The interpupillary distance acquired in this way is reflected in the distance between the left eye image and the right eye image in the display image 22 of FIG.

Regarding the above 3, a measuring instrument such as a rotary encoder or a rotary volume (not shown) built in the head-mounted display 100 acquires the mechanical adjustment result of the mounting mechanism unit 104 and the mounting band 106. The content processing system calculates the distance and angle from the eyepiece to the eye based on the adjustment result. The parameters acquired in this way are reflected in the enlargement ratio of the image and the position of the image in the display image 22 of FIG.

The above 1 to 3 are parameters specific to the user who wears the head-mounted display 100, or parameters which change arbitrarily such as the position and posture of the user, and it is difficult to reflect them in the map in advance. Therefore, the final pixel value may be determined by combining the transformation performed with reference to the displacement vector map and the transformation based on at least one of the above 1 to 3 parameters. Alternatively, the displacement vector map may be dynamically generated according to those parameters.

FIG. 5 shows the circuit configuration of the image processing integrated circuit 120 of the present embodiment. However, only the configuration according to this embodiment is shown, and the others are omitted. The image processing integrated circuit 120 includes an input / output interface 30, a CPU 32, an image signal processing circuit 42, an image calculation circuit 34, an image analysis circuit 54, an image composition circuit 56, and a display controller 44.

The input / output interface 30 establishes communication with the content processing device 200 by wired or wireless communication, and realizes data transmission / reception. The CPU 32 is a main processor that processes and outputs signals such as image signals and sensor signals, commands and data, and controls other circuits. The image signal processing circuit 42 acquires captured image data from the left and right image sensors of the stereo camera 110, and performs necessary processing such as demosaic processing on each of them. However, the lens distortion correction is not performed, and the data is stored in the buffer memory 38 described later in the order of the pixel strings in which the pixel values are determined. The image signal processing circuit 42 is synonymous with an ISP (Image Signal Processor).

The image calculation circuit 34 performs super-resolution processing for improving the definition of the image generated by the content processing device 200, image deformation processing, and other processing for editing an image (not shown) in cooperation with the CPU. In the image transformation processing, the amount of movement of the user's line of sight during the correction processing, enlargement / reduction, and the time required to transfer the image from the content processing device 200 or the stereo camera 110 to the image processing integrated circuit 120 shown in FIG. Dynamically generate a displacement vector map based on the orientation. Then, using it, the image is corrected and deformed according to the line of sight of the user.

Specifically, the image calculation circuit 34 includes a correction circuit 36, a buffer memory 38, a displacement vector map memory 40, and a super-resolution circuit 52. The first correction unit 46 of the correction circuit 36 corrects the captured image to generate a distorted display image for the eyepiece. The second correction unit 48 corrects an image formed by synthesizing an image transmitted from the content processing device 200 and a captured image to generate a display image. When the image transmitted from the content processing device 200 does not have distortion for the eyepiece in advance, the third correction unit 50 corrects the image and generates a display image having distortion for the eyepiece. To do.

The buffer memory 38 temporarily stores the image data before correction in the first correction unit 46, the second correction unit 48, and the third correction unit 50. The displacement vector map memory 40 stores the displacement vector map. The buffer memory 38 and the displacement vector map memory 40 may be integrally configured with the main memory. The super-resolution circuit 52 performs super-resolution processing for improving the definition of the captured image and the image transmitted from the content processing device 200 by a predetermined method.

As described above, the correction circuit 36 corrects the image by displacing each pixel in the captured image by the displacement vector. The target for setting the displacement vector in the displacement vector map may be all pixels in the captured image plane, or may be only discrete pixels at predetermined intervals.

In the latter case, the correction circuit 36 first obtains the displacement destinations of the pixels for which the displacement vector is set, and obtains the displacement destinations of the remaining pixels by interpolation based on the positional relationship with those pixels. When correcting chromatic aberration, the displacement vectors are different for each of the primary colors of red, green, and blue, so three displacement vector maps are prepared. Further, the correction circuit 36 determines the pixel value by appropriately interpolating the surrounding pixel values for the pixel whose value is not determined by the displacement of the pixel in the display image.

The correction circuit 36 refers to a UI (User Interface) plain image (also referred to as an OSD (On Screen Display) plain image) separately stored in the buffer memory 38, and synthesizes (superimposes) the captured image. May be good. The composition is performed between the UI plane image corrected by the displacement vector map and the captured image corrected by the displacement vector map. As the UI plane image, the image corrected by the displacement vector map may be stored in advance in the buffer memory 38, or the displacement vector map for the UI plane image and the distortion-free UI plane image may be stored in advance. The correction of the UI plane image referring to them may be performed in parallel with the correction of the captured image.

The correction circuit 36 outputs the pixel values thus determined in order from the upper line. When the correction circuit 36 outputs data to the display controller 44, a handshake controller (not shown) or the like is actually used to appropriately control communication between the two. That is, the correction circuit may include a handshake controller (not shown). In the handshake controller, the position where the image signal processing circuit 42 writes data to the buffer memory 38 and the amount of pixels stored in the buffer memory 38 are the pixel values of one line of the displayed image in the captured image. It constantly monitors that the amount required to determine is met and where the correction circuit 36 is reading data, resulting in data depletion, or buffer underrun, or data overflow, or buffer overrun. To prevent that.

If a buffer underrun or a buffer overrun has occurred, the CPU 32 is notified. The CPU 32 notifies the user of the occurrence of an abnormality and restarts the transfer. The display controller 44 sequentially converts the transmitted data into an electric signal, and drives the pixels of the display panel 122 at an appropriate timing to display an image.

The image analysis circuit 54 cooperates with the CPU 32 to analyze the captured image and acquire predetermined information. The analysis result is transmitted to the content processing device 200 via the input / output interface 30. The image synthesizing circuit 56 cooperates with the CPU 32 to synthesize an image transmitted from the content processing device 200 with the captured image corrected by the first correction unit 46. The combined image is stored in the buffer memory 38 and corrected by the second correction unit 48.

FIG. 6 is a diagram for explaining a procedure of display processing in the see-through mode of the present embodiment. First, the image signal processing circuit 42 processes the captured image 90 input from the image sensor in order from the top line and stores it in the buffer memory 38. Then, the correction circuit 36 generates a distorted display image 92 as described with reference to FIG. Here, the correction circuit 36 starts generating the display image 92 without waiting for all the captured images 90 for one frame to be stored in the buffer memory 38.

If the drawing of the line is started when the data of the number of pixels required to determine the pixel value for one line of the display image 92 of the captured image 90 is stored in the buffer memory 38, the line is displayed. Latency can be further suppressed. For example, when the pixel value of a certain row 94 of the display image 92 is determined at a certain timing, the corresponding row of the display panel 122 is driven by an electric signal based on the pixel value. After that, the entire display image 92 is displayed by repeating the same process toward the lower side of the image. The above is the case of the see-through mode, but the delay time of each can be suppressed by performing the same processing in the correction of each stage performed by the correction circuit 36.

FIG. 7 is a diagram for explaining the significance of the present embodiment in the time until the distortion-free image is processed and displayed. The horizontal direction of the figure represents the passage of time, and the drawing time of the display image by the correction circuit 36 or the like is indicated by a solid line arrow, and the output time to the display panel 122 is indicated by a broken line arrow. Further, in the description in parentheses described together with "drawing" and "output", the processing for one frame of the frame number m is (m), and the processing of the nth line of the frame number m is (m / n). (A) shows a mode in which a captured image for one frame is input and then output to the display panel as a comparison.

Specifically, from time t0 to time t1, the first frame is drawn and the data is stored in the main memory or the like. At time t1, drawing of the second frame is started, and the first frame is sequentially read from the main memory and output to the display panel 122. These processes are completed at time t2, and subsequently the third frame is drawn and the second frame is output. After that, each frame is drawn and output in the same cycle. In this case, the time required from the start of drawing the display image for one frame to the completion of output is equal to the output cycle for two frames.

According to the present embodiment shown in (b), the data is output to the display panel 122 when the drawing of the data in the first line of the first frame is completed. During that time, since the data on the second line is drawn, the data on the second line can be output to the display panel 122 following the data on the first line. By repeating this, at the time t1 when the drawing of the last (nth line) data is completed, the output of the data immediately before (n-1th line) is completed. Similarly, in the subsequent frames, the output to the display panel 122 is advanced in parallel with the drawing process.

As a result, the time required from the start of drawing the display image for one frame to the completion of output is a value obtained by adding the output time of one line of data to the output cycle for one frame. That is, as compared with the aspect (a), the required time is shortened by Δt, which is close to the output cycle for one frame. As a result, the image can be displayed with extremely low delay.

FIG. 8 is a diagram for explaining an example of a processing procedure in which the correction circuit 36 corrects a captured image. (A) shows a photographed image, and (b) shows a plane of a display image. S00, S01, S02 ... In the captured image plane represent the positions where the displacement vector is set in the displacement vector map. For example, the displacement vectors are set discretely (for example, every 8 pixels or 16 pixels at equal intervals) in the horizontal direction and the vertical direction of the captured image plane. D00, D01, D02, ... In the display image plane represent the positions of the displacement destinations of S00, S01, S02, ..., Respectively. In the figure, as an example, the displacement vector (Δx, Δy) from S00 to D00 is indicated by a white arrow.

The correction circuit 36 maps the captured image to the display image in the smallest triangular unit having the pixel for which the displacement vector is set as the apex. For example, a triangle having vertices S00, S01, and S10 of the captured image is mapped to a triangle having vertices D00, D01, and D10 of the displayed image. Here, the pixels inside the triangle are displaced linearly according to the distances from D00, D01, and D10, or to the positions interpolated by bilinear, trilinear, or the like. Then, the correction circuit 36 determines the pixel value of the display image by reading the value of the corresponding pixel of the captured image before correction stored in the buffer memory 38. At this time, the pixel values of the display image are derived by interpolating the values of a plurality of pixels within a predetermined range from the position of the read destination on the captured image by bilinear, trilinear, or the like.

As a result, the correction circuit 36 can draw the display images in raster order in units of triangles that are displacement destinations of the triangles in the captured image. Similarly, when adjusting the resolution, the pixels may be mapped for each of the smallest triangles. When correcting chromatic aberration, the position and shape of the displacement destination triangle change by a small amount by using a displacement vector map that is different for each primary color. FIG. 9 is a diagram for explaining the capacity of the buffer memory required for the correction process in the present embodiment. The figure shows the case where the corrected image is circular as the case where the correction is most necessary.

Let h be the vertical size of the image before correction and r (= h / 2) be the radius of the image after correction. The largest distance displaced by the correction is the pixels at the four corners of the image before correction. For example, the pixel at the upper left position S00 is displaced in the radial direction of the lens by the correction and appears at the position D00 on the circumference of the corrected image. Therefore, it is necessary to retain the data of the pixel at S00 until the pixel at position D00 is drawn. The vertical distance w = r-r / 2 ^1/2 from S00 to D00 is approximately 15% of the size h of the image before correction.

For example, in the case of a captured image having 2160 pixels in the vertical direction, the buffer memory 38 needs an area for storing data for 324 lines, which is 15% of the captured image. The time required from the shooting of S00 to the output of D00 is proportional to the distance w. For example, if the frame rate is 120 fps, the delay time from shooting to output is 1.25 msec. However, these values are only maximum values, and generally smaller capacity and delay time are required. Further, the processing delay time can be significantly reduced as compared with the route shown by the arrow B in FIG. In addition, the buffer memory 38 also requires an area for correction processing and an area for adding pixels when increasing the resolution.

In any case, in the present embodiment, the data for one frame of the captured image is sequentially corrected and output to the display panel 122 before being acquired, so that the display can be performed with an extremely small delay time. In addition, since the required memory capacity can be significantly smaller than the data size for one frame, a small-capacity buffer memory such as SRAM (Static Random Access Memory) can be installed at a position close to the correction circuit 36, which enables data transmission. It is possible to reduce the time and power consumption for this.

Next, when synthesizing the captured image and the image transmitted from the content processing device 200, a method of realizing the processing and synthesizing of the captured image in the integrated circuit 120 for image processing will be described. FIG. 10 shows the configuration of the functional block of the image processing device 128 built in the head-mounted display together with the content processing device 200. The functional block shown in the figure can be realized by the configuration of the integrated circuit 120 for image processing shown in FIG. 5 in terms of hardware, and the data input function loaded from a recording medium or the like into the main memory or the like in terms of software. It is realized by a program that exerts various functions such as data retention function, image processing function, and communication function.

Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various ways by hardware only, software only, or a combination thereof, and is not limited to any of them. The configuration of the entire system may be the same as that shown in FIG. In this example, the image processing device 128 includes a signal processing unit 150, an image analysis unit 152, a super-resolution processing unit 154, 158, a first correction unit 156, a third correction unit 159, a synthesis unit 160, and a second correction unit 162. An image display control unit 164 is provided.

The signal processing unit 150 is realized by the image signal processing circuit 42 of FIG. 5, acquires captured image data from the image sensor of the stereo camera 110, and performs necessary processing. The image analysis unit 152 is realized by the CPU 32, the image analysis circuit 54, and the input / output interface 30 of FIG. 5, analyzes the captured image, acquires predetermined information, and transmits it to the content processing device 200. For example, the distance between the subjects is obtained by stereo matching using the left and right captured images, and a depth map is generated in which the distance is expressed as a pixel value on the image plane. The position and orientation of the head-mounted display may be acquired by SLAM.

In addition, it is understood by those skilled in the art that various things can be considered as the content of image analysis. However, depending on the case, the captured image data itself processed by the signal processing unit 150 may be transmitted to the content processing device 200. In this case, the signal processing unit 150 includes the input / output interface 30 of FIG. Further, the analysis result by the image analysis unit 152 and the measurement value of the motion sensor (not shown) built in the head-mounted display 100 are also used for the above-mentioned image deformation processing. That is, the movement of the user's line of sight during the time required for processing inside the head-mounted display 100 and data transfer with the content processing device 200 is specified based on these parameters and dynamically reflected in the displacement vector map.

The super-resolution processing unit 154 is realized by the CPU 32 and the super-resolution circuit 52 of FIG. 5, and performs super-resolution processing for improving the definition of the captured image by a predetermined method. For example, the captured image is sharpened by synthesizing an image shifted in the horizontal and vertical directions of the image plane by a width smaller than one pixel and an image before shifting. In addition, various methods have been proposed for super-resolution, and any of them may be adopted.

The super-resolution processing unit 158 performs the same super-resolution processing as the super-resolution processing unit 154 on the image transmitted from the content processing device 200. Here, the image transmitted from the content processing device 200 may be an image to be combined with the captured image or an image to be displayed without being combined. The content processing device 200 draws an image using a depth map or the like transmitted from the head-mounted display 100. As a result, it is possible to draw an image according to the relationship between the image pickup surface of the stereo camera 110 and the position and posture of the subject, and eventually the relationship between the user's face and the position and orientation of the subject.

When drawing an image to be combined with a captured image, for example, a virtual object is drawn at an appropriate position and orientation, and an image in which other areas are filled with a predetermined color is generated. As a result, the virtual object can be superimposed and displayed at an appropriate position of the captured image by a general chroma key technique. Alternatively, an α value indicating transparency may be used to simultaneously generate an α image with α = 0 in a region other than the virtual object. In this case, the virtual object can be superimposed and displayed at an appropriate position by the alpha blending process with the captured image.

The first correction unit 156 is realized by the CPU 32 and the first correction unit 46 of FIG. 5, and corrects the captured image as shown in S14 of FIG. 4 to generate a display image having distortion for the eyepiece. However, when synthesizing the images transmitted from the content processing device 200, the first correction unit 156 does not perform chromatic aberration correction. That is, the same distortion is given to all primary colors. Considering the characteristics of the human eye looking at the display panel, all the red, green, and blue images are corrected using the displacement vector map generated for green. In addition, when the RAW image acquired by the image sensor is a Bayer array, green having the highest pixel density can be used.

In the case of the see-through mode in which the captured image is displayed without being combined with another image, the first correction unit 156 may generate a display image in which the chromatic aberration is corrected at once as described above. That is, the captured images of each color are corrected by using the displacement vector maps prepared for each of red, green, and blue. The third correction unit 159 is realized by the CPU 32 and the third correction unit 50 of FIG. 5, and if the image transmitted from the content processing device 200 does not have distortion for the eyepiece in advance, the third correction unit 159 is the third correction unit 159. 1 The same correction process as that of the correction unit 156 is performed. The third correction unit 159 may correct the image subjected to the super-resolution processing by the super-resolution processing unit 158, or the super-resolution processing unit 158 may superimpose the image corrected by the third correction unit 159. Resolution processing may be carried out.

The synthesizing unit 160 is realized by the CPU 32 and the image synthesizing circuit 56 of FIG. 5, and synthesizes an image transmitted from the content processing device 200 and subjected to super-resolution processing to the captured image corrected by the first correction unit 156. As described above, in the case of an image premised on chroma key composition, the pixels of the surrounding area filled with a predetermined color are replaced with the pixels of the captured image. When α images are transmitted at the same time, they are combined by a general alpha blend process using the α value of each pixel.

When an image to be displayed without compositing is transmitted from the content processing device 200, the compositing unit 160 may supply the transmitted image data to the second correction unit 162 as it is. Here, the "image to be displayed without compositing" is an image that is not combined with the captured image or an image that has been combined with the captured image in the content processing device 200. The second correction unit 162 is realized by the CPU 32 and the second correction unit 48 of FIG. 5, and corrects the image input from the composition unit 160. However, the second correction unit 162 performs only the correction that has not been performed, specifically the correction of chromatic aberration, among the corrections to be made for the displayed image.

In the prior art, when a display image that has been composited by the content processing device 200 is transmitted to the head-mounted display 100 for display, a composite image without distortion is generated, and then correction for an eyepiece and chromatic aberration correction are performed. Is common. On the other hand, in the present embodiment, since the data paths of the captured image and the image to be combined with the captured image are different, the correction process is separated into two stages.

That is, the image transmitted from the content processing device 200 and the captured image are given a common distortion corresponding to the eyepiece, and are corrected for each color after composition. When the first correction unit 156 corrects the image of all primary colors using the green displacement vector map for the captured image, the third correction unit 159 applies the same green color to the image transmitted from the content processing device 200. Make corrections.

Then, the second correction unit 162 further performs necessary corrections on the red and blue images of the combined image to complete the image. Color bleeding is performed by first correcting green, which is the wavelength band with the highest human visibility, and then scaling, super-resolution, and compositing, and then correcting red and blue aberrations. And contour abnormalities are less visible. However, the purpose is not to limit the order of colors used for correction.

When the content processing device 200 transmits an image to be displayed without compositing, the second correction unit 162 does not perform the correction processing if all the corrections necessary for the display are applied in the content processing device 200. Good. However, if there is an unimplemented correction, the second correction unit 162 may carry out the final correction. Further, in the case of the see-through mode, the synthesis process of the synthesis unit 160 and the correction process of the second correction unit 162 are omitted by performing the necessary correction in the first correction unit 156 as described above.

The CPU 32 switches the processing to be performed by each processing circuit depending on whether it is a see-through mode, an image to be combined with the captured image, or an image not to be combined with the captured image. As a result, the data path of the captured image can be completed in the head-mounted display 100 regardless of whether or not it is in the see-through mode. Further, by leaving the correction of chromatic aberration after synthesis, the boundary line of synthesis can be clearly defined.

That is, when the image is combined after the chromatic aberration is corrected, the boundary line set in the chroma key image or the α image contains an error depending on the primary color, and the contour after the composition causes color bleeding. A display image without bleeding on the contour can be generated by shifting the pixels by a small amount by chromatic aberration correction after synthesizing without color shift. Filter processing such as bilinear or trilinear is generally used in processing such as image scaling, super-resolution, and composition. If these filtering processes are performed after the chromatic aberration is corrected, the result of the chromatic aberration correction is destroyed at a micro level, and color bleeding and abnormal contours occur at the time of display. By setting the processing of the second correction unit 162 immediately before the display, such a problem can be avoided. The image display control unit 164 is realized by the display controller 44 of FIG. 5, and the display images thus generated are sequentially output to the display panel 122.

All the processing of the super-resolution processing unit 154, 158, the first correction unit 156, the third correction unit 159, the composition unit 160, the second correction unit 162, and the image display control unit 164 acquires the data of the entire area of the image frame. It can be done without waiting for it to be done. Therefore, as described above for the see-through mode, it is possible to display with a low delay time by sending the pixel values to the subsequent processing immediately after the pixel values are determined in order from the line on the image plane.

FIG. 11 is a diagram for explaining an example of an element included in the displacement vector as chromatic aberration correction. As shown in the figure, the display panel 122 forms pixels by a combination of red (R), green (G), and blue (B) light emitting elements. In the figure, one pixel 70 is enlarged and shown. However, the arrangement of the light emitting elements varies depending on the display panel. The pixel value represented by the data of the display image is a brightness value of red, green, and blue given to the entire region of the pixel 70, but strictly speaking, it represents the color of the image at the center 72 of the pixel region.

However, in the case of the illustrated arrangement, the brightness of red is originally determined by the color of the image at the position 74 shifted to the left by a minute amount from the center 72 of the pixel region. Therefore, by shifting the image of the red component in the display image to the right by a small amount, the value of the pixel on the left side is also reflected in the brightness of red. Similarly, the brightness of blue is originally determined by the color of the image at the position 76 shifted to the right by a small amount from the center 72 of the pixel region. Therefore, by shifting the image of the blue component of the displayed image to the left by a small amount, the value of the pixel on the right side is also reflected in the brightness of blue.

As a result, the position on the image plane and the color information appearing there can be accurately represented in sub-pixel units. Since the color arrangement of the light emitting elements constituting the pixels varies depending on the display panel as described above, the displacement vector is calculated based on the arrangement. In addition to the correction of chromatic aberration, in the correction for lens distortion using Equation 1, the difference in displacement when the distortion coefficient of the eyepiece is different for each color is included. That is, depending on the difference in the refractive index depending on the wavelength of light, axial chromatic aberration and chromatic aberration of magnification with respect to the lens occur, causing color shift in the image. The displacement vector includes a component for correcting this color shift.

The eyepiece included in the head-mounted display 100 may be a Fresnel lens in addition to a general convex lens. While the Fresnel lens can be made thinner, image distortion is more likely to occur as the resolution decreases and concentrically toward the periphery of the field of view, and the brightness can change non-linearly. This non-linear concentric luminance change can have different characteristics for red, green, and blue (eg, "distortion", Edmond Optics Technical Documentation, [online], Internet URL: https://www.edmundoptics. See .jp / resources / application-notes / imaging / distortion /). Therefore, the displacement vector may include a component that corrects this for each color.

When a liquid crystal panel is used as the display panel 122, the resolution can be increased, but the reaction speed is slow. When an organic EL panel is used, the reaction speed is high, but it is difficult to increase the resolution, and a phenomenon called Black Smearing, in which color bleeding occurs in and around the black region, may occur. In addition to the lens distortion described above, the correction circuit 36 may perform correction so as to eliminate such various adverse effects of the eyepiece and the display panel. In this case, the correction circuit 36 internally retains the characteristics of the display panel 122 as well as the characteristics of the eyepiece. For example, in the case of a liquid crystal panel, the correction circuit 36 resets the liquid crystal by inserting a black image between the frames to improve the reaction speed. Further, in the case of the organic EL panel, the correction circuit 36 offsets the luminance value and the gamma value in the gamma correction to make the color bleeding due to Black Smearing less noticeable.

FIG. 12 schematically shows the data stored in the displacement vector map memory 40. The displacement vector map memory 40a shown in (a) stores the displacement vector map 80 for red, green, and blue. The displacement vector map 80 represents the displacement of pixels from a captured image to a display image (or images in the regions to the left and right of the captured image). The first correction unit 156 refers to the displacement vector maps 80 in the see-through mode, corrects the images of the red, green, and blue components of the captured image, respectively, and generates a display image.

Further, in the displacement vector map memory 40a, a difference vector for red obtained by subtracting the green displacement vector from the red displacement vector and a difference vector for blue obtained by subtracting the green displacement vector from the blue displacement vector are placed on the image plane. The represented difference vector map 82 is stored. When synthesizing the images from the content processing device 200, the first correction unit 156 and the third correction unit 159 refer to the green displacement vector map of the displacement vector map 80 to display the red, green, and blue images of the captured image. Correct the image of the component.

Then, the second correction unit 162 generates a final display image by correcting the images of the red and blue components of the combined image with reference to the difference vector maps for red and blue, respectively. However, since the chromatic aberration correction requires that the red, green, and blue images are relatively displaced by an appropriate amount, the color of the displacement vector map referred to when the first correction unit 156 corrects is not limited. Then, the difference vector map may be generated for two colors other than the relevant color.

The displacement vector map memory 40b shown in (b) is different from (a) in that only the green displacement vector map 84 is stored as the displacement vector map. In this case, in the see-through mode, the first correction unit 156 first determines a part of the displacement vector map 84 for green and the corresponding difference vector maps 86 for red and blue, which are necessary for correcting the pixels to be processed. A part of the displacement vector map 80 in which red, green, and blue are aligned is dynamically generated with reference to a part of. Then, the first correction unit 156 corrects the image based on the dynamically generated displacement vector value.

Alternatively, in the see-through mode, the first correction unit 156 first corrects the image of the red, green, and blue components of the captured image with reference to the displacement vector map 84. Then, the first correction unit 156 generates a final display image by correcting the images of the red and blue components of the corrected image with reference to the difference vector map 86 for red and blue, respectively. To do. However, even in this case, the color of each map is not limited. In (b), the amount of data can be reduced as compared with the configuration of (a), and the memory capacity can be saved.

Next, the operation of the head-mounted display that can be realized by the above configuration will be described. FIG. 13 is a flowchart showing a processing procedure when the head-mounted display displays a captured image or the like. Although the series of procedures are shown in series in the figure, the next process is actually started without waiting for the end of the process for one frame in the previous stage as described above. As a result, each process is performed in parallel for different pixel sequences.

First, the head-mounted display 100 acquires an image that is the source of the displayed image (S20). Here, the original image includes at least one of an image captured by the stereo camera 110 and an image transmitted from the content processing device 200. The former is an image obtained by demosaicing a RAW image, and is in a state before correcting the lens distortion of the camera. The latter may be a distorted image for the eyepiece of the head-mounted display 100 or an undistorted image.

Next, the super-resolution processing units 154 and 158 appropriately perform super-resolution processing on the original image (S22). However, if super-resolution processing is not necessary, the processing may be omitted. In the case of the see-through mode (Y in S24), the first correction unit 156 uses the displacement vector maps created for red, green, and blue to correct the images of the red, green, and blue components of the captured image, respectively. (S26). Then, the first correction unit 156 outputs data to the display panel 122 in order from the line on the image plane together with the determination of the pixel value (S36).

As a result, the captured image is immediately displayed in an appropriate format. On the other hand, when the mode is other than the see-through mode (N in S24) and the image transmitted from the content processing device 200 and the captured image are combined (Y in S28), the first correction unit 156 determines the red, green, and blue of the captured image. The image of the component is corrected using the displacement vector map created for green (S30). Further, the third correction unit 159 also performs the same correction on the image transmitted from the content processing device 200, if necessary. Then, the compositing unit 160 synthesizes the captured image and the image transmitted from the content processing device 200 (S32).

Subsequently, the second correction unit 162 corrects the images of the red and blue components of the composite image, respectively, using the difference vector maps for red and blue (S34). When the captured image is not combined with the image transmitted from the content processing device 200, the processing of S30 to S34 can be omitted if the transmitted image is corrected as necessary (N in S28). However, when a common distortion is given to all the color components, the image of the red component and the image of the blue component are corrected by using the difference vector map in S34. Then, the second correction unit 162 outputs data to the display panel 122 in order from the line on the image plane together with the determination of the pixel value (S36).

Although the description so far has focused on the processing inside the image processing device 128, the image processing device 128 decodes the data of the image compressed and encoded by the content processing device 200 such as a cloud server and stream-transferred. When decompressing, the process may be advanced in the same manner. That is, the content processing device 200 and the image processing device 128 may perform compression coding, decoding / decompression, and motion compensation for each unit region formed by dividing the frame plane.

Here, the unit area is divided in the horizontal direction for each predetermined number of lines, for example, one line or two lines of pixels, or divided in both vertical and horizontal directions such as 16 × 16 pixels and 64 × 64 pixels. It is a rectangular area formed by. Each of the content processing device 200 and the image processing device 128 starts compression coding processing and decoding / decompression processing each time data to be processed for a unit area is acquired, and outputs the processed data for each unit area. .. As a result, the delay time until display can be further shortened, including the data transmission time from the content processing device 200.

According to the present embodiment described above, in the head-mounted display provided with the camera, a path for processing and displaying the captured image in the head-mounted display is provided separately from the path for displaying the image transmitted from the content processing device. Provide. As a result, it is possible to easily display the captured image with low delay during the period when the image of the content is not displayed. As a result, even when the head-mounted display is attached, the surrounding situation can be confirmed as in the case where the head-mounted display is not attached, and convenience and safety can be improved.

Further, in the present embodiment, a displacement vector map representing pixel displacement due to necessary correction elements such as removal of distortion by a camera lens, addition of distortion for an eyepiece, adjustment of resolution, and correction of chromatic aberration is used. Make corrections all at once. Since the correction can be performed independently for each pixel, the process from shooting to display can be performed in parallel for each pixel column. As a result, in addition to shortening the route from the camera to the display panel, the time required for the correction process itself can be shortened. Further, as compared with the case where one frame of data is stored and then output, not only the memory capacity but also the power consumption for data transmission can be saved.

Even when the image transmitted from the content processing device is included in the display target, the processing for the captured image is completed in the head-mounted display. At this time, the correction process for the captured image is switched to two stages. Specifically, before synthesizing the image transmitted from the content processing apparatus, a common correction is performed for all the primary color components, and the chromatic aberration is corrected after synthesizing. By correcting the chromatic aberration immediately before the display, the image after the chromatic aberration is destroyed at the micro level by the filter processing used for scaling, super-resolution, composition, etc., and color bleeding and abnormal contours appear on the display. Can be prevented.

As a result, a high-quality composite image can be displayed without transmitting the captured image data to the content processing device. As a result, the same effect as described above can be obtained without affecting the display result. Furthermore, the mode in which the image from the content processing device is combined with the captured image and the mode in which the image is not combined can be easily switched with a minimum change.

The present invention has been described above based on the embodiments. Embodiments are examples, and it is understood by those skilled in the art that various modifications are possible for each of these components and combinations of each processing process, and that such modifications are also within the scope of the present invention. ..

30 I / O interface, 32 CPU, 34 Image calculation circuit, 36 Correction circuit, 38 Buffer memory, 40 Displacement vector map memory, 42 Image signal processing circuit, 44 Display controller, 46 First correction section, 48 Second correction section, 50 Third correction unit, 52 Super-resolution circuit, 54 Image analysis circuit, 56 Image synthesis circuit, 100 Head mount display, 110 Stereo camera, 120 Integrated circuit for image processing, 122 Display panel, 128 Image processing device, 150 Signal processing Unit, 152 Image analysis unit, 154 Super-resolution processing unit, 156 1st correction unit, 158 Super-resolution processing unit, 159 3rd correction unit, 160 Synthesis unit, 162 2nd correction unit, 164 Image display control unit, 200 Content processing device.

Claims (15)

A signal processing unit that acquires captured image data,
A correction unit that corrects the captured image to an image suitable for display,
A compositing unit that synthesizes a compositing image transmitted from a device that is not integrally provided with the captured image, and a compositing unit.
An image display control unit that displays the combined image on the display panel,
With
Among the processes for correcting an image suitable for the display, the correction unit performs a part of the correction for the image taken before composition by the composition unit, and performs the remaining correction for the image after composition. An image processing device characterized by. As a partial correction, the correction unit performs a correction that adds distortion to be given to the display image viewed through the eyepiece, which is common to the primary color components of the captured image.
The image processing apparatus according to claim 1, wherein as the remaining correction, the chromatic aberration of the primary color component is corrected. A displacement vector map showing the displacement vector representing the displacement amount and the displacement direction of the pixels from the captured image required to perform the partial correction on the image plane, and the remaining correction required to perform the correction. A displacement vector map storage unit for storing a difference vector map representing the displacement amount and the displacement direction of the pixels from the composited image on the image plane is further provided.
The image processing apparatus according to claim 1 or 2, wherein the correction unit refers to the displacement vector map in the correction before synthesis and refers to the difference vector map in the correction after synthesis. The correction unit performs the partial correction with reference to the displacement vector map for the green component of the primary colors of the captured image, and refers to the difference vector map for the red and blue components to perform the remaining correction. The image processing apparatus according to claim 3, wherein the correction is performed. When it is not necessary to combine images for compositing transmitted from the device that is not integrally provided,
The image processing apparatus according to any one of claims 1 to 4, wherein the correction unit switches to perform processing for correcting an image suitable for the display at one time. The image processing apparatus according to any one of claims 1 to 5, wherein the partial correction includes a correction for eliminating distortion caused by a lens of the image pickup device. The image processing apparatus according to any one of claims 1 to 6, further comprising a super-resolution processing unit that performs super-resolution processing on the captured image before correction and the composite image before composition. The image processing apparatus according to any one of claims 1 to 7, wherein the correction unit and the synthesis unit sequentially output data of the processed pixels together with determination of a pixel value. A buffer memory for storing pixel data generated by the correction by the correction unit in the order of generation is further provided.
The image display control unit is characterized in that every time data of a predetermined number of pixels smaller than the total number of pixels of the captured image is stored in the buffer memory, the data is controlled to be transmitted. 8. The image processing apparatus according to any one of 8. The correction unit is characterized in that when the data of the number of pixels required for determining the pixel value for one row of the corrected image is acquired, the generation of the data of the row is started. The image processing apparatus according to claim 9. From claim 1, the correction unit further corrects the captured image based on at least one of the posture and orientation of the user, the interpupillary distance of the user, and the distance between the display panel and the eyes of the user. 10. The image processing apparatus according to any one of 10. The image processing apparatus according to any one of claims 1 to 8.
An imaging device that supplies captured images to the signal processing unit,
With the display panel
A head-mounted display featuring a head-mounted display. An image analysis unit that analyzes the captured image and transmits the result to the device that is not integrally provided is further provided.
The head-mounted display according to claim 12, wherein the composite image is generated based on the result. The steps to acquire the data of the captured image and
A step of performing some corrections in the process of correcting the captured image to an image suitable for display, and
A step of synthesizing a composite image transmitted from a device that is not integrally provided with a captured image that has been partially corrected, and
A step of correcting the composited image to an image suitable for the display and performing the remaining correction to obtain a display image.
The step of displaying the display image on the display panel and
An image display method by an image processing apparatus, which comprises. The function to acquire the data of the captured image and
A function to correct the captured image to an image suitable for display, and
A function for synthesizing a composite image transmitted from a device that is not integrally provided with the captured image, and
A function to display the combined image on the display panel,
To the computer,
The correction function performs a part of the correction for the image suitable for the display, the image taken before the composition by the composition function, and the remaining correction for the image after the composition. A computer program characterized by doing.