JP2017188757A - Information processing apparatus, image display apparatus, image display system, and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus, an image display apparatus, an image display system, and an information processing method for suppressing the total amount of data required for transmission of image data as a whole system in the image display system.SOLUTION: The image display system acquires information on an imaging unit 10 and a display unit 13, corrects distortion of the imaging unit 10 and distortion of the display unit 13 on the basis of the information, and cuts out a part of the angle of view for the corrected image to display the cut-out image on the display unit 13.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for processing image data captured by an imaging unit provided in an image display device.

  In recent years, mixed reality (MR) technology and augmented reality (AR) technology are known as technologies that seamlessly merge real space and virtual space in real time. As one of these technologies, there is one using video see-through HMD (Head Mounted Display). In this image display system, first, a video camera or the like is imaged with a subject that substantially matches the subject observed from the pupil position of the HMD wearer. An HMD wearer can observe an image obtained by superimposing and displaying CG (Computer Graphics) on the captured image through the HMD internal panel.

  At this time, the image observed by the HMD wearer includes distortion characteristics generated in the imaging optical system and distortion characteristics generated in the display optical system. Patent Document 1 proposes a display device with an imaging function that performs distortion correction of an imaging system and distortion correction of a display system in the entire system including an external device in order to provide mixed reality to an HMD wearer. Yes.

JP 2008-5460 A

  In the case of a system that displays a part of the angle of view (region) of a captured image by the imaging optical system in accordance with the angle of view of the display optical system, it is necessary to cut out a part of the captured image. In the conventional system, magnification fluctuation occurs in the video input / output for each of the imaging system distortion correction process and the display system distortion correction process. Therefore, it is necessary to perform an image cutout process for adjusting the angle of view after both distortion corrections. However, it is necessary to transmit surplus pixel data in order to perform the image cutout process last. Therefore, an object of the present invention is to reduce the total amount of video data to be transmitted as a whole system.

  In order to solve the above-described problems, the present invention is an information processing apparatus connected to an image display device that includes an imaging unit that captures an image and a display unit that displays a display image to be displayed to a user. Based on at least one of the information related to the imaging optical system of the imaging unit and the information related to the display optical system of the display unit, the captured image of the captured image by at least one of the imaging optical system and the display optical system The image processing apparatus includes a correction unit that corrects distortion, and a cutout unit that cuts out a part of the corrected image.

  According to the above configuration, the present invention can suppress the total amount of video data to be transmitted as a whole system.

1 is a block diagram showing an image display system according to a first embodiment. The conceptual diagram which shows the imaging distortion and the display distortion in 1st Embodiment. The figure which shows an example of the information which concerns on the imaging distortion and imaging distortion in 1st Embodiment. The block diagram which shows the function structure of the image | video clipping part which concerns on 1st Embodiment. 5 is a flowchart showing a flow of processing by an angle-of-view change calculation unit according to the first embodiment. The figure which shows the change of the resolution and angle of view of video data in 1st Embodiment. The block diagram which shows the image display system which concerns on the modification of 1st Embodiment.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an image display system in the present embodiment. As shown in FIG. 1, the image display system of the present embodiment is roughly composed of four functional units, and performs image processing on an imaging unit 10 that generates a captured video, and the captured video and the display video. And an image processing unit 11. Furthermore, there is provided an image composition unit 12 that calculates CG data to be superimposed according to the content of the captured image and creates a display image (display image), and a display unit 13 that displays the display image and allows the user to visually recognize the image. doing.

  The imaging unit 10 and the display unit 13 are provided in a head mounted display (hereinafter referred to as HMD) corresponding to an image display device in the present system. On the other hand, the image processing unit 11 and the image composition unit 12 are provided in a PC corresponding to an information processing apparatus in this system. The HMD and the information processing apparatus (PC) are connected to each other by a wired method or a wireless method, and each has a hardware configuration such as a CPU, a ROM, and a RAM. Then, when the CPU executes a program stored in a ROM, an HD, or the like, for example, each functional configuration and flowchart processing described later are realized. The RAM has a storage area that functions as a work area where the CPU develops and executes the program. The ROM has a storage area for storing programs executed by the CPU.

  In the HMD including the imaging unit 10 and the display unit 13, a user wearing the HMD on the head can see an image in which CG is superimposed on an image of the outside world, and the mixed reality in which real space and virtual space are fused in real time. You can experience the feeling. The imaging unit 10 includes an imaging lens 100 that is an imaging optical system and an image sensor 101 that is an imaging element. A subject image in a real space imaged through the imaging lens 100 is captured by photoelectric conversion by the image sensor 101. To get. The image sensor 101 is composed of a semiconductor sensor such as a CMOS image sensor or a CCD image sensor, for example. FIG. 2 conceptually shows imaging distortion and display distortion, and the captured image created by the imaging unit 10 has a geometric shape with respect to the subject, for example, as shown in FIG. This causes geometric distortion and chromatic aberration.

  The display unit 13 includes a panel 130 and a display lens 131, displays the display image generated by the image composition unit 12 on the panel 130, and forms an image of the light flux on the observation target via the display lens 131. The panel 130 is comprised from display elements, such as a liquid crystal display panel and an organic electroluminescent panel, for example. In the image observed by the display unit 13, for example, geometric distortion and chromatic aberration are generated with respect to the display image as shown in FIG. 2B due to the optical characteristics of the display lens 131.

  Next, functions of the image processing unit 11 and the image composition unit 12 provided on the information processing apparatus side will be described. The image processing unit 11 includes a video cutout unit 110 that performs video cutout processing on the captured video input from the imaging unit 10 and an imaging optical characteristic holding unit 112 that holds imaging optical characteristics. Further, an imaging distortion correction unit 111 that performs distortion correction based on imaging optical characteristics, a display optical characteristic holding unit 114 that holds display optical characteristics, and a display image input from the image synthesis unit 12 based on display optical characteristics. And a display distortion correction unit 113 that performs distortion correction. Details of each functional unit will be described later.

  The image composition unit 12 has a function of superimposing a CG on the captured video input from the image processing unit 11 and outputting it again to the image processing unit 11 as a display video. Further, the image composition unit calculates CG and a position to be superimposed on the captured video based on information on the position and orientation of the HMD user estimated by a CG superimposed position estimation unit (not shown). Then, CG data is synthesized with the calculated image position and output as a display image. As a position and orientation estimation method by the CG superimposed position estimation means, for example, there is a technique using an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, GPS, and the like. In addition, an image sensor different from the image capturing unit 10 is mounted, and the position and orientation are estimated using markers in the captured image, and if there are spatial feature points, the position and orientation are estimated from the feature points. You can also

  Further, the image composition unit 12 includes a resolution conversion unit (not shown), and there are cases where the image conversion resolution is changed in accordance with the display angle of view, and resolution conversion processing is performed for matching with the image pickup angle of view. In the present embodiment, the resolution conversion process is performed by the image composition unit 12, but the present invention is not limited to such a structure. For example, the resolution conversion process may not be performed, or the resolution conversion process may be performed after the image composition unit 12.

  In this way, the captured video imaged by the imaging unit 10 is input to the image composition unit 12 via the image processing unit 11, and CG is added to the captured video based on information in the captured video, other sensor information, and the like. A superimposed display image is created. The created display image is input again to the display unit 13 via the image processing unit 11, and can provide the user with a mixed reality in which the real world and the virtual world are fused in real time. At this time, since a calculation process for estimating the CG superposition position is performed in accordance with the captured image, a predetermined performance may not be achieved depending on the capability of the device that operates the system (for example, a PC) and the resolution of the captured image.

<Details of Image Processing Unit 11>
Here, in the image display system of the present embodiment, details of processing of the imaging distortion correction unit 111 and the imaging optical characteristic holding unit 112 included in the image processing unit 11 will be described. The imaging distortion correction unit 111 performs distortion correction processing based on information on the imaging optical characteristics held in the imaging optical characteristic holding unit 112. The imaging distortion correction unit 111 and the imaging optical characteristic holding unit 112, for example, mount an LSI having an imaging distortion correction function on the same substrate and a FLASH ROM that holds the imaging optical characteristics, and read information on the imaging optical characteristics from the FLASH ROM. It is realized with. The application may have a configuration that realizes an imaging distortion correction function, and is not limited to a specific configuration as long as the configuration can correct imaging distortion.

  FIG. 3 shows an example of information about the imaging optical characteristics held in the imaging optical characteristic holding unit 112 and the display optical characteristics held in the display optical characteristic holding unit 114 in the present embodiment. a) is an example of imaging optical characteristic information. As shown in the figure, information of the imaging lens 100 and the image sensor 101 in the imaging unit 10 is held as the image information 300, and specifically, information on resolution, angle of view, optical axis center, and the like are applicable. . In addition, the imaging distortion table 301 holds local information on distortion generated by the imaging unit 10, and a magnification that indicates a reference position R indicating an imaging distortion correction amount at an arbitrary image position P, magnification change information with respect to the resolution, and the like. M and the like are included. In addition, when chromatic aberration occurs, it is possible to correct chromatic aberration by holding distortion information for each color together.

  The imaging distortion correction unit 111 reads the distortion correction table illustrated in FIG. 3A from the imaging optical characteristic holding unit 112, and executes distortion correction processing on the input captured video. The distortion correction processing is realized by, for example, calculating a distortion correction amount corresponding to the target pixel position from the reference position R at an arbitrary target image position, and generating a captured image at the calculated pixel position as an output. Can do. Various methods such as bilinear interpolation and spline interpolation have been proposed as the above-described distortion correction amount calculation method, and the present invention is not limited to a specific calculation method.

  As described above, the imaging distortion correction unit 111 reads the distortion correction table as illustrated in FIG. 3A and executes the distortion correction processing on the input captured video, so that the input captured video is output. The magnification is changed with respect to the captured image after the imaging distortion correction. The fact that the magnification fluctuates before and after the input / output is synonymous with the change of the imaging angle of view when there is no change in resolution at the input / output. That is, taking the imaging optical characteristic of FIG. 3A as an example, the horizontal maximum change magnification is -3.4%, so the horizontal angle of view is 49.39 ° × 0.966 = 47.71 °. In the present embodiment, for simplicity of explanation, the maximum distortion change magnification and the change rate of the angle of view are regarded as equivalent, but the calculation method uses the focal length, the pixel pitch of the image sensor, the mounting tolerance, and the like. May be.

  Next, details of processing of the display distortion correction unit 113 and the display optical characteristic holding unit 114 included in the image processing unit 11 in the image display system of the present embodiment will be described. The display distortion correction unit 113 reads information related to distortion correction held in the display optical characteristic holding unit 114, and performs distortion correction of video information displayed on the display unit 13, and its basic operation is imaging distortion. This is the same as the correction unit 111. FIG. 3B shows an example of display optical characteristic information held in the display optical characteristic holding unit 114.

  As described above, in the imaging distortion correction unit 111 and the display distortion correction unit 113, a change in magnification, that is, a change in angle of view occurs according to each optical characteristic. For this reason, it is desirable that the display image input to the display unit 13 is cut out immediately before being input to the display unit 13 to adjust the resolution and the angle of view. However, in a system having a conventional configuration in which the clipping process is performed after the display distortion correction unit 111, the resolution of the captured video input to the image composition unit 12 is high, and the process is performed in a state including overhead. As a result, performance was not optimal. Therefore, in the configuration of the present embodiment, as will be described below, the cut-out resolution optimal for the system is calculated by the angle-of-view calculation processing unit, and cut-out processing is executed on the captured video.

<Details of Video Clipping Unit 110>
Next, details of the processing of the video cutout unit 110 according to the present embodiment will be described. FIG. 4 is a schematic block diagram illustrating a functional configuration of the video cutout unit 110. The video cutout unit 110 includes an angle of view change calculation unit 1100 and a cutout execution unit 1101. The angle-of-view change calculation unit 1100 calculates the angle-of-view change of the input video in both distortion corrections based on the imaging optical characteristic information and the display optical characteristic information read from the imaging optical characteristic holding unit 112 and the display optical characteristic holding unit 114. Calculate the optimal clipping resolution for the system. In addition, the cutout execution unit 1101 executes cutout processing on the input captured video based on the cutout resolution calculated by the view angle change calculation unit 1100.

  FIG. 5 is a flowchart showing the flow of processing by the view angle change calculation unit 1100. Here, the description will be made using an example of information on imaging optical characteristics and display optical characteristics shown in FIGS. In the flowchart of FIG. 5, in step S <b> 501, the field angle change calculation unit 1100 reads the display resolution DR (1920 × 1080) and the display field angle DA (40 × 22.5 °) from the display optical characteristic holding unit 114. Further, in step S502, the view angle change calculation unit 1100 reads the display distortion correction table DT from the display optical characteristic holding unit 114.

  The view angle change calculation unit 1100 can calculate the input resolution PR and the input view angle PA of the display distortion correction unit 113 based on these pieces of information. For example, when it is assumed that DR = PR for simplification, as described above, a change in magnification due to distortion becomes a change in view angle. That is, according to the display distortion correction table of FIG. 3B, the maximum change magnification of the table is horizontal = 0.1% and vertical = 5.3%, so that the angle of view is DA = 0. It can be obtained as 999 × PA and the vertical direction is DA = 0.947 × PA. In step S503, in this way, from the display angle of view DA (40 × 22.5 °) and the display distortion correction table DT, the resolution PR before display distortion correction (1920 × 1080) and the angle of view PA before display distortion correction (40 × 23.7 °). ) Is determined.

  Next, in step S504, the angle-of-view change calculation unit 1100 reads the imaging resolution CR (1920 × 1080) and the imaging angle of view CA (49.4 × 29.4) from the imaging optical characteristic holding unit 112. In step S505, the angle-of-view change calculation unit 1100 reads the imaging distortion correction table CT from the imaging optical characteristic holding unit 112. In step S506, the view angle change calculation unit 1100 first calculates the input view angle QA to the imaging distortion correction unit 111 based on these pieces of information. As the calculation method, as in step S503, the imaging distortion correction table is referred to, and the calculation is performed based on the magnification change information due to the imaging distortion. In step S506, the pre-imaging angle of view QA (41.16 × 24.5 °) is determined in this way.

  When the view angle QA to be cut out is determined, in step S507, the view angle change calculation unit 1100 calculates a final cut-out resolution QR. Since the cut-out view angle QA (41.16 × 24.5 °) is determined, the cut-out resolution QR is determined from the image pickup resolution CR (1920 × 1080) and the image pickup view angle CA (49.4 × 29.4 °). That is, since the horizontal direction QRx = 1920 × 41.16 / 49.4 = 1600 and the vertical direction RQy = 1080 × 24.5 / 29.4 = 900, it is determined as QR (1600 × 900). Finally, in step S508, the angle-of-view change calculation unit 1100 transmits the cutout resolution QR (1600 × 900) to the cutout execution unit 1101.

  The cutout execution unit 1101 cuts out a part of the captured video based on the cutout resolution information input from the view angle change calculation unit 1100. The cutout position is cut out, for example, as a position where the center position of the optical axis for imaging is aligned before and after the cutout.

  FIG. 6 is a diagram showing the resolution and the angle of view of video data in a series of processes of the image processing system of the present embodiment. In the figure, as a comparison target, the resolution and the angle of view in the system of the conventional configuration described above are also shown. In FIG. 6, “A. Output from image sensor” indicates the resolution and angle of view of the captured image transferred from the image sensor 101 in FIG. 1 to the image cutout unit 110. “B. Output from imaging distortion pre-correction cut-out process” indicates the resolution and angle of view of the captured video transferred from the video cut-out unit 110 to the imaging distortion correction unit 111.

  Further, “C. Output from the imaging distortion correction unit” indicates the resolution and angle of view of the captured video transferred from the imaging distortion correction unit 111 to the image composition unit 12. “D. Output from the image composition unit” indicates the resolution and angle of view of the display image transferred from the image composition unit 12 to the display distortion correction unit 113. Here, the increase in resolution between C and D is due to the resolution conversion processing in the image composition unit 12 described above. In this process, an image that matches the display resolution is generated while maintaining the angle of view of imaging.

  “E. Output from display distortion correction unit” indicates the resolution and angle of view of the display video output from display distortion correction unit 113. “F. Output from cutout processing after display distortion correction” refers to the resolution and image of the display video output from the cutout processing unit arranged between the display distortion correction unit and the panel in the image display system having the conventional configuration. Shows corners. Finally, “G. Input to display panel” indicates the resolution and angle of view of the display video input to panel 130. In the system of the conventional configuration, there is no output of B, and in the system of the present embodiment, there is no output of F, so the respective resolution and angle of view are not described.

  Here, paying attention to “G. Input to display panel” in FIG. 6, there is no difference in resolution and angle of view between the conventional configuration and the present embodiment. On the other hand, in “C. Output from the imaging distortion correction unit”, the resolution in the conventional configuration is 1920 × 1080, whereas the resolution in the present embodiment is 1600 × 900, and the resolution of this embodiment is about 30%. Is low. In this way, without affecting the resolution and angle of view of the video to be displayed, the amount of video data to be transferred and processed in the processing prior to the input to the display unit is suppressed, and the input resolution to the image composition unit 12 is reduced. Can be optimized. Thereby, in this embodiment, it becomes possible to optimize the performance of the whole system.

  In the present embodiment, as described above, information of the imaging optical system and the display optical system is acquired, and both distortion caused by the imaging optical system and distortion of the display optical system are corrected from the information. It is also possible to pay attention to only one of the distortions and correct only one of them.

  As described above, according to the present embodiment, it is possible to suppress the input resolution to the image composition unit by calculating the clipping resolution according to the imaging optical characteristics and the display optical characteristics and performing the clipping process before the imaging distortion correction. . Therefore, the amount of video data to be transferred and processed is suppressed, the load on the device that operates the system is reduced, and the performance of the entire image display system can be optimized.

[Modification of First Embodiment]
Here, a modified example of the above-described first embodiment will be described. FIG. 7 is a block diagram showing an image display system according to this modification. This modification is different from the first embodiment described above in the internal configuration of the image processing unit 11. Specifically, in the present modification, the imaging distortion correction process and the display distortion correction process are processed in common by the common distortion correction unit 702 and a common distortion table calculation unit 701 is provided. The video cutout unit 700 is configured to determine the cutout resolution based on the input from the common distortion table calculation unit 701.

  The common distortion table calculation unit 701 newly generates a table based on the table information read from the imaging optical characteristic holding unit 112 and the display optical characteristic holding unit 114. Then, the video cutout unit 700 uses the newly generated table to determine the cutout resolution in the same manner as in the first embodiment described above. Thereby, it is possible to determine the cutout region including the imaging optical characteristic and the display optical characteristic. As described above, in this modification, it is possible to optimize the performance of the entire system even when a common distortion correction process is performed.

[Other Embodiments]
In the above description, the configuration in which the HMD side includes the imaging unit 10 and the display unit 13 and the information processing apparatus (PC) side includes the image processing unit 11 and the image composition unit 12 is illustrated. However, the present invention is not limited to such a form. For example, the HMD side has all of the above four functional units, and all processes are executed only on the HMD side. Good.

  In the above description, the example in which the imaging optical characteristic holding unit 112 and the display optical characteristic holding unit 114 are configured separately has been described. However, the imaging optical characteristic information and the display optical characteristic information are stored in a single holding unit. It may be configured to be held by. Further, such information may be held outside the image display system according to the above-described embodiment, and may be acquired and used.

  In addition, the present invention supplies software (program) for realizing the functions of the above-described embodiments to a system or apparatus via a network or various storage media, and the computer of the system or apparatus (or CPU, MPU, etc.) programs Is read and executed. Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. The present invention is not limited to the above embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these are excluded from the scope of the present invention. is not. That is, the present invention includes all the combinations of the above-described embodiments and modifications thereof.

DESCRIPTION OF SYMBOLS 10 Imaging part 11 Image processing part 12 Image composition part 13 Display part

Claims (10)

An information processing apparatus connected to an image display device comprising: an imaging unit that captures an image; and a display unit that displays a display image to be displayed to a user,
Distortion of the captured image by at least one of the imaging optical system and the display optical system based on at least one of information related to the imaging optical system of the imaging unit and information related to the display optical system of the display unit Correction means for correcting
Clipping means for cutting out a part of the corrected image;
An information processing apparatus comprising:   The correction means includes an imaging distortion correction means for correcting image distortion caused by the imaging optical system based on information relating to the imaging optical system, and an image distortion caused by the display optical system based on information relating to the display optical system. The information processing apparatus according to claim 1, further comprising display distortion correction means for correcting Based on the information related to the imaging optical system and the information related to the display optical system, it further includes a calculation means for calculating information related to the imaging optical system and information common to the display optical system,
The information processing apparatus according to claim 1, wherein the correction unit corrects image distortion based on information calculated by the calculation unit.   2. The information related to the imaging optical system and the information related to the display optical system are a resolution, a field angle, and a correction amount at a target pixel position in the imaging optical system and the display optical system. 4. The information processing apparatus according to any one of items 1 to 3.   The information processing apparatus according to claim 1, wherein an angle of view of the captured image is larger than an angle of view of a display image displayed on the display unit.   6. The information processing apparatus according to claim 1, further comprising a holding unit configured to hold information related to an imaging optical system of the imaging unit and information related to a display optical system of the display unit. .   The information processing apparatus according to claim 1, wherein the image display apparatus is used by being mounted on a user's head. An imaging unit that captures an image;
Based on at least one of the information related to the imaging optical system of the imaging unit and the information related to the display optical system of the display unit, distortion of the captured image by at least one of the imaging optical system and the display optical system is reduced. Correction means for correcting;
Clipping means for cutting out a part of the corrected image;
A display unit for displaying a display image to be displayed to the user based on the clipped image;
An image display device comprising: An imaging unit that captures an image;
Based on at least one of the information related to the imaging optical system of the imaging unit and the information related to the display optical system of the display unit, distortion of the captured image by at least one of the imaging optical system and the display optical system is reduced. Correction means for correcting;
Clipping means for cutting out a part of the corrected image;
A display unit for displaying a display image to be displayed to the user based on the clipped image;
An image display system comprising: An information processing method in an information processing apparatus connected to an image display apparatus comprising: an imaging unit that captures an image; and a display unit that displays a display image to be displayed to a user,
Distortion of the captured image by at least one of the imaging optical system and the display optical system based on at least one of information related to the imaging optical system of the imaging unit and information related to the display optical system of the display unit A step of correcting
Cutting out a portion of the corrected image;
An information processing method characterized by comprising:

Images