WO2017179912A1

WO2017179912A1 - Apparatus and method for three-dimensional information augmented video see-through display, and rectification apparatus

Info

Publication number: WO2017179912A1
Application number: PCT/KR2017/003973
Authority: WO
Inventors: 유범재; 이주성
Original assignee: 재단법인 실감교류인체감응솔루션연구단
Priority date: 2016-04-15
Filing date: 2017-04-12
Publication date: 2017-10-19

Abstract

An apparatus for a three-dimensional information augmented video see-through display according to one embodiment of the present invention comprises: a camera interface module for obtaining at least two real images from at least two camera modules; a rectification module for performing rectification for the at least two real images; a lens distortion compensation module for correcting at least two composite images in which a virtual image is combined to each of the at least two real images, on the basis of a lens distortion compensation value representing a value for compensating for the distortion of a wide-angle lens with respect to the at least two real images; and an image generation module for generating a three-dimensional image for a virtual reality (VR) or augmented reality (AR) by performing side-by-side image processing for the at least two composite images.

Description

3D information enhancement video see-through display device and method, rectification device

Embodiments of the present invention relate to a three-dimensional information augmented video see-through display apparatus and method and a rectification apparatus for the same.

Furthermore, the present invention claims the benefits of Korean Patent Application No. 10-2016-0046176, filed April 15, 2016 and Korean Patent Application No. 10-2017-0047277, filed April 12, 2017, all of the contents of Is included herein.

A head mounted display (HMD) device is an image display device that can be worn on a head, such as glasses, to view an image, and is a next generation image display device that is used for viewing a large screen and performing surgery or diagnosis while carrying. Conventional HMDs have been used for special purposes such as military and training purposes, but as HMDs become smaller and lighter due to the development of display technology, low-priced HMDs are commercially available. These low-cost HMDs have been used mainly for 3D movie watching until recently, but are actively researching and developing them as visualization devices for virtual reality and augmented reality by doubling the technical characteristics that can provide high immersion.

Video See-Through Head Mounted Display (HMD) is a device equipped with a camera in front to enable viewing of the real world through the HMD. The real world image (real image) acquired through the camera is converted into a digital signal and visualized by HMD. In this way, the user sees the real world in front as a digital image.

Video see-through HMDs have several benefits. Firstly, information that cannot be seen with the naked eye, such as infrared, ultraviolet, ultrasonic, and electromagnetic waves, can be visually identified using a special camera. In addition, by using the zoom-in function of the camera, a far object can be magnified from several times to several tens of times. In addition, when the fisheye lens is mounted on the camera, the viewing angle may be greatly widened to provide an omnidirectional view.

Second, the video see-through HMD can store the front view as seen by the user as a stereoscopic image. The saved video information can be shared with others through the network, and the user can view it again in the future. This storage of visual experiences allows users to play / share experiences beyond time and space constraints.

Third, video see-through HMDs provide the ability to protect the user's eyes from the risk of damaging vision, such as sunlight, lasers, dirt, and hazardous materials. By providing new information services using network means, including video see-through HMD, wireless video communications and the Internet, a visual experience sharing system that can be put to practical use in the near future will be possible. Accordingly, there is a need for a technology development of a smooth information service that enables sharing of visual experiences among these various benefits that can be obtained when using a video see-through HMD.

In this regard, there is a Korean Patent Laid-Open Publication No. 10-2015-0044488 entitled "Video O-Through Head Mounted Display Image Content Operation Method and System".

An object of the present invention is to generate a three-dimensional image for virtual reality or augmented reality in a video see-through method by processing the three-dimensional image in a module implemented by a hardware chip without the help of a PC from the plurality of camera modules. The present invention provides a three-dimensional information enhancement video see-through display device and method.

In addition, an object of the present invention is to reduce the storage space of the breakpoint lookup table by sequentially storing a plurality of breakpoint information for rectification, the empty space on the breakpoint lookup table.

Furthermore, an object of the present invention is to store the difference value between the next target coordinate and the current target coordinate, which are coordinates to be transformed, in advance in the current coordinate transformation information for rectification, in the breakpoint lookup table. This is to perform redirection by minimizing access.

The problem to be solved by the present invention is not limited to the problem (s) mentioned above, and other object (s) not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, the three-dimensional information augmented video see-through display device according to an embodiment of the present invention is a camera interface module for obtaining at least two real images from at least two camera modules, A rectification module that performs rectification, combining the virtual image into each of the at least two real images based on a lens distortion compensation value representing a value for compensating for the distortion of the wide-angle lens for the at least two real images Lens distortion compensation module correcting at least two composite images and side by side image processing on at least two composite images to generate a three-dimensional image for virtual reality (VR) or augmented reality (AR). It includes an image generation module for generating.

According to an embodiment of the present disclosure, the rectification module may be configured to perform coordinate transformation equal to each other among a plurality of pixels included in each of the at least two real images based on the current reference target break point information on a previously stored break point lookup table. A current coordinate transformation target pixel group consisting of at least one consecutive pixel required, each of which is determined, and each coordinate coordinate transformation target pixel group includes a current coordinate value including a current difference value that is a difference value from a current target coordinate that is a coordinate to be converted Based on the conversion information, coordinates of each pixel group of the current coordinate conversion target are converted.

For example, the pre-stored breakpoint lookup table may sequentially store a plurality of breakpoint information in an empty space on the memory, and at least one memory of at least one breakpoint information among the plurality of sequentially stored breakpoint information. More storage address information.

For example, the current coordinate transformation information may be a next difference that is a difference value with a next target coordinate that is a coordinate to be transformed, respectively, based on the next reference target breakpoint information on the previously stored breakpoint lookup table. The difference value between the value and the current difference value is stored in advance.

For example, the rectification module converts the coordinates of each of the next coordinate conversion target pixel groups based on the difference between the next difference value and the current difference value.

According to an embodiment, when the at least two camera modules are each fisheye lens cameras, a differential field, which is a field indicating a current difference value, includes at least four bits.

According to one embodiment, the three-dimensional information augmented video see-through display device improves image processing reliability for at least two camera modules for taking at least two real images, an infrared camera module for taking infrared images and at least two real images. The apparatus may further include a camera calibration module that performs camera calibration on at least two camera modules based on the infrared image.

For example, the three-dimensional information augmentation video see-through display device is a light detection module for detecting a light position within a certain radius of the current position based on the current position and the direction data for each light based on the current position based on the detected light position. The apparatus further includes a spatial generating module for generating a virtual image, wherein the virtual image is input from a computer that is interoperable with the three-dimensional information augmentation video see-through display device and combined with each of at least two real images. Gives an object a shadow effect.

For example, the light detection module further detects illuminance of each light for each light position, and the computer adjusts a shadow depth of an object in the virtual image based on the detected illuminance when applying a shadow effect.

For example, the three-dimensional information augmented video see-through display apparatus further includes a microphone module for detecting a sound generated within a predetermined radius based on the current position and a communication module for transmitting the detected sound to a preset sound transmission target.

According to one embodiment, when there are a plurality of microphone modules, based on azimuth azimuth of each of the plurality of microphone modules, further comprising a sense of space generating module for generating the direction data of the sound sensed by each of the plurality of microphone modules, communication The module further transmits the generated direction data to the sound transmission object.

For example, the 3D information enhancement video see-through display device further includes a lens distortion compensation lookup table that stores lens distortion compensation values matched for each of the plurality of display devices, and the lens distortion compensation module uses at least two lens distortion compensation lookup tables. The two composite images.

According to an embodiment, at least one of the camera interface module, the rectification module, the lens distortion compensation module, and the image generating module is implemented as a hardware chip.

According to an embodiment of the present invention, in order to achieve the above object, the three-dimensional information augmented video see-through display method is to obtain at least two real images from at least two camera modules, rectification for at least two real images performing a rectification, based on a lens distortion compensation value representing a value for compensating for distortion of the wide-angle lens for at least two real images, at least two composite images combining the virtual image into each of the at least two real images Correcting and generating a 3D image for virtual reality (VR) or augmented reality (AR) by performing side by side image processing on the at least two composite images.

For example, the step of performing rectification may include transforming coordinates identical to each other among a plurality of pixels included in each of the at least two real images based on the current reference target break point information on a pre-stored break point lookup table. Determining a current coordinate transformation target pixel group composed of at least one continuous pixel required, and each current coordinate transformation target pixel group including a current difference value that is a difference value from a current target coordinate that is a coordinate to be converted. And converting the coordinates of each pixel group of the current coordinate transformation target based on the current coordinate transformation information.

According to an embodiment, the current coordinate transformation information is a difference value from a next target coordinate, which is a coordinate to which each of the next coordinate transformation target pixel groups predetermined based on the next reference target breakpoint information on the pre-stored breakpoint lookup table is converted. The difference between the next difference and the current difference is stored in advance.

For example, the step of performing rectification may include converting coordinates of each of the next coordinate transformation target pixel groups based on a difference value between a next difference value and a current difference value.

For example, the three-dimensional information augmentation video see-through display method detects an illumination position within a predetermined radius based on the current position, and generates directional data for each illumination based on the current position based on the detected illumination position. And applying a shading effect to the object in the virtual image based on the generated direction data.

According to an embodiment, the 3D information enhancement video see-through display method may further include storing lens distortion compensation values matched for each of the plurality of display apparatuses in a lens distortion compensation lookup table, and correcting the lens distortion compensation lookup table. And correcting the at least two composite images using.

In order to achieve the above object, a rectification apparatus according to an embodiment of the present invention is a rectification (rectification) for at least two real images and a camera interface module for obtaining at least two real images from at least two camera modules A repetition module for performing a), wherein the repetition module includes a plurality of pixels included in each of the at least two actual images based on current reference target break point information on a pre-stored break point lookup table. Determine a current coordinate transformation target pixel group composed of at least one consecutive pixel requiring the same coordinate transformation, and each include a difference value with the current target coordinate that is a coordinate to be converted. Based on the current coordinate conversion information, each pixel group of the current coordinate conversion target It converts the coordinates.

Specific details of other embodiments are included in the detailed description and the accompanying drawings.

According to an embodiment of the present invention, by processing a three-dimensional image in a module implemented by a hardware chip without the help of a PC, the image obtained from the plurality of camera modules, a three-dimensional image for virtual reality or augmented reality in a video see-through method Can be generated.

According to an embodiment of the present invention, by performing a camera calibration for the stereo camera using the infrared image captured by the infrared camera module when the user's surroundings are dark, to increase the sharpness of the image taken by the stereo camera image The processing reliability can be improved.

According to an embodiment of the present invention, rectification may be performed on images captured from a plurality of camera modules to change the plurality of captured images as if they were captured from a camera arranged on the same column. This can provide high quality images.

According to an embodiment of the present invention, by considering the distortion of the wide-angle lens for each of the plurality of video see-through display device, by correcting the lens distortion of the wide-angle lens for each display device using a lens distortion compensation lookup table, The image quality can be improved by adding an opposite distortion to the image incident through the wide-angle lens.

According to an embodiment of the present invention, a virtual image to be combined with a photographed image (actual image) by detecting a surrounding light position and illumination of each light position through a light sensing module such as a CDS, and using the detection result. You can create a more vibrant three-dimensional image by adding a shading effect to it.

According to an embodiment of the present invention, by sensing the surrounding sound and the azimuth angle to which the sound is directed through the microphone module, and using the result of the detection to transmit the sound having a directionality to the other party, the other party to see the atmosphere of the field Make it feel vivid.

According to an embodiment of the present invention, the mounting base may be detachably installed on various display devices through fastening of the mounting base and the cover.

According to an embodiment of the present invention, the storage space of the breakpoint lookup table may be reduced by sequentially storing empty spaces on the breakpoint lookup table for the plurality of breakpoint information for rectification.

According to one embodiment of the present invention, the break point lookup table by previously storing a difference value between a next target coordinate and a current target coordinate, which are coordinates to be transformed, in the current coordinate transformation information for rectification, respectively. This can be done by minimizing access to.

1 is a block diagram illustrating a three-dimensional information augmentation video see-through display device according to an embodiment of the present invention.

2 is a diagram illustrating an operation of a three-dimensional information augmentation video see-through display device according to an embodiment of the present invention.

FIG. 3 is a PCB block diagram illustrating a connection state between a control unit and a peripheral device of a 3D information enhanced video see-through display device according to an embodiment of the present invention.

4 to 6 are flowcharts illustrating a three-dimensional information augmentation video seed display method according to an embodiment of the present invention.

FIG. 7 is a diagram illustrating a structure in which a three-dimensional information augmentation video seed display device according to an embodiment of the present invention is detachably coupled to a display device.

8 is a flowchart illustrating a step of performing rectification of the 3D information enhancement video seed display method according to an embodiment of the present invention.

9A and 9B are diagrams for explaining a breakpoint lookup table utilized in a conventional rectification algorithm.

10A and 10B are diagrams for explaining a breakpoint lookup table used in performing rectification of a 3D information enhancement video seed display method according to an embodiment of the present invention.

11A and 11B are diagrams for explaining current coordinate transformation information utilized in performing rectification of a 3D information enhancement video seed display method according to an embodiment of the present invention.

FIG. 12 is a diagram for explaining a double target field of current coordinate transformation information used in performing rectification of a 3D information enhancement video seed display method according to an embodiment of the present invention. to be.

FIG. 13 is a diagram illustrating a differential field of current coordinate transformation information used in performing rectification of the 3D information enhancement video seed display method according to an embodiment of the present invention.

FIG. 14 is a view illustrating a continuous information field defining a next target coordinate among current coordinate transformation information used in performing rectification of a 3D information enhancement video seed display method according to an embodiment of the present invention. It is a figure shown for description.

Advantages and / or features of the present invention and methods for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only the present embodiments to make the disclosure of the present invention complete, and common knowledge in the art to which the present invention pertains. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

Hereinafter, with reference to the accompanying drawings will be described embodiments of the present invention;

Referring to FIG. 1, a three-dimensional information augmentation video seed display device 100 according to an embodiment of the present invention may include a camera module 110, a camera interface module 120, a rectification module 130, Lens Distortion Compensation Module 140, Image Generation Module 150, Display Module 160, Camera Calibration Module 170, Light Detection Module 180, Microphone Module 185, Space Creation Module 190 , And the controller 195.

The camera module 110 may include two camera modules for capturing two actual images. That is, the camera module 110 may include a stereoscopic camera for capturing two actual images, which are images corresponding to the right eye and the left eye of the user. Therefore, the two actual images may be divided into left and right images (some regions overlap) on the basis of the photographing target.

According to an embodiment, the actual image may mean a captured image of the visible light region photographed by the two camera modules.

The camera module 110 may include three camera modules. In this case, two camera modules may be used as the stereo camera, and the other camera module may be used to detect a depth of an image.

In addition, the camera module 110 may include an infrared camera module for capturing infrared images. Infrared images can be used to enhance the image processing reliability of two real images when the surroundings are dark. That is, the infrared camera module may be driven together with the stereo camera when the surroundings become dark.

The camera interface module 120 obtains an image from the camera module 110. That is, the camera interface module 120 obtains two actual images from the stereo camera module consisting of two camera modules. In addition, the camera interface module 120 may obtain three real images from three camera modules, or alternatively, two real images and infrared images from a stereo camera module and an infrared camera module.

To this end, the camera interface module 120 may include an interface selection module (see "210" in FIG. 2) and a color space conversion module (see "220" in FIG. 2). The interface selection module selects an interface suitable for the camera in consideration of different interfaces for each camera. The color space conversion module converts color signals into other colors, such as converting YUV analog video to digital RGB (red, green, blue) video, converting RGB to CMYK (cyan, magenta, yellow, black), and so on. Do it. That is, the color space conversion module converts an image input through the interface selection module into a format for image processing.

The rectification module 130 performs rectification on two actual images. That is, the rectification module 130 can change the two actual images as if they were taken from a camera arranged on the same column. For example, the rectification module 130 may make the optical axis and the reference point parallel and intersect at infinity. Accordingly, scene overlap can be maximized, and distortion can be minimized.

In order to perform the replication, a reference point is needed. The rectification module 130 may extract a reference point from a chess board or a 3D object. The rectification module 130 may perform rectification using epipolar geometry. In this way, the reticulation module 130 may provide a high quality image by correcting the error of the instrument (alignment error of two cameras) with software.

According to an embodiment, the rectification module 130 may coordinate the same among the plurality of pixels included in each of the at least two real images based on the current reference target break point information on the prestored break point lookup table. Determine a current coordinate transformation target pixel group composed of at least one continuous pixel to be transformed, and each current coordinate transformation target pixel group includes a current difference value that is a difference from the current target coordinate that is a coordinate to be transformed The coordinates of each pixel group of the current coordinate transformation target may be converted based on the current coordinate transformation information.

In this case, a more detailed description of the rectification module 130 will be described later with reference to FIGS. 8 to 14, and overlapping description thereof will be omitted.

The lens distortion compensation module 140 combines two composite images by combining virtual images with each of the two real images based on a lens distortion compensation value representing a value for compensating for the distortion of the wide-angle lens with respect to the two actual images. Correct the image.

According to one embodiment, the lens distortion compensation module 140 is based on a lens distortion compensation value representing a value for compensating for the distortion of the wide-angle lens for the two rectified real image, the virtual image in each of the two actual image You can also correct two composite images combining.

In detail, the spectacle display device such as a head mount display (HMD) uses a wide-angle lens to widen the field of view because the gap between the user's eyes and the lens is narrow. At this time, an image incident through the wide-angle lens is distorted (distorted). This will occur. Therefore, in the present exemplary embodiment, the lens distortion compensation module 140 may apply lens distortion compensation values to two composite images to compensate for distortion of the wide-angle lens by applying opposite distortion to an image incident through the wide-angle lens.

In this case, the virtual image may be input from a computer linked with the three-dimensional information augmentation video see-through display device 100 and combined with each of two real images. The virtual image refers to a graphic image for providing a virtual reality (VR) or augmented reality (AV).

The spectacle display device may have a different lens distortion compensation value for each type or model. For example, the HMD of the A product may have a greater distortion of the wide-angle lens than the HMD of the B product. That is, corresponding lens distortion compensation values may be different for each of the plurality of video see-through based display devices. Therefore, in the present embodiment, the lens distortion compensation value matched for each of the plurality of video seed-based display devices may be stored in the lens distortion compensation look-up table.

The lens distortion compensation module 140 may correct two composite images using the lens distortion compensation lookup table. In other words, the lens distortion compensation module 140 extracts a lens distortion compensation value matched with the corresponding display device from the lens distortion compensation lookup table, and uses the extracted lens distortion compensation value to have two composite images having opposite distortions. You can correct it.

The image generation module 150 performs side by side image processing on the two composite images to generate a 3D image for the virtual reality (VR) or the augmented reality (AR). That is, the image generating module 150 may generate a 3D image for virtual reality or augmented reality by performing an image processing operation of converting two composite images divided into left and right into one image having the same size.

According to an embodiment, the side-by-side image processing is an algorithm for generating a 3D image and may refer to image processing for setting two images to be arranged left and right, and a detailed description thereof will be omitted.

The display module 160 displays the generated 3D image on the screen. Accordingly, the user can view the 3D image through the screen. However, in another embodiment, the display module 160 may not be included in the three-dimensional information augmented video seed display device 100, and may be implemented as an external display device.

The camera calibration module 170 may perform camera calibration on the two camera modules based on the infrared image in order to improve image processing reliability on the two actual images.

In detail, when the surroundings are bright, two actual images are clear, so the image processing reliability is high. However, when the surroundings are dark, the two actual images are not clear and the reliability of the image processing is not high. In order to solve this problem, in this embodiment, the image processing reliability of two real images may be improved through the camera calibration module 170.

That is, the camera calibration module 170 performs camera calibration for two camera modules, that is, stereo cameras, by using an infrared image captured by the infrared camera module when the surroundings become dark, thereby increasing the sharpness of two actual images. The image processing reliability thereof can be improved.

The light detection module 180 may detect a light position within a predetermined radius based on the current position. In addition, the light detection module 180 may further detect the illuminance of each light for each lighting position. That is, the light detection module 180 may detect the illumination position around the user and the illumination intensity of each illumination position. The sensed illumination position and illuminance can be used to add a shadow effect to the virtual image.

The microphone module 185 may detect a sound generated within a predetermined radius based on the current position. That is, the microphone module 185 may detect sound generated around the user. In this case, a plurality of microphone modules 185 may be provided. In this case, the plurality of microphone modules 185 may have direct azimuth angles. The detected sound may be transmitted to the other party (a preset sound transmission target, for example, the HMD of the other party in a video conference) through a communication module (not shown).

The sense of space generation module 190 may generate direction data for each illumination based on the current position based on the detected illumination position. Accordingly, the computer may give a shadow effect (shadow) to the object in the virtual image based on the generated direction data. In addition, the computer may adjust the shadow depth of the object in the virtual image based on the sensed illuminance when applying the shadow effect.

The spatial feeling generating module 190 generates direction data of sound detected by each of the plurality of microphone modules 185 based on the direction of azimuth of each of the plurality of microphone modules 185 when there are a plurality of microphone modules 185. can do. The generated direction data may be transmitted to the sound transmission target together with the detected sound.

The controller 195 is a three-dimensional information augmented video seed display device 100 according to an embodiment of the present invention, that is, a camera module 110, a camera interface module 120, a rectification module 130, Lens Distortion Compensation Module 140, Image Generation Module 150, Display Module 160, Camera Calibration Module 170, Light Detection Module 180, Microphone Module 185, Space Creation Module 190 Etc., the overall operation can be controlled.

The controller 195 may include at least one of the camera interface module 120, the rectification module 130, the lens distortion compensation module 140, and the image generating module 150. It may be implemented as a hardware chip.

Meanwhile, the 3D information enhancement video seed display device 100 according to an embodiment of the present invention may be combined with a plurality of various display devices. Here, the display device may include all of the various video see-through based display devices including the eyeglass display device. Therefore, the 3D information enhancement video seed display device 100 may be detachably coupled to the display device. For example, as shown in FIG. 7, the three-dimensional information augmentation video see-through display device (Stereoscopic Camera Module) 100 is mounted with a mounting base (Mounting base) and a cover (Back), respectively, in a state where the mounting base ( The mounting base may be detachably coupled to various display devices through a method in which the mounting base is detachably attached to the oculus.

Referring to FIG. 2, the camera interface module 120 selects an interface suitable for two left and right camera modules 110, CAM CON (L) and CAM CON (R), through the interface selection module 210, and a color space. The conversion module 220 converts two actual images captured by the CAM CON (L) and the CAM CON (R) into an image processing format.

At this time, the camera interface module 120 selects an interface suitable for one central camera module 110 CAM CON (C) through the interface selection module 210, and selects the CAM CON () through the color space conversion module 220. The image captured by C) can be converted into an image processing format. Here, the CAM CON (C) may be an infrared camera module for capturing an infrared image, and the camera calibration module 170 may improve image processing reliability of two actual images by performing a camera calibration using the infrared image. .

The image passed through the camera interface module 120 is input to the rectification module 130, and the rectification module 130 performs rectification on two actual images. The rectified image 230 passed through the rectification module 130 is transmitted to the PC 250 (240), the PC 250 generates a virtual image that is a graphic image to be combined with the Rectified Image (230) . The Rectified Image 230 is combined with the Rendered Image 260 from the PC 250 and input to the lens distortion compensation module 140. In the present embodiment, an image combining the Rectified Image 230 and the Rendered Image 260 is defined as a composite image. Therefore, two composite images are input to the lens distortion compensation module 140.

The two composite images are applied to the image generating module 150 after the opposite distortion is applied through the lens distortion compensation module 140 to be corrected to the same or similar to the actual image. The two composite images are image-processed in a side by side manner through the image generating module 150 and converted into one 3D image. The three-dimensional image is displayed on the screen through the display module (Oculus Lift / HMD Display) 160, so that the user can display the three-dimensional image for virtual reality (VR) or augmented reality (AV) through the screen of the Oculus Lift or HMD. You will be able to see the image.

On the other hand, the PC 250 may receive the illumination and direction data for each of the surrounding light through the light detection module 180 and the sense of space generation module 190 to give a shadow effect (shadow) to the object in the virtual image, You can also adjust the shadow depth. The virtual image 260 to which the shadow effect is applied may be combined with the Rectified Image 230 and input to the lens distortion compensation module 140.

In addition, the PC 250 may receive the surrounding sound through the microphone module 185 and the space generating module 190 and transmit the ambient sound to the counterpart through a network. In this case, the PC 250 may receive the direction data of the sound detected by each of the plurality of microphone modules 185 through the sense of space generating module 190 and transmit the received direction data to the counterpart through the network. As a result, the other party can directly hear the sound around the user, and can also feel the liveliness through the direction of the sound.

Referring to FIG. 3, the controller 195 may be implemented as a field programmable gate array (FPGA). Three camera modules 110 may be connected to the controller 195, of which two camera modules (Connector (L) and Connector (R)) 310 and 320 are stereo camera modules and the other one camera module. IR sensor (C) 330 is an infrared camera module.

The control unit 195 is connected to two main memories (main memory, for example, DDR3) 340, which are memory for image processing, and booting memory, which is an OS processing memory in which data for initializing the FPGA is stored. Memory 350 may be connected. The boot memory 350 may be implemented as an SD card.

The controller 195 may be connected to an HDMI 360 that is an input / output interface for transmitting an actual image to the PC 250 or receiving a virtual image from the PC 250. In this case, the HDMI 360 may also function as an input / output interface for transmitting the 3D image to the Oculus Rift / HMD for displaying the 3D image. Here, the HDMI 360 may be replaced by a wireless module that transmits high resolution video and audio in real time at a short range.

The control unit 195 may be connected to an ADC 170 that performs analog-to-digital conversion of sensing data sensed by the CDS 180 as a light sensing module and the MIC 185 as a microphone module. In addition, the controller 195 may be connected to the serial communication module 380 for data initialization (input at the time of setting the keyboard, etc.). In addition, the control unit 195 may be connected to a GPIO (General Purpose Input / Output) 390 and a SOM Cont. 395 as selectable options.

4 to 6, in step 410, the 3D information augmentation video see-through display device obtains two actual images from two camera modules.

At this time, the three-dimensional information augmented video see-through display device obtains two real images from two camera modules (510), then obtains an infrared image from an infrared camera module (520), and then processes the two real images. To improve reliability, camera calibration may be performed for two camera modules based on the infrared image.

Next, in step 420, the 3D information enhancement video seed display device performs rectification on the two real images.

At this time, a more detailed description of the step 420 will be described later with reference to FIGS. 8 to 14, and overlapping descriptions are omitted.

Next, in step 430, the three-dimensional information augmented video see-through display device combines the virtual image to each of the two real images to generate two composite images.

At this time, the three-dimensional information augmented video see-through display device detects the illumination position within a predetermined radius and the illumination of each light based on the current position (610), and then each of the illumination based on the current position based on the detected illumination position After generating the directional data (620), a shadow effect may be applied to the object in the virtual image based on the illumination of each light and the directional data for each light (630). Subsequently, the three-dimensional information augmented video see-through display device may combine the virtual image with the shadow effect to each of the two real images (640).

Next, in step 440, the 3D video enhancement video see-through display device corrects the two composite images based on a lens distortion compensation value representing a value for compensating for the distortion of the wide-angle lens for the two actual images. .

Next, in operation 450, the 3D information augmented video see-through display device performs side by side image processing on the two composite images to perform virtual reality (VR) or augmented reality (AR). Create a three-dimensional image.

Next, in step 460, the display device to which the 3D information-separation video see-through display device is detached and displays the generated 3D image.

Referring now to FIGS. 8-14, a step 420 of the redirection module 130 and the three-dimensional information augmentation video seed display method of the three-dimensional information augmentation video seed display device 100 will be described.

As shown in FIG. 8, step 420, which is a step of performing rectification on two actual images, is a step of determining a current coordinate transformation target pixel group 810 and a current coordinate transformation, respectively. In operation 820, the coordinates of each target pixel group may be converted.

In operation 810, the rectification module 130 transforms coordinates identical to each other among a plurality of pixels included in each of the at least two real images based on the current reference target break point information on the previously stored break point lookup table. This may mean a step of determining a current coordinate transformation target pixel group including at least one continuous pixel required.

For example, in performing the rectification in operation 810, some of the pixels consecutive to each other in each of the two real images may require the same coordinate transformation.

For example, a total of eight pixels are continuous with each other, and the first pixel, the second pixel, and the third pixel need to be coordinated by +1 in the x-axis direction, respectively, and the fourth, fifth, sixth, If the 7th pixel and the 8th pixel require coordinate transformation by +3 in the x-axis direction, the 1st to 3rd pixels may refer to the current coordinate transformation target pixel group, which is a continuous pixel requiring the same coordinate transformation. The fourth to eighth pixels may refer to a next coordinate transformation target pixel group that is a continuous pixel requiring the same coordinate transformation.

In this case, information that stores which pixels among consecutive pixels are required to have the same coordinate transformation may be defined as break point information, and in particular, break point information for determining a current coordinate transformation target pixel group may be defined as a break point for the current reference object. Defined as point information.

A plurality of break point information for distinguishing consecutive pixels requiring the same coordinate transformation among the plurality of pixels included in each of the two actual images may be stored on the break point lookup table.

9A and 9B, a conventional breakpoint lookup table applied in a rectification algorithm utilizing a conventional breakpoint lookup table will be described.

As shown in FIGS. 9A and 9B, the conventional break point lookup table may be separately stored for each of the X and Y axis directions.

In this case, as shown in FIGS. 9A and 9B, in the conventional breakpoint lookup table, the breakpoint information corresponding to the breakpoint information corresponding to each address previously allocated on the internal memory exists. If the corresponding breakpoint information does not exist, the table sets the corresponding address to an empty state.

At this time, the break point information is shown in "00000", "00000", "01002", "00001", "00802", "00803", "00002", "01003", "00001" and FIG. 9B shown in FIG. 9A. It may mean "00000", "00004", "04001", "04802", "01803", and "00004" shown.

That is, the conventional breakpoint lookup table has a problem in that wasted memory occurs when there is no breakpoint information corresponding to a pre-assigned address.

Referring now to FIGS. 10A and 10B, a breakpoint lookup table used in the step of performing rectification of the 3D information augmentation video seed display method according to an embodiment of the present invention will be described.

As shown in FIGS. 10A and 10B, the breakpoint lookup table utilized in performing the redirection of the 3D information augmented video see-through display method according to an embodiment of the present invention is similar to the conventional breakpoint lookup table. It may be stored separately for each of the X-axis direction and Y-axis direction.

10A and 10B, the breakpoint lookup table utilized in the step 420 of the present invention sequentially stores a plurality of breakpoint information in an empty space on an internal memory. As shown in each of the boxes of FIGS. 10A and 10B, it may refer to a table that separately stores storage address information on at least one memory for at least one break point information among the plurality of break point information.

As a result, the breakpoint lookup table utilized in the step 420 of the present invention stores the breakpoint information without wasting memory on the internal memory, thereby effectively utilizing the storage space of the internal memory.

Now, referring to FIGS. 9A and 10A simultaneously, the conventional break point lookup table is compared with the break point lookup table utilized in step 420 of the present invention.

The storage address information of "00000" which is the break point information of the m + 5th column of FIG. 9A is "0", and the storage address information of "00000" which is the first breakpoint information of the m + 10th column is "1", m The storage address information of "01002", which is the second breakpoint information of the + 10th column, is "2", and the storage address information of "00001", which is the first breakpoint information of the m + 15th column, is "3", m + 15 The storage address information of "00802", the second break point information of the first column, is "4", the storage address information of "00803", the third break point information of the m + 15th column, is "5", and the storage address information of the m + 20th column is "5". The storage address information of the first breakpoint information "00002" is "6", the storage address information of "01003" which is the second breakpoint information of the m + 20th column is "7", and the breakpoint of the m + 25th column The storage address information of the information "00001" is defined as "8", respectively.

In this case, in the break point lookup table utilized in the step 420 of the present invention shown in FIG. 10A, the respective break point information "00000", "00000", "01002", "00001", "00802", and "00803" "," 00002 "," 01003 "," 00001 "are sequentially stored, and then" 0 "and" storage address information corresponding to the first break point information for each row shown in FIG. 9A. 1 "," 3 "," 6 ", and" 8 "can be stored separately.

Similarly, the storage address information of "00000" which is breakpoint information of the n + 5th column shown in FIG. 9B is "0", and the storage address information of "00004" which is first breakpoint information of the n + 10th column is "1". If the storage address information of “00004”, which is the first breakpoint information of the n + 15 th column, is defined as “5”, each breakpoint is used in the breakpoint lookup table utilized in the step 420 of the present invention shown in FIG. 10B. After storing the point information "00000", "00004", "04001", "04802", "01803", and "00004" sequentially, the first break point for each row shown in Fig. 9B. The storage address information "0", "1", and "5" corresponding to the information may be separately stored.

In other words, the pre-stored breakpoint lookup table sequentially stores a plurality of breakpoint information in an empty space in the memory, and stores at least one breakpoint information of at least one breakpoint information among the plurality of sequentially stored breakpoint information. You can save more storage address information.

Referring back to FIG. 8, step 820, which is a step of transforming coordinates of each pixel group of the current coordinate transformation target, is described.

In operation 820, the rectification module 130 may convert the current coordinate transformation object based on the current coordinate transformation information including a current difference value that is a difference value between the current coordinate transformation target pixel group and the current target coordinate that is a coordinate to be converted. It may mean a step of converting coordinates of each pixel group.

For example, in step 820, the coordinates of the first pixel included in the current coordinate transformation target pixel group are (1,0), the coordinates of the second pixel are (2,0), and the coordinates of the third pixel are (3 , 0), and the current difference value included in the current coordinate transformation information when the first target coordinate is (1,1), the second pixel is (2,1), and the third pixel is (3,1) May mean as much as +1 in the y-axis direction.

Referring now to FIGS. 11A and 11B, current coordinate transformation information will be described.

For example, as illustrated in FIGS. 11A and 11B, a case in which the current coordinate conversion information is data composed of 18 bits for each of the X axis direction and the Y axis direction will be described. It is not limited to data.

As shown in FIG. 11A, the current coordinate transformation information for the X-axis direction includes a double target field including a predetermined number of bits, a differential field including a predetermined number of bits, and It may include a row number field including a predetermined number of bits.

According to an embodiment, the dual target field is a field indicating neighboring pixels for interpolating pixel values that are empty when at least two of the pixels included in the current coordinate conversion target pixel group are mapped to the same current target coordinate. It can contain a total of three bits.

According to an embodiment, the difference field may mean a field indicating a difference value between a current target coordinate, which is a coordinate to which each of the current coordinate transformation target pixel groups, and a current coordinate transformation target pixel group.

For example, when at least two camera modules are fisheye lens cameras, a differential field, which is a field indicating a current difference value, may include at least four bits.

This is because, when using a fisheye lens, which is a kind of wide-angle lens, the distortion caused by the lens becomes larger than that of a general lens. If the difference field includes four bits in total, the difference field is -8 to +7. Since it can be stored, the coordinate transformation of each pixel group of the current coordinate transformation target can be performed in a wider range.

For example, the horizontal column field is a field for storing coordinate information of pixels included in each pixel group of the current coordinate transformation target, and may include a total of 11 bits.

As shown in FIG. 11B, the current coordinate transformation information for the Y-axis direction includes a continuous information field including a predetermined number of bits, a differential field including a predetermined number of bits, and It may include a column number field including a preset number of bits.

In this case, the description of the difference field and the column field is the same as described with reference to FIG. 11A, and thus redundant descriptions thereof will be omitted.

For example, the continuous information field may include a next difference value and a current coordinate conversion target pixel, which is a difference value from the next target coordinate, which is a coordinate to be transformed, next to the current coordinate conversion target pixel group. A field that stores in advance a difference between a current difference value that is a difference value from a current target coordinate that is a coordinate to be converted to each group.

For example, if the current difference value for the current group of pixel transformation target coordinates is +5 in the X-axis direction, and the current difference value for the next pixel group of coordinate transformation targets is +4 in the X-axis direction, The continuous information field may store in advance a value of −1, which is a difference between a current difference value of +5 and a next difference value of +4.

That is, information that the next difference value of the next coordinate conversion target pixel group is less than 1 in the X-axis direction than the current difference value of the current coordinate conversion target pixel group may be stored in advance in the current coordinate conversion information.

As a result, the coordinate transformation of the next coordinate transformation target pixel group can be performed based on the difference value between the current difference value of the current coordinate transformation target pixel group and the next difference value of the next coordinate transformation target pixel group without repeatedly accessing the memory. Since it can be, the operation speed is shortened.

In other words, the current coordinate transformation information is a next difference value that is a difference value with a next target coordinate that is a coordinate to which each of the next coordinate transformation target pixel groups determined in advance is based on the next reference target breakpoint information on the previously stored breakpoint lookup table. The difference between the current difference and the current difference can be stored in advance.

As a result, in step 420, the rectification module 130 may convert the coordinates of each of the next coordinate conversion target pixel groups based on the difference between the next difference value and the current difference value.

Referring now to FIG. 12, a dual target field is described.

As shown in FIG. 12, when the bit value of 3 bits of the dual target field is 0,0,0, this may mean no empty target, and the bit value is 0,0,1. In this case, it may mean that the pixel value of an empty pixel is stored as the pixel value of the Up Forward. If the bit value is 0,1,0, the pixel value of the empty pixel is converted to the pixel value of the Up. If the bit value is 0,1,1, it can mean to save as the pixel value of the upper back. If the bit value is 1,0,0, it is the lower forward. If the bit value is 1,0,1, it means that it is stored as the pixel value of the lower. If the bit value is 1,1,0, it means the lower left. It may mean storing as a pixel value of Backward).

Referring now to FIG. 13, the difference field is described.

As shown in FIG. 13, when the difference field includes a total of four bits, the difference field is a difference value between a current target coordinate, which is a coordinate at which the current coordinate transformation target pixel group is to be transformed, and a coordinate of a current coordinate transformation target pixel group. The current difference value can be stored in the range of -8 to 7.

For example, if the bit value of the difference field on the X axis is 1,0,0,1, each of the plurality of pixels included in the current coordinate conversion pixel group should be coordinate converted in the X axis direction by -2. Can mean.

In this case, -2 corresponding to 1,0,0,1 may mean a current difference value.

Referring now to FIG. 14, the continuous information field is described.

As shown in FIG. 14, the continuous information field may store in advance a difference value between a current difference value for the current coordinate transformation target pixel group and a next difference value for the next coordinate transformation target pixel group.

For example, when the continuous information field includes three bits in total, one bit may mean a flag field and the remaining two bits may mean a difference value field.

In this case, the first bit of the difference value field may mean a sign bit and the second bit may mean an absolute value of the difference value.

As a result, when the continuous information field includes a total of three bits, the difference between the current difference value and the next difference value may be stored as one of -1, 0, 1, and in this case, the flag field may be set to one value. Have.

In this case, when the difference between the current difference value and the next difference value is one of -1, 0, 1, the coordinate conversion of the next coordinate conversion target pixel group can be performed without accessing the memory, and the current difference value and the next difference value If the difference between is less than -1 or greater than 1, the memory must be accessed again to perform coordinate transformation of the next coordinate transformation pixel group.

Similarly, although not shown in the drawing, when the continuous information field includes a total of five bits, a total of three bits remain except for the flag field 1 bit and the sign bit 1 bit, in which case the current information field is presently different. The difference between the value and the next difference is 15 in total: -7, -6, -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7. The difference value can be stored. As a result, when the difference value between the current difference value and the next difference value is in the range of -7 to 7, the coordinate conversion of the next coordinate transformation target pixel group can be performed without accessing the memory. Only when the difference between the difference value and the next difference value is less than or equal to -7 is greater than 7, the memory is accessed again and the coordinate transformation of the next coordinate transformation target pixel group is performed.

According to an embodiment, the lens distortion compensation module 140 may compensate for the lens distortion by using the same algorithm as the rectification module 130 described above.

In this case, the lens distortion compensation module 140 is identical to each other among a plurality of pixels included in each of the at least two actualized images based on the current reference lens distortion compensation breakpoint information on the prestored lens distortion compensation lookup table. Determine a current coordinate conversion target pixel group composed of at least one continuous pixel for which coordinate transformation is required, and include a current difference value that is a difference value from the current target coordinate, which is a coordinate to be converted, respectively. The coordinates of each pixel group of the current coordinate transformation target may be converted based on the current coordinate transformation information.

In this case, the pre-stored lens distortion compensation lookup table sequentially stores the plurality of lens distortion compensation break point information in an empty space on the memory, and compensates at least one lens distortion of the plurality of sequentially stored lens distortion compensation break point information. The storage address information on at least one memory for the break point information may be further stored.

In this case, the current coordinate transformation information may be a difference value from a next target coordinate, which is a coordinate to be transformed with each of the next coordinate transformation target pixel groups determined based on the next reference target lens distortion compensation breakpoint information on the previously stored lens distortion compensation lookup table. The difference between the next difference and the current difference can be stored in advance.

As a result, the lens distortion compensation module 140 may convert the coordinates of each of the following coordinate conversion target pixel groups based on the difference between the next difference value and the current difference value.

Now, a rectification apparatus (not shown) according to an embodiment of the present invention is described.

Here, the rectification apparatus (not shown) according to an embodiment of the present invention is the camera interface module 120 and the rectification module 130 in the three-dimensional information augmented video see-through display device 100 shown in FIG. ) May mean only a device.

For example, in accordance with one embodiment of the present invention, a rectification apparatus (not shown) may be directed to a camera interface module 120 that acquires at least two actual images from at least two camera modules and to at least two actual images. It may include a repetition module 130 to perform the application (rectification).

In this case, the rectification module 130 requests the same coordinate transformation among the plurality of pixels included in each of the at least two real images based on the current reference target break point information on the previously stored break point lookup table. Determine a current coordinate transformation target pixel group composed of at least one continuous pixel, and each current coordinate transformation target pixel group is based on current coordinate transformation information including a difference value with a current target coordinate that is a coordinate to be converted. Coordinates of each pixel group of the current coordinate conversion target are converted.

In this case, the description of each configuration of the rectification apparatus (not shown) according to an embodiment of the present invention is the same as described with reference to FIGS.

Embodiments of the invention include a computer readable medium containing program instructions for performing various computer-implemented operations. The computer readable medium may include program instructions, local data files, local data structures, or the like, alone or in combination. The media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs, DVDs, magnetic-optical media such as floppy disks, and ROM, RAM, flash memory, and the like. Hardware devices specifically configured to store and execute the same program instructions are included. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

While specific embodiments of the present invention have been described so far, various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the claims below, but also by the equivalents of the claims.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above-described embodiments, which can be variously modified and modified by those skilled in the art to which the present invention pertains. Modifications are possible. Accordingly, the spirit of the present invention should be understood only by the claims set forth below, and all equivalent or equivalent modifications thereof will belong to the scope of the present invention.

Claims

A camera interface module for obtaining at least two actual images from at least two camera modules;

A rectification module that performs rectification on the at least two actual images;

Lens distortion compensation for correcting at least two composite images combining a virtual image with each of the at least two real images based on a lens distortion compensation value representing a value for compensating for distortion of the wide-angle lens for the at least two real images module; And

And an image generation module configured to generate a 3D image for virtual reality (VR) or augmented reality by performing side by side image processing on the at least two composite images. 3D information enhancement video see-through display device.
The method of claim 1,

The repetition module,

Based on the current reference target break point information on a pre-stored break point lookup table, the plurality of pixels included in each of the at least two real images are composed of at least one consecutive pixel requiring the same coordinate transformation. Determine the current coordinate conversion target pixel group,

And converting the coordinates of each of the current coordinate transformation target pixel groups based on current coordinate transformation information including a current difference value that is a difference value from each current coordinate transformation target pixel group that is a coordinate to be converted. 3D information enhancement video see-through display device.
The method of claim 2,

The pre-stored break point lookup table is

A plurality of break point information is sequentially stored in an empty space in the memory,

And storing at least one storage address information on at least one memory of at least one break point information of the plurality of sequentially stored break point information.
The method of claim 2,

The current coordinate conversion information,

The difference value between the next difference value and the current difference value, which is a difference value from the next target coordinate, which is a coordinate to be transformed, respectively, according to the next reference target break point information on the previously stored break point lookup table. 3D information augmentation video see-through display device, characterized in that for storing in advance.
The method of claim 4, wherein

The repetition module,

And a coordinate of each of the next coordinate transform target pixel groups based on a difference value between the next difference value and the current difference value.
The method of claim 2,

When the at least two camera modules are each fisheye lens camera,

And a differential field, which is a field representing the current difference value, includes at least four bits.
The method of claim 1,

The at least two camera modules for capturing the at least two actual images;

An infrared camera module for photographing infrared images; And

And a camera calibration module configured to perform camera calibration on the at least two camera modules based on the infrared image to improve image processing reliability of the at least two real images. See-through display device.
The method of claim 1,

An illumination detection module configured to detect an illumination position within a predetermined radius based on the current position; And

Further comprising a sense of space generating module for generating the direction data for each light based on the current position based on the detected light position,

The virtual image is

Input from a computer interoperating with the three-dimensional information augmentation video see-through display device and combined with each of the at least two real images,

The computer is

And a shadow effect is applied to the object in the virtual image based on the generated direction data.
The method of claim 8,

The light detection module

Further detect the illuminance of each light for each light position,

The computer is

And a shadow depth of the object in the virtual image based on the detected illuminance when the shadow effect is applied.
The method of claim 1,

A microphone module detecting a sound generated within a predetermined radius of the current position; And

And a communication module configured to transmit the detected sound to a predetermined sound transmission target.
The method of claim 10,

If there are a plurality of microphone modules,

Based on the azimuth angle of each of the plurality of microphone modules, further comprising a sense of space generating module for generating the direction data of the sound sensed by each of the plurality of microphone modules,

The communication module

And transmitting the generated direction data to the sound transmission target.
The method of claim 1,

The apparatus may further include a lens distortion compensation lookup table configured to store the lens distortion compensation values matched by a plurality of display devices.

The lens distortion compensation module

And the at least two composite images are corrected by using the lens distortion compensation lookup table.
The method of claim 1,

And at least one of the camera interface module, the rectification module, the lens distortion compensation module, and the image generation module is implemented as a hardware chip.
Obtaining at least two actual images from at least two camera modules;

Performing rectification on the at least two actual images;

Correcting at least two composite images combining a virtual image with each of the at least two real images based on a lens distortion compensation value representing a value for compensating for distortion of the wide-angle lens for the at least two real images; And

And performing a side by side image processing on the at least two composite images to generate a three-dimensional image for virtual reality (VR) or augmented reality (AR). Augmented Video See-Through Display Method.
The method of claim 14,

The step of performing the rectification,

Based on the current reference target break point information on a pre-stored break point lookup table, the plurality of pixels included in each of the at least two real images are composed of at least one consecutive pixel requiring the same coordinate transformation. Determining a current group of coordinate transformation target pixels; And

Converting coordinates of each of the current coordinate transformation target pixel groups based on current coordinate transformation information including a current difference value that is a difference value from each current coordinate transformation target pixel group that is a coordinate to be converted. 3D information augmentation video see-through display method characterized in that.
The method of claim 15,

The current coordinate conversion information,

The difference value between the next difference value and the current difference value, which is a difference value from the next target coordinate, which is a coordinate to be transformed, respectively, according to the next reference target break point information on the previously stored break point lookup table. 3D information enhancement video see-through display method characterized in that for storing in advance.
The method of claim 16,

The step of performing the rectification,

And converting coordinates of each of the next coordinate transformation target pixel groups based on a difference value between the next difference value and the current difference value.
The method of claim 14,

Detecting a lighting position within a predetermined radius of the current position;

Generating directional data for each illumination based on the current position based on the detected illumination position; And

And applying a shadow effect to the object in the virtual image based on the generated direction data.
The method of claim 14,

Storing the lens distortion compensation value matched for each of a plurality of display devices in a lens distortion compensation lookup table,

The correcting step

And correcting the at least two composite images by using the lens distortion compensation lookup table.
A camera interface module for obtaining at least two actual images from at least two camera modules; And

A rectification module for performing rectification on the at least two actual images,

The repetition module,

Based on the current reference target break point information on a pre-stored break point lookup table, the plurality of pixels included in each of the at least two real images are composed of at least one consecutive pixel requiring the same coordinate transformation. Determine the current coordinate conversion target pixel group,

Rectification, characterized in that for converting the coordinates of each of the current coordinate conversion target pixel group on the basis of the current coordinate conversion information including a difference value with each current coordinate transformation target pixel group is a coordinate to be converted. Device.