KR102151265B1 - Hmd system and rehabilitation system including the same - Google Patents

Abstract

Disclosed are an HMD system and a rehabilitation training system including the same. The HMD system comprises: an HMD device; and a smartphone. The HMD device includes: a half-mirror that transmits light incident on a first surface and reflects light incident on a second surface; a smartphone holder that holds a smartphone such that a display of the smartphone faces the second surface; and an HMD camera that acquires an actual image of a first surface direction. The smartphone includes: a display held in the smartphone holder and outputting augmented reality content; and a processor for communicating with the HMD device in a wired/wireless manner. The processor acquires an actual image photographed through the HMD camera, recognizes at least one partial area from the actual image, generates an augmented reality layer including each augmented reality content corresponding to each partial area, and transmits an augmented reality layer to a display. The generation of the augmented reality layer may comprise the operations of generating a converted image obtained by converting the actual image, determining each augmented reality content data set including a location, area and direction of each augmented reality content, on the basis of each partial area data set including a location, area and direction of each partial area on the converted image, and generating each augmented reality content to include the same in the augmented reality layer, on the basis of the each augmented reality content data set.

Description

HMD system and rehabilitation training system including the same {HMD SYSTEM AND REHABILITATION SYSTEM INCLUDING THE SAME}

The following embodiments relate to a technology for providing an HMD system including a smartphone and a head-mounted display (HMD) device and a rehabilitation training system using the same.

As a background technology related to the embodiments, Korean Patent Application Publication No. KR 10-1761751 B1 discloses HMD correction in which direct geometric modeling is performed. Specifically, the prior literature (a) at least using a set of calibration matrices prescribed based on geometric information of the head model and the HMD model, a view of the virtual object corresponding to the reference object (view). Projecting, comprising: projecting the projected virtual object to be perceived as being substantially anchored to the reference object through the HMD; (b) in response to an instruction received through the user interface, changing at least one parameter in the correction matrix set by an amount indicated by the instruction; (c) projecting a view of the virtual object based on the correction matrix set including the at least one parameter changed by the amount; And (d) the (b) for parameters other than the at least one parameter in the correction matrix set until it can be recognized that the virtual object projected in step (c) is substantially aligned with the reference object. And (c) allowing the step to be performed, thereby adjusting the correction matrix, of an optical perspective (OST) head-mounted display (HMD) of an augmented reality (AR) system comprising a user interface. The correction method is disclosed.

Through this, prior literature provides a system/method that simplifies correcting an AR system into a real world scene. In addition, prior literature provides an AR system capable of rendering a virtual camera display by combining information obtained in real time with information of a virtual object in real time.

In addition, Korean Patent Application Publication No. KR 10-2028376 B1 discloses a calibration method and apparatus between a lip motion and an HMD using a bundle adjustment algorithm. Specifically, the prior literature is a step of converting n 2D images of different viewpoints obtained by photographing the front of an HMD with lip motion and infrared light bulbs formed with an infrared camera into n sparse matrices, through a bundle adjustment algorithm. Obtaining n projection matrices using one 3D point group and n sparse matrices generated by approximating n sparse matrices to a 3 dimensional original space, and n 2D 3D point groups through n projection matrices A step of obtaining n 2D point groups projected by projecting each onto an image. For each of n 2D images, a bulb cluster among the belonging points in the 2D point group after detecting a plurality of bulb clusters corresponding to the shape of the infrared bulb Determining the two-dimensional points located within, acquiring a three-dimensional center point of each light bulb cluster based on the three-dimensional points corresponding to the two-dimensional points located in the light bulb cluster among belonging points in the three-dimensional point group, and each light bulb cluster Disclosed is a method including the step of estimating and calibrating the lip motion and the origin of the HMD using the 3D center point and the cluster size of.

However, prior literatures have shown that when an actual image is taken at a position higher than the HMD wearer's eye level, for example, at a position about the wearer's forehead, the actual image is converted so that it is close to the actual image in front of the wearer with the naked eye, and the converted image is converted. It does not disclose a method or system for minimizing inconsistency or misalignment between the augmented reality content and the actual image in front of the wearer by determining the location, area, and direction of the augmented reality content on the basis. In addition, prior documents specify ranges related to augmented reality content in an actual image, or a method of generating a transformed image that is the basis for creating augmented reality content after assigning weight to ranges related to augmented reality content in an actual image. Does not start the system. Furthermore, prior literature discloses a method or system for obtaining a linear transformation so that a reference image taken with a reference camera positioned at eye level and a test image taken with an HMD camera positioned at a position other than eye level have a degree of agreement less than or equal to a predefined error. I never do that.

Accordingly, when the actual image is taken at a position higher than the HMD wearer's eye level, for example, at a position about the wearer's forehead, the actual image is converted to be close to the actual image of the front viewed by the wearer with the naked eye, and the converted image is based. As a result, there is a need to implement a technology that determines the location, area, and direction of the augmented reality content and minimizes the discrepancy and mismatch between the augmented reality content and the actual image in front of the wearer. In addition, after specifying ranges related to augmented reality content in an actual image, or assigning weights to ranges related to augmented reality content in an actual image, implementation of a technology that generates a transformed image that is the basis for creating augmented reality content is Is requested. Furthermore, there is a need for implementation of a technique for obtaining a linear transformation so that a reference image captured by a reference camera positioned at eye level and a test image captured by an HMD camera positioned at a position other than eye level have a degree of agreement less than or equal to a predefined error.

Korean patent publication KR 10-1761751 B1 Korean patent publication KR 10-2028376 B1 Korean Patent Application Publication KR 10-2019-0061825 A Korean Patent Application Publication KR 10-2015-0122975 A

In the embodiments, when an actual image is taken at a position higher than the HMD wearer's eye level, for example, at a position about the wearer's forehead, the actual image is converted to be close to the actual image in front of the wearer with the naked eye, and based on the converted image By determining the location, area, and direction of the augmented reality content, it is intended to provide a system that minimizes inconsistency or mismatch between the augmented reality content and the actual image in front of the wearer.

Embodiments provide a system that designates ranges related to augmented reality content in an actual image, or assigns weights to ranges related to augmented reality content in an actual image, and then generates a converted image that is the basis for creating augmented reality content. I want to.

The embodiments are intended to provide a system for obtaining a linear transformation so that a reference image photographed with a reference camera positioned at eye level and a test image photographed with an HMD camera positioned at a position other than eye level have a degree of matching less than or equal to a predefined error.

Further, the embodiments are intended to provide a method and system for solving the problems mentioned in the background art and the problems of the relevant technical field disclosed in the present specification.

An HMD system according to an embodiment includes an HMD device and a smart phone, and the HMD device transmits light incident on a first surface and reflects light incident on a second surface (half-mirror) ; A smart phone cradle for mounting the smart phone so that the display of the smart phone faces the second surface; And a HMD camera that acquires an actual image in the direction of the first surface, wherein the smartphone is mounted on the smartphone cradle, and includes a display configured to output augmented reality content; And a processor that communicates with the HMD device via wired or wireless communication, wherein the processor acquires an actual image photographed through the HMD camera, recognizes at least one partial region in the actual image, and corresponds to each partial region Generates an augmented reality layer including each of the augmented reality content to be transmitted, transmits the augmented reality layer to the display, and generating the augmented reality layer generates a converted image obtained by converting the actual image, and the converted image Each augmented reality content data including the location, area, and direction of each of the augmented reality content based on each partial area data set including the location, area, and direction of each of the partial areas on the image A set may be determined, and based on each of the augmented reality content data sets, each of the augmented reality contents may be generated and included in the augmented reality layer.

According to an embodiment, in the generation of the converted image, based on the real image, a left eye real image and a right eye real image are acquired, and the real image tensor corresponding to the left eye real image and the right eye real image An operation of generating a first transformed image tensor by applying a first linear transform, and generating the transformed image based on the first transformed image tensor may be performed.

According to an embodiment, in the generation of the converted image, based on the real image, the real left eye image and the real right eye image are acquired, and the real image tensor corresponding to the real left eye image and the real right eye image A cost (weight) is applied to elements corresponding to each of the partial regions, and based on the cost, the first linear transform is converted into a second linear transform, and the second is applied to the actual image tensor to which the cost is applied. An operation of generating a second transformed image tensor by applying a linear transform, and generating the transformed image based on the second transformed image tensor may be performed.

According to an embodiment, in the first linear transformation, a left-eye reference camera and a right-eye reference camera are installed at both eyes of a reference wearer when the HMD system is worn, and a left-eye reference photographed through the left-eye reference camera and the right-eye reference camera Acquire an image and a right-eye reference image, acquire a test image taken through the HMD camera, acquire a left-eye test image and a right-eye test image based on the test image, and the left-eye test image and the right-eye test image A reference estimated image tensor is generated by applying a candidate linear transformation to a corresponding test image tensor, and a norm of a difference between the reference image tensor corresponding to the left-eye reference image and the right-eye reference image, and the reference estimated image tensor is calculated. As a basis, it may be obtained by defining an error and defining a candidate linear transform that causes the error to be less than or equal to a predefined value as a first linear transform.

The rehabilitation training system including the HMD system includes an HMD system, a server, and predefined rehabilitation training units, and the server, while the HMD system wearer performs predefined movements for each predefined rehabilitation training unit. , The movement of the wearer is recorded, and each predefined rehabilitation unit includes some predefined areas having a corresponding relationship with the predefined movements, and each predefined area includes at least one Having a predefined correspondence relationship with the above augmented reality content, the HMD system includes an HMD device and a smartphone, and the HMD device transmits light incident on the first surface and light incident on the second surface. A half-mirror reflecting the light; A smart phone cradle for mounting the smart phone so that the display of the smart phone faces the second surface; And a HMD camera that acquires an actual image in the direction of the first surface, wherein the smartphone is mounted on the smartphone cradle, and includes a display configured to output augmented reality content; And a processor that communicates with the HMD device via wired or wireless communication, wherein the processor acquires an actual image photographed through the HMD camera, recognizes at least one partial region in the actual image, and corresponds to each partial region Generates an augmented reality layer including each of the augmented reality content to be transmitted, and transmits the augmented reality layer to the display, and the generation of the augmented reality layer generates a converted image obtained by converting the actual image, and the converted image Based on each partial region data set including the location, area, and direction of each of the partial regions on the image, the location, area, and direction of each of the augmented reality contents corresponding to each of the partial regions Each augmented reality content data set to be included may be determined, and based on each of the augmented reality content data sets, the augmented reality content may be generated and included in the augmented reality layer.

In the embodiments, when an actual image is taken at a position higher than the HMD wearer's eye level, for example, at a position about the wearer's forehead, the actual image is converted to be close to the actual image in front of the wearer with the naked eye, and based on the converted image By determining the location, area, and direction of augmented reality content, it is possible to provide a system that minimizes inconsistency or misalignment between the augmented reality content and the actual image in front of the wearer.

Embodiments provide a system that designates ranges related to augmented reality content in an actual image, or assigns weights to ranges related to augmented reality content in an actual image, and then generates a converted image that is the basis for creating augmented reality content. can do.

The embodiments may provide a system for obtaining a linear transformation such that a reference image photographed with a reference camera positioned at eye level and a test image photographed with an HMD camera positioned at a position other than eye level have a degree of matching less than or equal to a predefined error.

On the other hand, the effects according to the embodiments are not limited to those mentioned above, and other effects that are not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 is an exemplary diagram of a configuration of an HMD system according to an embodiment.
2 is a flow chart for explaining the operation of the HMD system according to an embodiment.
3 is a diagram for describing an operation of generating a converted image according to an embodiment.
4 is a diagram illustrating an operation of generating a converted image to which a cost is applied according to an exemplary embodiment.
5 is a diagram illustrating an operation of generating a first linear transform according to an embodiment.
6 is a view for explaining a rehabilitation training system according to an embodiment.
7 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the rights of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes to the embodiments are included in the scope of the rights.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed in various forms and implemented. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

When a component is referred to as being "connected" to another component, it is to be understood that it may be directly connected or connected to the other component, but other components may exist in the middle.

The terms used in the examples are used for illustrative purposes only and should not be interpreted as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc., as shown in the figure It can be used to easily describe the correlation between a component and other components. Spatially relative terms should be understood as terms including different directions of components during use or operation in addition to the directions shown in the drawings. For example, if a component shown in a drawing is turned over, a component described as "below" or "beneath" of another component will be placed "above" the other component. I can. Accordingly, the exemplary term “below” may include both directions below and above. Components may be oriented in other directions, and thus spatially relative terms may be interpreted according to orientation.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

In addition, in the description with reference to the accompanying drawings, the same reference numerals are assigned to the same components regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiments, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the embodiments, the detailed description thereof will be omitted.

1 is an exemplary diagram of a configuration of an HMD system according to an embodiment.

The HMD system 100 may include an HMD device 110 and a smartphone 120. The smartphone 120 may be mechanically and electrically connected to the HMD device 110. The HMD device 110 and the smartphone 120 may communicate via wired or wireless. In addition, the HMD device 110 and the smartphone 120 may transmit or receive data such as a server, an external environment detection sensor, a body change detection sensor, etc. through wired or wireless communication, or may be connected to the Internet or an intranet. The HMD device 110 may configure the HMD system 100 to provide augmented reality content to a wearer by interworking with the smartphone 120.

The HMD system 100 may be used for rehabilitation training. Specifically, in a rehabilitation training system conducted using a rehabilitation training tool, the HMD system 100 may provide augmented reality content to patients undergoing rehabilitation training. By combining the rehabilitation training tool and augmented reality content, patients can receive mixed reality rehabilitation training. Through this, the patient's immersion in rehabilitation increases and concentration is improved, so that the fun and therapeutic effect of rehabilitation training can be enhanced. Patients who use the rehabilitation training system may be mainly neurological patients, but as long as they are patients who need rehabilitation training, there are no special restrictions on the patient's symptoms, age, gender, and race. The specific configuration and operation of the rehabilitation training system including the HMD system 100 will be described later with reference to FIG. 6.

Looking back at the configuration of the HMD system 100, first, the HMD device 110 may include a half mirror 111, a smartphone cradle 112, and an HMD camera 113.

The half mirror 111 has a curved surface and may have a shape that covers both eyes of the wearer when the HMD system 100 is worn. The half mirror 111 may include a first surface and a second surface that are opposite to each other. The half mirror 111 may transmit light incident on the first surface and reflect light incident on the second surface. To this end, the half mirror 111 may adopt a configuration in which a metal such as aluminum, chromium, or nickel is coated on a transparent plate such as acrylic, polycarbonate (PC), or glass with suitable transmittance (20-25%, etc.). When the HMD system 100 is worn, the half mirror 111 may be fastened to the HMD device 110 such that the first side faces the outside and the second side faces the wearer. Since light incident on the first surface of the half mirror 111 is transmitted, the wearer of the HMD system 100 can see the light incident from the outside. Through this, the wearer of the HMD system 100 can see the actual appearance of the front side even when wearing the HMD system 100.

The smartphone holder 112 may mount the smartphone 120. The smartphone holder 112 may be configured by a combination of an elastic member, a binding member, a support member, and the like to mount the smartphone 120 of various standards. In addition, the smartphone cradle 112 may include a USB port or the like to enable wired communication between the smartphone 120 and the HMD device 110.

Specifically, the smartphone holder 112 may mount the smartphone 120 such that the display 121 of the smartphone 120 faces the second surface of the half mirror 111. Through this, light output from the display 121 of the smartphone 120 may enter the second surface of the half mirror 111 of the HMD device 110. The second side of the half mirror 111 faces the wearer when the HMD system 100 is worn, and light incident on the second side of the half mirror 111 is reflected, so that the wearer of the HMD system 100 is the smartphone 120 You can see the light (augmented reality layer) output from the display 121 of.

The HMD camera 113 may be positioned at a position where an actual image in the direction of the first surface of the half mirror 111 can be acquired. For example, when the HMD system 100 is worn, the HMD camera 113 may be positioned above the half mirror 111. In this case, the HMD camera 113 is positioned at the wearer's forehead position.

The HMD camera 113 may be a digital camera capable of capturing an image in front of the wearer of the HMD system 100. The image captured through the HMD camera 113 may be transmitted to the smartphone 120 mounted on the smartphone cradle 112.

In addition, the HMD device 110 may include a speaker, a head wearing unit, and an adjustment button. The speaker may output a sound corresponding to an augmented reality content or a pre-programmed sound corresponding to a rehabilitation unit or the like. The head wearing unit may be a string-band-belt-type configuration in which an elastic member and a fixing member are combined so that the wearer can wear the HMD system 100.

The adjustment button may mechanically and electrically adjust the relative angle of the half mirror 111 with respect to the HMD device 110. Through this, in case a discrepancy or mismatch occurs between the augmented reality contents and the actual image in front despite the operation described below with reference to FIGS. 2 to 5, the wearer of the HMD system 100 minimizes the discrepancy and misalignment. , It is possible to finely adjust the position of the half mirror 111 according to the characteristics of the face.

The smartphone 120 may be mounted on the smartphone cradle 112 of the HMD device 110. The smartphone 120 may be a general smartphone used by the wearer of the HMD device 110. The smart phone 120 has a calculation function of a typical computer; Save/reference function; Input/output function; And it may be configured to perform all or part of the control function. Specifically, the smartphone 120 may include a display 121 and a processor 122. The display 121 may output an augmented reality layer including augmented reality content. The processor 122 may function to allow the smartphone 120 to communicate with the HMD device 110 by wire or wirelessly.

An application developed or distributed by a person or organization that provides, operates, and manages a rehabilitation training system using the HMD system 100 may be installed on the smartphone 120. The application may perform a series of operations for providing augmented reality content by interlocking with other hardware/software components of the smartphone 120 and manage user feedback data related to the augmented reality content. The smartphone 120 may be connected to at least one artificial neural network that performs an inference function through an application. The application installed on the smartphone 120 may be configured to be updated in real time/non-real time through wired or wireless communication.

2 is a flow chart for explaining the operation of the HMD system according to an embodiment.

The operation of the HMD system 100 may be described based on the operation of the processor 122 of the smartphone 120.

First, the processor 122 may acquire an actual image captured through the HMD camera (210).

Specifically, the processor 122 of the smartphone 120 may acquire an actual image in front of the wearer of the HMD system 100 photographed through the HMD camera 113 of the HMD device 110. The HMD camera 113 may capture an image in front of the wearer at a position higher than the wearer's eye level, for example, about the wearer's forehead. Accordingly, the actual image acquired by the processor 122 may not match the front view seen by the wearer of the HMD system 100 with the naked eye.

Next, the processor 122 may recognize at least one partial region in the actual image (220).

The processor 122 may recognize some predefined areas through an object recognition algorithm or a tag recognition algorithm. Here, the tags may be located at characteristic points (edges, borders, center points, division points, vertices, etc.) of objects or objects having a relevance to the augmented reality content. For example, if an object or object having a relationship with the augmented reality content is a cube, tags may be placed at each vertex of the cube. The processor 122 recognizes tags in an actual image, infers positions of hidden or invisible tags based on the recognized tags, classifies the recognized tags and inferred tags by tag group, and classifies each tag group. Each can be recognized as a partial area.

Furthermore, the processor 122 may calculate a location, area, direction, etc. occupied by a partial area on the actual image. The calculation of a location, area, direction, etc. occupied by some areas may be performed based on tags.

To this end, each tag may have the same area. Through this, the processor 122 calculates a tag that is recognized as a relatively small area in an actual image as a tag located relatively far from the wearer, and a tag that is recognized as a relatively large area in the actual image is a tag located relatively close to the wearer. Can be calculated with Through this, the processor 122 may calculate the 3D positions of the tags from the 2D actual image.

In addition, each tag can be distinguished from other tags. Since each tag in the tag group is distinguished from other tags, the processor 122 uses the area, location, and three-dimensional space occupied by the tag group on the actual image based on the three-dimensional position of each tag forming the tag group. You can calculate enemy direction, placement, etc. Since the processor 122 recognizes a partial area based on the tag group, the area occupied by a partial area, a location, and a three-dimensional direction based on the area, position, three-dimensional direction, and arrangement of the tag group on the actual image , Placement, etc. can be obtained.

Each partial area may have a corresponding relationship with at least one augmented reality content. Some areas may be predefined for each predefined unit. For example, when the HMD system 100 is used in a rehabilitation training system, some areas may be predefined for each rehabilitation training unit. For example, each rehabilitation training tool used for each rehabilitation training unit may include tags that can be recognized by the processor 122 at a location related to augmented reality content. The processor 122 may recognize a partial area based on a tag group consisting of a plurality of predefined tags. Also, the processor 122 may calculate an area, a location, a direction, etc. occupied by a partial area on the actual image.

Subsequently, the processor 122 may generate an augmented reality layer including each augmented reality content corresponding to each partial region (230).

At this time, the HMD camera 113 captures an image in front of the wearer at a position higher than the wearer's eye level, for example, about the wearer's forehead. Accordingly, when the processor 122 determines the location, area, and direction of the augmented reality content to be output to the display 121 based on the location, area, and direction of a partial area on the actual image captured by the HMD camera 113 , Inconsistencies or deviations in location, area, and direction may occur between the output augmented reality contents and the actual image of the front viewed by the wearer with both eyes. For example, water expressed in augmented reality should be output as it is actually contained in a paper cup, but water expressed in augmented reality may be output inconsistent or out of alignment as it is floating around the actual paper cup. In order to prevent such a problem, the processor 122 needs to convert the actual image captured by the HMD camera 113 and determine the location, area, and direction of the augmented reality content to be output based on the converted image.

Specifically, the processor 122 may generate a converted image obtained by converting the actual image (231). The operation of generating the converted image by the processor 122 will be described later with reference to FIGS. 3 and 4. The transformed image may be closer to the actual image of the front viewed by the wearer with the naked eye than the actual image photographed at a position other than the wearer's eye level of the HMD system 100.

Subsequently, the processor 122 may determine each augmented reality content data set based on each partial region data set (232). The data set may be an object, an array or a list, and may refer to individual variables and functions. Processor 122 is based on each partial region data set (data set) including the position, area, and direction of each partial region on the converted image, each including the position, area, and direction of each augmented reality content Augmented reality content data set can be determined.

Subsequently, the processor 122 may generate each augmented reality content based on each augmented reality content data set and include it in the augmented reality layer (233). To this end, the smartphone 120 may have previously converted augmented reality contents corresponding to some areas into a database, or interlocked with a server that has previously converted augmented reality contents corresponding to some areas into a database. Each augmented reality content may be a content having a relationship with a partial area having a corresponding relationship. For example, when a part of the area is the edge of a paper cup, the databaseized augmented reality content having a corresponding relationship may be a water graphic contained in the paper cup. Each augmented reality content may have a degree of freedom with respect to location, area, direction, and arrangement.

Processor 122 is based on each partial region data set (data set) including the position, area, and direction of each partial region on the converted image, each including the position, area, and direction of each augmented reality content Augmented reality content data set can be determined. Subsequently, the processor 122 may generate each augmented reality content by determining a location, area, direction, arrangement, etc. of each augmented reality content based on each augmented reality content data set. Subsequently, the processor 122 may include the generated augmented reality contents in one augmented reality layer.

In the following order, the processor 122 may transmit the augmented reality layer to the display 121 (240). The display 121 may output an augmented reality layer. The display 121 of the smartphone 120 may be mounted on the smartphone holder 112 to face the second surface of the half mirror 111 of the HMD device 110. Through this, the augmented reality layer output from the display 121 of the smartphone 120 may enter the second surface of the half mirror 111 of the HMD device 110. The second side of the half mirror 111 faces the wearer when the HMD system 100 is worn, and light incident on the second side of the half mirror 111 is reflected, so that the wearer of the HMD system 100 is the smartphone 120 You can view augmented reality contents on the augmented reality layer output from the display 121 of.

Through the above, a wearer of the HMD system 100 can view augmented reality contents on the augmented reality layer output by the HMD system 100. On the other hand, since light incident on the first surface of the half mirror 111 of the HMD device 110 is transmitted, the wearer of the HMD system 100 can see the augmented reality contents as well as the actual image of the front. Through this, the wearer of the HMD system 100 can see an actual image in front of the augmented reality contents. In particular, the HMD system 100 does not determine the location, area, direction, and arrangement of augmented reality contents based on the actual image captured by the HMD camera 113 not located at the wearer's eye level, but the HMD camera ( 113) is determined on the basis of the converted image that is converted so that it is close to the actual image of the front viewed by the wearer with the naked eye. Through this, the HMD system 100 may provide an augmented reality layer with less inconsistency or misalignment between the augmented reality contents and the actual image in front to the wearer of the HMD system 100.

3 is a diagram for describing an operation of generating a converted image according to an embodiment.

The HMD camera 113 captures an image in front of the wearer at a position higher than the wearer's eye level, for example, about the wearer's forehead. Accordingly, when the processor 122 determines the location, area, and direction of the augmented reality content to be output to the display 121 based on the location, area, and direction of a partial area on the actual image captured by the HMD camera 113 , Inconsistencies or deviations in location, area, and direction may occur between the output augmented reality contents and the actual image of the front viewed by the wearer with both eyes. For example, water expressed in augmented reality should be output as it is actually contained in a paper cup, but water expressed in augmented reality may be output inconsistent or out of alignment as it is floating around the actual paper cup. In order to prevent such a problem, the processor 122 needs to convert the actual image captured by the HMD camera 113 and determine the location, area, and direction of the augmented reality content to be output based on the converted image.

The operation of generating the converted image by the processor 122 may include the following operations.

First, the processor 122 may acquire a left-eye real image and a right-eye real image based on the real image 301. The actual image can be adjusted to a real left-eye image and a real right-eye image, respectively, through a curved surface calibration algorithm. Rather than using a two-dimensional real image, when using two two-dimensional images, a real left-eye image and a real right-eye image in combination, the processor 122 determines the three-dimensional properties and properties of some areas on the real image (perspective, Depth, lighting, shadows, etc.) can be calculated more easily.

Next, the processor 122 may generate a real image tensor 302 corresponding to the real left eye image and the real right eye image. The processor 122 extracts information on each pixel or mass of the real left-eye image and the real right-eye image, and uses the information of each pixel or mass of the real left-eye image and the real right-eye image as an element. Can be created. The information corresponding to each pixel or mass may include a horizontal position, a vertical position, a depth, color, brightness, saturation, and luminance of each pixel or mass.

Subsequently, the processor 122 may generate the first transformed image tensor 322 by applying the first linear transform 310 to the actual image tensor 302.

The first linear transformation 310 may be a linear transformation such that an error becomes less than or equal to a predefined value. The error may be defined based on a norm of a difference between the reference image tensor corresponding to the left-eye reference image and the right-eye reference image, and the reference estimated image tensor. A reference image tensor may be generated based on a left-eye reference image and a right-eye reference image. The left-eye reference image and the right-eye reference image may be acquired through a pre-installed left-eye reference camera and a right-eye reference camera in the step of obtaining the first linear transformation. The left-eye reference camera and the right-eye reference camera may be installed at both eyes of the reference wearer. The reference estimated image tensor may be generated by applying a candidate linear transformation to a test image tensor generated based on the left eye test image and the right eye test image. The left eye test image and the right eye test image may be acquired based on the test image captured by the HMD camera 113 in the step of obtaining the first linear transformation. The operation of obtaining the first linear transformation will be described later with reference to FIG. 5.

The first transformed image tensor 322 may be generated by applying the first linear transform 310 to the actual image tensor 302. Since the norm of the difference between the first converted image tensor 322 and the reference image tensor is less than or equal to a predefined value, the converted image 321 corresponding to the first converted image tensor 322 is a reference wearer corresponding to the reference image tensor. The left-eye reference image and the right-eye reference image acquired through the left-eye reference camera and the right-eye reference camera installed at both eyes of the may show a degree of correspondence equal to or less than a predefined value.

In the following order, the processor 122 may generate the converted image 321 based on the first converted image tensor 322. An operation in which the processor 122 generates an image based on the tensor may be performed through a reverse operation of the operation of generating a tensor based on the image. That is, the processor 122 may generate the transformed image 321 in a manner of determining information on each pixel or mass point on the transformed image based on each element of the first transformed image tensor 322.

Through the above, the processor 122 may compare the HMD system 100 with an actual image photographed at a position other than the wearer's eye level, and obtain a converted image close to the actual image of the front viewed by the wearer with the naked eye. When determining the location, area, and direction of the augmented reality contents to be output to the display 121 based on the converted image, the position, area, and direction are inconsistent between the output augmented reality contents and the actual image in front of the wearer viewed with both eyes · Mismatch can be reduced. For example, inconsistent or mismatched output of water expressed in augmented reality as floating around a paper cup can be prevented. Through this, the HMD system 100 can reduce unsatisfactory user experience of wearers, while increasing concentration and interest in augmented reality content.

4 is a diagram illustrating an operation of generating a converted image to which a cost is applied according to an exemplary embodiment.

The processor 122 does not generate a converted image based on the entire range of the actual image captured through the HMD camera 113, and designates ranges related to augmented reality content from the actual image, or relates to augmented reality content from the actual image. After assigning weights to the ranges, a transformed image can be generated. Through this, the processor 122 performs the conversion by focusing on a partial range, not the entire range of the actual image, so that the converted image is generated in real time and outputs augmented reality contents without buffering, or augmented reality contents from the actual image By intensively minimizing the discrepancies and discrepancies corresponding to the parts related to the HMD system 100, discrepancy and discrepancy of ranges related to the augmented reality content in the actual appearance in front of the wearer of the HMD system 100 may be further minimized.

The operation of the processor 122 generating the converted image to which the cost is applied may include the following operations.

First, the processor 122 may acquire an actual left eye image and an actual right eye image based on the actual image 401. The operation of acquiring the real left eye image and the real right eye image may be the same as the operation described with reference to FIG. 3. Meanwhile, compared with the embodiment of FIG. 3, the actual image 401 may include some areas 430 having a corresponding relationship with the augmented reality contents. For example, some of the areas 430 may be edges of paper cups. The augmented reality content corresponding to some areas 430 may be water graphics contained in a paper cup.

Next, the processor 122 may generate an actual left eye image and an actual image tensor 302 corresponding to the right eye actual image. The actual image tensor 302 may be the same as the actual image tensor 302 described with reference to FIG. 3.

Subsequently, the processor 122 may obtain the actual image tensor 402 to which the cost is applied by applying a cost 431 to elements corresponding to each partial region in the actual image tensor 302. The cost 431 may be applied by designating some areas related to augmented reality content in an actual image, or applying weights to some areas related to augmented reality content in an actual image.

When the cost 431 is a method of designating some areas related to augmented reality content in the actual image, the cost 431 leaves the elements corresponding to some areas in the actual image tensor 302 and does not correspond to some areas. Unless elements can be made to zero.

When the cost 431 is a method of assigning weights to some areas related to augmented reality content, the cost 431 is the actual image tensor 302 multiplying the values of elements corresponding to some areas by the weights, and Elements that do not correspond to can be left alone. In the actual image tensor 302, the weight of the cost 431 multiplied by the elements corresponding to a partial region will be a Gaussian distribution based on the distance between the HMD system 100 and each tag of the partial region. I can. That is, the weight of the cost 431 may be larger as an element corresponding to a pixel or a material point located at the center of the tag relatively close to the HMD system 100 in the actual image tensor 302 is. On the other hand, the weight of the cost 431 may decrease as an element corresponding to a pixel or a material point located at the periphery of the tag, or an element corresponding to a pixel or a material point of a tag farther from the HMD system 100 than the corresponding tag. .

In the following order, the processor 122 may convert the first linear transform 310 into the second linear transform 410 based on the cost 431. The second linear transformation 410 may be a linear transformation obtained by correcting an effect of applying the cost 431 to the first linear transformation 310 and the actual image tensor 302. Since the first linear transformation 310 is a linear transformation such that the error is less than or equal to a predefined value, the second linear transformation 410 is the norm of the reference image tensor to which the cost is applied and the actual image tensor 402 to which the cost is applied. It may be an estimation of a linear transformation such that the error obtained based on the difference of) becomes less than or equal to a predefined value.

Next, the processor 122 may generate the second transformed image tensor 422 by applying the second linear transform 410 to the actual image tensor 402 to which the cost is applied. The error obtained based on the difference between the second transformed image tensor 422 and the norm of the reference image tensor to which the cost is applied may be expected to be less than or equal to a predefined value.

Subsequently, the processor 122 may generate a transformed image 421 based on the second transformed image tensor 422. The operation of generating the converted image may be the same as the operation described with reference to FIG. 3. For some regions 430 corresponding to the cost 431, the converted image 421 corresponding to the second converted image tensor 422 is a converted image 321 corresponding to the first converted image tensor 322 Compared with, inconsistencies or deviations between augmented reality contents and an actual appearance of the wearer of the HMD system 100 may be further minimized.

Through the above, the processor 122 compares the converted image generated through the operation described with reference to FIG. 3 and performs the conversion by focusing on a partial range rather than the entire range of the actual image. By generating and outputting augmented reality contents without buffering, or by intensively minimizing inconsistencies and misalignments corresponding to parts related to augmented reality contents in the actual image, the augmented reality contents and the augmented reality contents in the real image in front of the wearer of the HMD system 100 Inconsistency and misalignment of related ranges can be further minimized. Through this, the HMD system 100 can reduce dizziness, motion sickness, discomfort, and the like of wearers due to buffering occurrence, etc., while also naturally expressing complex augmented reality content.

In addition, when some areas are predefined for each predefined unit, for example, when the HMD system 100 is used in the rehabilitation training system and some areas are predefined for each rehabilitation training unit, the processor 122 Converted images corresponding to the second transformed image tensor 422 by simply applying the cost corresponding to the partial regions corresponding to each predefined unit to the actual image tensor 302 and the first linear transform 310 (421) can be created. Through this, it is possible to operate the system more efficiently than the case of individually obtaining a linear transformation that causes the error to fall below a predefined value for each actual image tensor to which the cost is applied. For example, if the processor 122 follows the operation of FIG. 4, when a correction occurs in a partial region of a predefined unit, the cost is corrected in response to the correction of the partial region, and the corrected cost is converted into the actual image tensor 302. And the first linear transform 310, a transformed image 421 corresponding to the second transformed image tensor 422 may be generated. This is more efficient than the case of newly obtaining a linear transformation that causes the error between the actual image tensor to which the corrected cost is applied and the reference image tensor to which the corrected cost is applied to be less than or equal to a predefined value.

On the other hand, there may be a case in which the coast 431 is designed to move due to the movement of some areas in the actual image. For example, when the actual image shows two paper cups, and a part of the area is the edge of the paper cup, when the wearer of the HMD system 100 moves the paper cup, the part of the area may move on the actual image. If there is a preceding first viewpoint and a following second viewpoint, at the first viewpoint, the processor 122 generates a location, area, direction, etc. of the partial region of the second viewpoint based on the movement of the partial region; Generate an expected cost based on the expected partial area; Based on the expected cost, generate an expected second linear transform; The expected cost and the expected second linear transform can be cached. At the second time point, if it is determined that the partial region has moved within the predicted motion and the predefined error range, the processor 122 uses the predicted cosort as it is or generates a cost based on the predicted cost; Using the expected second linear transform as it is, or generating a second linear transform based on the expected second linear transforms; An actual image tensor, a second transformed image tensor, and a transformed image can be generated more quickly. Through this, even in a situation in which some areas related to augmented reality content are moving, the HMD system 100 can reduce dizziness, motion sickness, and discomfort of wearers due to buffering, etc., while also expressing moving augmented reality content naturally.

5 is a diagram illustrating an operation of generating a first linear transform according to an embodiment.

In the step of obtaining the first linear transformation 310, the HMD system 100 may be equipped with a left-eye reference camera 501 and a right-eye reference camera 502 at both eyes of a reference wearer when worn. Information such as gender, age, race, and glabellar distance of the reference wearer may be previously databased.

The first linear transformation 310 will be generated based on an operation of converting an image photographed through the left-eye reference camera 501 and the right-eye reference camera 502 and an image photographed through the HMD camera 113 to match as much as possible. I can. A left-eye reference camera 501; Right eye reference camera 502; And, the HMD camera 113 may assume a situation in which the same tag 500 is photographed.

First, the HMD system 100 may acquire a left-eye reference camera 501 and a left-eye reference image 531 and a right-eye reference image 532 photographed through the right-eye reference camera 502. The left-eye reference image 531 and the right-eye reference image 532 may correspond to an actual front view of the reference wearer when wearing the HMD system 100.

Next, the HMD system 100 may generate a reference image tensor corresponding to the left-eye reference image 531 and the right-eye reference image 532. The operation of generating a tensor from an image may be consistent with the operation described with reference to FIGS. 3 and 4.

Subsequently, the HMD system 100 may acquire a test image 510 photographed through the HMD camera 113. The HMD system 100 may acquire a left eye test image 521 and a right eye test image 522 based on the test image 510. The operation of acquiring the left eye test image 521 and the right eye test image 522 may be consistent with the operation of acquiring the actual left eye image and the actual right eye image described with reference to FIGS. 3 and 4.

In the following order, the HMD system 100 may generate a reference estimated image tensor. To this end, the HMD system 100 may generate a test image tensor corresponding to the left eye test image 521 and the right eye test image 522. Subsequently, the HMD system 100 may generate a reference estimated image tensor by applying a candidate linear transformation to the test image tensor.

Next, the HMD system 100 may define an error based on a norm of a difference between the reference image tensor and the reference estimated image tensor. As the norm, Euclidean norm, p-norm, maximum norm, or the like may be employed. The error may be defined by using the norm as it is or by removing unnecessary information from the norm (cutting the digits after a predefined decimal point, etc.).

Subsequently, the HMD system 100 may define a candidate linear transform that causes the error to be less than or equal to a predefined value as the first linear transform 310. Candidate linear transformations that cause errors to fall below a predefined value can be derived through algorithms consisting of a system of equations, multiple regression analysis, machine learning, iteration, and combinations thereof. I can. The predefined value may be defined as a ratio of an initial error defined based on a norm of a difference between a test image tensor and a reference image tensor. For example, the predefined value may be a value obtained by multiplying the initial error by p (0<p<1). p can be, for example, 1/10 or 1/20.

Through the above, the HMD system 100 may obtain the first linear transformation 310 such that the tensor corresponding to the image photographed through the HMD camera 113 matches the reference image tensor to a predetermined value or less. . When the first linear transform 310 is applied to the tensor of the image captured through the HMD camera 113 at a position other than the wearer's eye level, and a transformed image is generated based on the tensor to which the first linear transform 310 is applied, The transformed image can be matched as much as possible with the actual image of the front viewed from the reference wearer's eye level. Through this, the HMD system 100 can minimize discrepancies or misalignments between the location, area, and direction in which the augmented reality content is to be output and the actual image in front of the reference wearer.

Meanwhile, when a converted image is to be generated by focusing on a partial range rather than the entire range of the actual image, the cost 431 may be applied to the actual image tensor. In this case, the second linear transform 410 to be applied to the actual image tensor to which the cost 431 is applied may be generated based on the first linear transform 310. Therefore, if you want to apply different costs to the actual image of the same front, for example, if some areas corresponding to augmented reality content move or can move, in order to obtain a linear transformation for the situation where each cost is applied, By installing the left-eye reference camera 501 and the right-eye reference camera 502, there is no need to obtain a linear transformation in each case through the operation described with reference to FIG. 5, and the cost is applied based on the linear transformation in which the cost is not applied. Since linear transformation can be obtained, efficient system operation and update, and high applicability can be achieved.

Furthermore, the HMD system 100 may adjust the first linear transformation 310 using a wearer-specific characteristic variable. To this end, the HMD system 100 may obtain information such as the wearer's gender, age, race, and glabellar distance. Subsequently, the HMD system 100 includes information such as gender, age, race, and glabellar distance of the wearer; The first linear transformation 310 may be adjusted based on information such as gender, age, race, and glabellar distance of the reference wearer.

Specifically, the HMD system 100 can check the increasing tendency of the error-defined based on the difference between the norm of the reference image tensor and the reference estimated image tensor, as the wearer's gender is inconsistent with the reference wearer's gender. have. Subsequently, the HMD system 100 may adjust the first linear transform 310 in a direction that cancels the increasing tendency of the error. For example, the first linear transformation 310 may be adjusted so as to obtain elements that change beyond a predefined level in the reference estimated image tensor due to gender discrepancy, and compensate for changes in the corresponding elements. Furthermore, the HMD system 100 may adjust the first linear transformation 310 by performing the same operation on the wearer's age, race, and glabellar distance.

The HMD system 100 may pre-database the adjustment degree of the first linear transformation 310 for each wearer, or may be linked with a server that has previously converted the adjustment degree of the first linear transformation 310 for each wearer into a database. Through this, the HMD system 100 may adjust and use the first linear transformation 310 according to the wearer, even if the wearer's gender, age, race, and glabellar distance are different from the reference wearer.

6 is a view for explaining a rehabilitation training system according to an embodiment.

The rehabilitation training system includes an HMD system 100; Server 600; And it may include rehabilitation training units (610, 620, 630). The HMD system 100 wearer (rehabilitation patient) may perform rehabilitation training using one rehabilitation training unit while wearing the HMD system 100. The HMD system 100 may provide augmented reality content. The rehabilitation training units 610, 620, 630 may include rehabilitation training tools 611, 621, 631. Through this, the rehabilitation training system may provide real rehabilitation training tools 611, 621, 631 and augmented reality content together. Through this, rehabilitation patients can be provided with mixed reality rehabilitation training in which real objects and augmented reality are combined. Through this, the patient's immersion in rehabilitation increases and concentration is improved, so that the fun and therapeutic effect of rehabilitation training can be enhanced. Patients who use the rehabilitation training system may be mainly neurological patients, but as long as they are patients who need rehabilitation training, there are no special restrictions on the patient's symptoms, age, gender, and race.

The HMD system 100 includes the components described with reference to FIGS. 1 to 5 and may perform the operations described with reference to FIGS. 1 to 5. The HMD system 100 may provide augmented reality content to patients undergoing rehabilitation training. Specifically, the rehabilitation training units 610, 620, and 630 may include partial areas 613, 623, and 633 corresponding to augmented reality content. The HMD system 100 acquires the actual images 612, 622, 632 taken through the HMD camera 113; Recognize at least one partial area 613, 623, 633 in the actual image; Generating an augmented reality layer including each augmented reality content corresponding to each partial area; The augmented reality layer may be transmitted to the display 121.

In this process, the generation of the augmented reality layer generates a converted image obtained by converting the actual image; Each augmented reality content data including the location, area, and direction of each augmented reality content on the basis of each partial area data set including the location, area, and direction of each partial area on the converted image Determine the set; Based on each augmented reality content data set, each augmented reality content may be generated and included in an augmented reality layer. Through this, the location, area, and direction of the output augmented reality content may be determined based on the converted image. At this time, the transformed image is applied to the actual image by applying the first linear transform 310 such that the error-defined based on the difference between the norm of the reference image tensor and the reference estimated image tensor, is equal to or less than a predefined value. Can be created. Accordingly, the augmented reality contents that are output and the actual image of the front that the wearer sees with both eyes can minimize the discrepancy and misalignment of the location, area, and direction.

The server 600 may be its own server owned by a person or organization providing a rehabilitation training system; May be a cloud server; It may be a peer-to-peer (p2p) set of distributed nodes. The server 600 includes arithmetic functions of an ordinary computer; Save/reference function; Input/output function; And it may be configured to perform all or part of the control function. The server 600 may include at least one artificial neural network that performs an inference function. The server 600 may be configured to communicate with the HMD system 100 over wired or wirelessly.

The server 600 may record the movement of the wearer (rehabilitation patient) while the wearer of the HMD system 100 performs predefined movements for each of the predefined rehabilitation units 610, 620, and 630. . The server 600 evaluates the achievement of rehabilitation patients based on the recorded movements; Based on the evaluation, a rehabilitation program for rehabilitation patients can be scheduled.

The rehabilitation training units 610, 620, and 630 may be configured according to a rehabilitation training program. Each of the rehabilitation training units 610, 620, and 630 may include some predefined areas 613, 623, and 633 having a correspondence relationship with predefined movements. Each of the predefined partial areas 613, 623, and 633 may have a predefined correspondence relationship with at least one augmented reality content. In the embodiment, three rehabilitation training units are listed, but the number of rehabilitation units is not particularly limited.

The predefined movement for each rehabilitation unit may be a movement for practicing one gross motion or one fine motion. Alternatively, the predefined movement may be a movement for practicing a complex overall movement, a complex fine movement, and a linkage between the overall movement and the fine movement.

Each of the rehabilitation training units 610, 620, and 630 may include rehabilitation training tools 611, 621, and 631. Rehabilitation training parishes (611, 621, 631) are one whole exercise; One micro-motion; Complex total movement; Complex micromotor; Alternatively, it may be a teaching tool for practicing the linkage of the overall movement and the micro movement. For example, the first parish 611 of the first rehabilitation unit 610 may be a parish in which clothespins are inserted into an iron bar rising upward; The second parish 621 of the second rehabilitation unit 620 may be a parish that inserts blocks into a wooden board with holes; The third parish 631 of the third rehabilitation training unit 630 may be a parish for practicing water pouring with two paper cups.

The rehabilitation training tools 611, 621, 631 may include some areas 613, 623, 633 corresponding to augmented reality content, and some areas 613, 623, and 633 may include predefined movements and You can have a correspondence relationship. For example, the augmented reality content of the first parish 611 is a graphic in which smoke is emitted from an arbitrary part of the iron bar, a partial area 613 corresponding to the augmented reality content is an iron bar, and the predefined movement is an iron bar with a clothespin. It may be a movement to pick up the part where the smoke comes from; The augmented reality content of the second parish 621 is a graphic blooming from a block inserted into a wooden board, a partial area 613 corresponding to the augmented reality content is a wooden board, and a predefined movement is a block in the hole of the wooden board. May be a movement to fit; The augmented reality content of the third parish (631) is water graphics contained in a paper cup, and some areas 613 corresponding to the augmented reality content are the edges of the paper cups, and a predefined movement draws water from one paper cup to the other. It can be a follow movement.

The operations performed by the rehabilitation training system may include the following.

First, a situation in which the wearer (rehabilitation patient) of the HMD system 100 performs rehabilitation training using one rehabilitation unit may be assumed. For example, a wearer of the HMD system 100 may perform rehabilitation training using the first rehabilitation training unit 610. The rehabilitation training patient may move to perform a predefined movement using the first parish 611 of the first rehabilitation unit 610 while wearing the HMD system 100.

The server 600 may record the movement of the wearer while the wearer of the HMD system 100 performs predefined movements of the first rehabilitation training unit 610. The predefined movement may be a movement of picking up a part where smoke comes out of an iron bar with a clothespin. The server 600 evaluates the rehabilitation training achievement of the rehabilitation patient according to the degree to which the recorded movements of the HMD system 100 correspond to the predefined movements; Based on the evaluation, it is possible to deepen, change, maintain, and weaken the rehabilitation program of rehabilitation patients.

The first rehabilitation unit 610 may include a first parish 611. The first parish 611 may be a parish in which clothespins are inserted into an iron bar rising above. The first rehabilitation training unit 610 may include some predefined areas 613 having a correspondence relationship with predefined movements. The predefined movement of the first rehabilitation unit 610 may be a movement of picking up a part of the iron bar where smoke is emitted with a clothespin. The partial area 613 of the first rehabilitation training unit 610 may be an iron bar. A partial region 613 of the first rehabilitation unit 610 may have a predefined correspondence relationship with at least one augmented reality content. The augmented reality content corresponding to the partial area 613 may be a graphic in which smoke is emitted from an arbitrary part of the iron bar.

The HMD system 100 acquires an actual image 612 including the first parish 611 taken through the HMD camera 113; Recognize at least one partial area 613 in the actual image; Generating an augmented reality layer including each augmented reality content corresponding to each partial area; The augmented reality layer may be transmitted to the display 121. The augmented reality content may be a graphic in which smoke is emitted from an arbitrary part of the iron bar.

In this process, the generation of the augmented reality layer generates a converted image obtained by converting the actual image; Each augmented reality content data including the location, area, and direction of each augmented reality content on the basis of each partial area data set including the location, area, and direction of each partial area on the converted image Determine the set; Based on each augmented reality content data set, each augmented reality content may be generated and included in an augmented reality layer. Through this, the location, area, and direction of the output augmented reality content may be determined based on the converted image. In this case, the converted image may be close to the actual image of the front viewed by the wearer with the naked eye as compared with an actual image photographed at a position other than the wearer's eye level of the HMD system 100. Accordingly, the augmented reality contents that are output and the actual image of the front that the wearer sees with both eyes can minimize the discrepancy and misalignment of the location, area, and direction. For example, it is possible to prevent inconsistencies or discrepancies in which augmented reality content that smokes from an iron bar starts to rise in an area other than the iron bar, or starts to rise in a location other than where the clothespin should be picked up.

Through the above, the embodiment may be made of a rehabilitation motion linked with a real teaching aid, rather than performing an operation in the air like the existing augmented reality rehabilitation training program. Through this, the embodiment may provide a mixed reality rehabilitation training system using real teaching tools in addition to augmented reality content. The mixed reality rehabilitation training system can include not only gross motion but also fine motion such as finger movements in the rehabilitation program, and rehabilitation effects not only for chronic patients but also for severely ill and mild patients through the rehabilitation training platform. Can be seen. Further, through rehabilitation linked with augmented reality and real teaching aids, the patient can match the sensations during rehabilitation with the sensations in real life, and the patient's concentration during rehabilitation training can be improved.

7 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

The device 701 according to an embodiment includes a processor 702 and a memory 703. The processor 702 may include at least one of the devices described above with reference to FIGS. 1 to 6, or may perform at least one method described above with reference to FIGS. 1 to 6. Specifically, the device 701 may be an HMD device 110, a smartphone 120, a server 600, an artificial neural network learning device, or the like.

The memory 703 may store information related to the above-described methods or a program in which methods described below are implemented. The memory 703 may be a volatile memory or a nonvolatile memory.

The processor 702 can execute a program and control the device 701. The code of a program executed by the processor 702 may be stored in the memory 703. The device 701 is connected to an external device (eg, a personal computer or a network) through an input/output device (not shown), and may exchange data through wired or wireless communication.

The device 701 may be used to train an artificial neural network or use the learned artificial neural network. The memory 703 may include a learning artificial neural network or a learned artificial neural network. The processor 702 may train or execute an artificial neural network algorithm stored in the memory 703. The apparatus 701 for training the artificial neural network and the apparatus 701 using the learned artificial neural network may be the same or may be separate.

The embodiments described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices, methods, and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, such as one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

As described above, although the embodiments have been described by the limited drawings, various technical modifications and variations can be applied based on those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments and claims and equivalents fall within the scope of the following claims.

Claims (5)

In an HMD system including an HMD device and a smartphone,
The HMD device,
A half-mirror that transmits light incident on the first surface and reflects light incident on the second surface;
A smart phone cradle for mounting the smart phone so that the display of the smart phone faces the second surface; And
HMD camera that acquires an actual image in the direction of the first surface
Including,
The smartphone,
It is mounted on the smartphone cradle,
A display for outputting augmented reality content; And
Processor for wired or wireless communication with the HMD device
Including,
The processor,
Acquire an actual image taken through the HMD camera,
Recognize at least one partial area in the actual image,
Creating an augmented reality layer including each augmented reality content corresponding to each partial area,
Transmitting the augmented reality layer to the display,
Generation of the augmented reality layer,
Generate a converted image by converting the actual image,
On the basis of each partial region data set including the position, area, and direction of each of the partial regions on the converted image, each augmentation including the position, area, and direction of each of the augmented reality contents Determine the real content data set,
Based on the respective augmented reality content data sets, generating each of the augmented reality content and including it in the augmented reality layer
Including motion,
Generation of the converted image,
On the basis of the real image, the real image of the left eye and the real image of the right eye are acquired,
A first transformed image tensor is generated by applying a first linear transform to the real image tensor corresponding to the real image of the left eye and the real image of the right eye,
Generating the converted image based on the first converted image tensor
Consisting of actions,
HMD system.


delete The method of claim 1,
Generation of the converted image,
Applying a cost (weight) to elements corresponding to each of the partial regions in the actual image tensor,
Based on the cost, converting the first linear transform into a second linear transform,
Applying the second linear transformation to the actual image tensor to which the cost is applied to generate a second transformed image tensor,
Based on the second transformed image tensor, generating the transformed image
Consisting of actions,
HMD system.
The method of claim 1,
The first linear transformation is,
When wearing the HMD system, a left-eye reference camera and a right-eye reference camera are installed at both eyes of the reference wearer,
Acquire a left-eye reference image and a right-eye reference image taken through the left-eye reference camera and the right-eye reference camera,
Acquiring a test image taken through the HMD camera,
Based on the test image, a left eye test image and a right eye test image are obtained,
Applying a candidate linear transformation to a test image tensor corresponding to the left eye test image and the right eye test image to generate a reference estimated image tensor,
An error is defined based on a norm of a difference between the left-eye reference image and a reference image tensor corresponding to the right-eye reference image, and the reference estimated image tensor,
Defining a candidate linear transformation that causes the error to be less than or equal to a predefined value as a first linear transformation
Obtained by including actions,
HMD system.
The rehabilitation training system including the HMD system includes an HMD system, a server, and predefined rehabilitation training units,
The server,
While the HMD system wearer performs predefined movements for each predefined rehabilitation unit, the wearer's movement is recorded,
Each of the predefined rehabilitation training units,
Includes some predefined areas having a corresponding relationship with the predefined movements,
Each of the predefined areas has a predefined correspondence relationship with at least one augmented reality content,
The HMD system,
Including an HMD device and a smartphone,
The HMD device,
A half-mirror that transmits light incident on the first surface and reflects light incident on the second surface;
A smart phone cradle for mounting the smart phone so that the display of the smart phone faces the second surface; And
HMD camera that acquires an actual image in the direction of the first surface
Including,
The smartphone,
It is mounted on the smartphone cradle,
A display that outputs augmented reality content; And
Processor for wired or wireless communication with the HMD device
Including,
The processor,
Acquire an actual image taken through the HMD camera,
Recognize at least one partial area in the actual image,
Creating an augmented reality layer including each augmented reality content corresponding to each partial area,
Transmitting the augmented reality layer to the display,
Generation of the augmented reality layer,
Generate a converted image by converting the actual image,
Location and area of each of the augmented reality content corresponding to each of the partial regions based on a data set of each partial region including the position, area, and direction of each partial region on the converted image , Determine each augmented reality content data set including the direction,
Based on the respective augmented reality content data sets, generating each of the augmented reality content and including it in the augmented reality layer
Including motion,
Generation of the converted image,
On the basis of the real image, the real image of the left eye and the real image of the right eye are acquired,
A first transformed image tensor is generated by applying a first linear transform to the real image tensor corresponding to the real image of the left eye and the real image of the right eye,
Generating the converted image based on the first converted image tensor
Consisting of actions,
Rehabilitation training system including HMD system.

Images