KR102500237B1 - Ar/vr skeletal training method, apparatus and system using model - Google Patents

Abstract

An electronic device according to one embodiment of the present invention may comprise a memory and a processor. The processor can detect approach of a model tool within a specified level to at least one point connecting a first part and a second part on a modeled carcass, generate first vibration on the model tool in response to the approach of the model tool, display information indicating that the model tool was allowed to come in contact with a correct point to dismantle the modeled carcass, detect the approach of the model tool within a specified level of a point corresponding to a bone other than at least one point connecting the first part and the second part on the modeled carcass, generate second vibration on the model tool in response to the approach of the model tool, and display information indicating that the model tool was in contact with an incorrect point to dismantle the modeled carcass. According to the present invention, the modeled carcass is composed of a plurality of parts and includes a magnetic object at at least one point connecting the first part and the second part, so that the points are connected by using magnetism and other points having no magnetism are physically connected by using at least one of a rubber band, a yarn, or a Velcro. The first part and the second part can be configured to be allowed to be separated from each other or joined to each other. The model tool can include an electronic control device that provides vibration when approaching the specific point containing a magnetic object within a specified level on the modeled carcass. Accordingly, the education of learners can be effectively performed.

Description

AR/VR bone training method, apparatus and system using model {AR/VR SKELETAL TRAINING METHOD, APPARATUS AND SYSTEM USING MODEL}

The following embodiments relate to an AR/VR bone training method, apparatus, and system using a model.

Recently, demand for electronic devices supporting augmented reality (AR) or mixed reality (MR) services, which provide information by overlapping virtual images on images or backgrounds of real-world elements, is rapidly increasing. are doing

Augmented Reality (AR), a field of virtual reality, is a computer graphics technique that synthesizes virtual objects or information in a real environment to make them look like objects existing in the original environment. Augmented reality can reinforce and provide additional information that is difficult to obtain in the real world by synthesizing virtual objects on the basis of the real world.

As a next-generation immersive media device (hereinafter referred to as AR device) supporting augmented reality services, wearable electronic devices that can be worn on the body (e.g., head-mounted device, head-mounted display, or AR glasses) are being developed. As communication technology develops to enable large-capacity data transmission, such as 5G communication, interest in AR devices is increasing.

If a VR/AR immersive Korean beef bone training system is applied to train Korean beef bone boning by applying the above AR/VR technology, it will greatly reduce the time and cost required for bone bone training and contribute to the development of the domestic Korean beef industry.

Prior Document 1 (KR Patent Publication 10-2022-0107506, Mobile Augmented Reality-based Cooking Experience Learning System) provides a cooking experience based on mobile augmented reality, but, like the present application, actually uses tools on models for meat, meat, and body parts. There is a limit to not providing a direct sense of use.

Prior Document 2 (KR Publication No. 10-2019-0023003, Bone age reading device and method based on image processing and machine learning) classifies bones or reads age based on an input image, but VR and AR There is a limitation in not being able to provide a method of providing learning to the user using a model of the device and meat or meat and body parts.

(Prior Document 0001) KR Publication No. 10-2022-0107506 (Prior Document 0002) KR Publication No. 10-2019-0023003

One of the most difficult parts when constructing a bone training system for Korean cattle using VR/AR technology is to provide direct feedback to the experiencer while simultaneously satisfying economic feasibility and function. There is no proper tool to solve the problem.

The present invention is an augmented reality that can safely and efficiently provide a boning education effect that was difficult for actual learners to experience, taking into account the characteristics of cooking tools such as knives and carcass that must be used so that the experiencer can feel the physical resistance of the knife during actual boning. An object of the present invention is to provide a bone experiential learning system based on the bone structure.

An object of the present invention is to provide an augmented reality-based bone experiential learning system that allows learners to effectively perform education rather than simply displaying 3D objects on a display of a mobile device.

An electronic device according to an embodiment may include a memory and a processor.

The processor detects that the model tool approaches within a specified level to at least one point connecting the first part and the second part on the model body, and causes a first vibration on the model tool in response to the approach of the model tool. and displays information indicating that the model tool has been brought into contact with the correct point to dismantle the model carcass, and at least one point connecting the first part and the second part on the model carcass but not a bone Detects that the model tool approaches the corresponding point within a specified level, generates a second vibration on the model tool in response to the model tool's approach, and contacts the model tool at an inappropriate point to dismantle the model intellect. You can display information indicating that you have done it. Here, the model carcass consists of a plurality of parts, includes a magnetic object at at least one point connecting the first part and the second part, and is connected using magnetism, and the remaining points that are not magnetic are rubber bands or threads. Alternatively, they may be physically connected using at least one of Velcro, and the first part and the second part may be separated or coupled to each other. The model tool may include an electronic control device that provides vibration when a specific point, including a magnetic object, on the model piece of land is approached within a specified level.

An electronic device according to an embodiment may be combined with hardware and controlled by a computer program stored in a medium to execute any one of the methods described above.

The electronic device according to an embodiment may provide a sense of bone bone so that dynamics and hand sensations can be felt by visually synchronizing the actual image of Korean beef carcass with the actually installed teaching aids using VR technology.

An electronic device according to an embodiment may install an electronic device related to a motion in a model knife linked with a cognitive model to detect a motion, evaluate a learner's boning motion performance, and analyze and provide problems.

The AR/VR boning education method using a model according to an embodiment can provide a learning experience on boning without using expensive meat by using model carcass that is divided into small divisions and combined into large divisions, and can provide body parts It is possible to provide a learning experience on body anatomy without using

In the AR/VR boning education method using a model according to an embodiment, resistance during boning can be fed back through a model knife and a Korean beef carcass training model, and at the same time, information necessary for education can be obtained in real time, providing more realistic experiential education. can do.

1 is an exemplary diagram of a configuration of an electronic device according to an embodiment.
Figure 2 is a flow chart showing an AR, VR bone training method using a model according to an embodiment.
3 illustrates a method for distributed rendering of an AR device according to an embodiment.
4 is a diagram showing the configuration of an artificial intelligence model according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes can be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents or substitutes to the embodiments are included within the scope of rights.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be modified and implemented in various forms. Therefore, the embodiments are not limited to the specific disclosed form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Terms used in the examples are used only for descriptive purposes and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

In addition, in the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description will be omitted.

The embodiments may be implemented in various types of products such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices.

1 is an exemplary diagram of a configuration of an electronic device according to an embodiment.

The electronic device 101 according to an embodiment may include a processor 120 and a memory 130, and some of the illustrated components may be omitted or replaced. The electronic device 101 according to an embodiment may be a server or a terminal. According to an embodiment, the processor 120 is a component capable of performing calculations or data processing related to control and/or communication of each component of the electronic device 101, and may include one or more processors. The memory 130 may store information related to the above-described method or a program in which the above-described method is implemented. Memory 130 may be volatile memory or non-volatile memory. The memory 130 may store various file data, and the stored file data may be updated according to the operation of the processor 120 .

According to one embodiment, the processor 120 may execute a program and control the device 101 . Program codes executed by the processor 120 may be stored in the memory 130 . Operations of the processor 120 may be performed by loading instructions stored in the memory 130 . The electronic device 101 may be connected to an external device (eg, a personal computer or network) through an input/output device (not shown) and exchange data.

According to one embodiment, there will be no limitations on the calculation and data processing functions that the processor 120 can implement on the electronic device, but hereinafter, a function of performing bone training using model education, AR, and VR will be described.

Figure 2 is a flow chart showing an AR, VR bone training method using a model according to an embodiment.

Operations described through FIG. 2 may be implemented based on instructions that may be stored in a computer recording medium or memory (eg, the memory 130 of FIG. 1). The illustrated method may be executed by the electronic device (eg, the electronic device 101 of FIG. 1 ) previously described with reference to FIG. 1 , and technical features described above will be omitted below. The order of each operation in FIG. 2 may be changed, some operations may be omitted, or some operations may be performed simultaneously.

In operation 210, the processor (eg, the processor 120 of FIG. 1 ) may detect that the model tool approaches at least one point connecting the first part and the second part on the model map within a specified level.

Model carcass consists of a plurality of parts, includes a magnetic object at at least one point connecting the first part and the second part, and is connected using magnetism, and the remaining points that are not magnetic are rubber bands, threads, or It may be physically connected using at least one of Velcro, and the first part and the second part may be configured to be separated or coupled to each other. The model tool may include an electronic control device that provides vibration when a specific point, including a magnetic object, on the model piece of land is approached within a specified level.

In operation 220, the processor 120 may generate a first vibration on the model tool in response to the approach of the model tool and display information indicating that the model tool has been brought into contact with the correct location to dismantle the model object.

In operation 230, the processor 120 may detect that the model tool approaches a point corresponding to a bone other than at least one point connecting the first and second portions on the model body within a specified level.

In operation 240, the processor 120 may generate a second vibration on the model tool in response to the approach of the model tool, and display information indicating that the model tool has been contacted at an inappropriate location to dismantle the model object. .

According to one embodiment, the processor 120 detects the position of the model tool on the model cognitive model, displays the name of a specific part of the model cognitive education on a display based on the position of the model tool, and displays the name of the specific part on the display. Based on the model tool, it is possible to provide a control method of the model tool that includes information on at least one of the angle, depth, position, or intensity of force applied to a specific part of the model tool. The processor 120 detects that the model tool approaches at least one point connecting the first part and the second part on the model map within a specified level, maintains the direction of magnetism, and determines the strength of the magnetism. It is controlled below the level to provide information indicating that the model is positioned at the right point to dismantle the model carcass and that an appropriate level of force is being applied. The processor 120 detects that the model tool approaches a point corresponding to a bone, which is not at least one point connecting the first part and the second part on the model cognitive body within a specified level, and changes the direction of magnetism. Controlling the model tool to push it to provide information indicating that it is located at an incorrect point to dismantle the carcass, and controlling the magnetic intensity to exceed a specified level to indicate that an inappropriate level of force is being applied can provide information.

The processor 120 sets a designated time to provide the user with a user experience to develop the model intellect, and at least one of the angle, depth, position of the model tool or the intensity of force applied to a specific part using an artificial intelligence learning model Give a score based on one, provide an analysis of why the score is low for users whose scores are below the specified level, set a specified time again to provide users with a user experience to develop the model education, and artificial intelligence learning On the model, it is possible to learn the user's learning experience, score, and analysis of problems with model carcass bone bone, and to provide an analysis of the wrong bone bone method for each user.

According to an embodiment, the processor 120 displays an actual cognitive education image corresponding to the model education together with the model education, synchronizes the education image and the model education, and when the shape of the model education is changed by a model tool. , The carcass image may also be changed into a shape corresponding to the shape of the model carcass. When the model carcass is cut into the first part and the second part by the model tool, the processor 120 determines at least one of intramuscular fat degree, meat color, fat color, texture, maturity, loin cross-sectional area, and back fat thickness of the first part. Information may be displayed, and information on at least one of intramuscular fat degree, meat color, fat color, texture, maturity, loin cross-sectional area, and back fat thickness of the second part may be displayed. The model tool may be manufactured to have more than a specified level of bending or less than a specified level of strength.

According to an embodiment, the processor 120 may perform a head tracking task, a hand tracking task, an eye tracking task, a scene analysis (eg, scene understanding) task, or a rendering task. Performance data or AR usage data of the electronic device may be obtained by using a sensor while performing a task including at least one and performing an augmented reality service. The processor 120 performs performance evaluation corresponding to the AR context based on the performance data of the electronic device or AR use data, calculates a performance evaluation score for each AR performance evaluation factor, and analyzes the performance evaluation to obtain information about vulnerabilities. It is provided to the user, a personalized performance preset in which at least some of configuration parameters of the basic performance preset are adjusted is generated based on a performance evaluation result, and the generated personalized performance preset may be applied to the augmented reality service.

The performance data of the electronic device includes GPU clock and CPU load of the electronic device, GPU clock and GPU load of the electronic device, network type, and tethering network type between the electronic device and the AR device. , tethering network signal strength, rendering frames per second (FPS), a surface temperature of the electronic device, a threshold temperature of the electronic device, and a resolution of the electronic device.

The AR usage data includes the display frequency of the AR device, the operating temperature of the AR device, the refresh rate of the AR device, the screen brightness of the AR device, the threshold temperature of the AR device, network traffic, tracking frequency, tracking coordinates, tracking latency, or tracking accuracy, tracking input image quality, scene analysis result, scene analysis accuracy, location information of the electronic device and GPS-based location information of the AR device, It may include at least one of error process information during augmented reality service and AR app usage time.

According to one embodiment, the processor 120 captures the motion of the model tool using a camera, analyzes physical properties related to the motion of the model tool from the captured video frame images, and then the user's body It is possible to determine the size of the model tool to be suitable for the part, render the augmented content object for the bone, and set the interaction. The processor 120 overlaps and displays the model intellect and the augmented content object for the bone in a see-through manner, and based on at least one of the location, moving direction, or strength of the moving force of the model tool on the model intellect Information on the bones of the model carcass to be provided to the user may be determined, and the information about the bones of the model carcass may be provided using a display.

The display may include any one of a Head Mounted Display (HMD) and smart glasses.

3 illustrates a method for distributed rendering of an AR device according to an embodiment.

An application programming interface (API) hosted by the server may provide at least one function of data access, upload, download, or deletion through a processor (eg, the processor 120 of FIG. 1 ).

In operation 302, the processor 120 may estimate a reality feature using the AR device. There are various characteristics of reality displayed using AR devices, but hereinafter, virtual cognitive education images and virtual tools corresponding to real model education and model tools will be described.

In operation 304, the processor 120 may use the AR device to estimate features of model education and model tools.

At operation 306, the processor 120 may render the model tool bitmap to the cloud.

At operation 308, the processor 120 may apply the bitmap to improve local rendering.

4 is a diagram showing the configuration of an artificial intelligence model according to an embodiment.

An artificial intelligence model according to an embodiment may include an input layer, a hidden layer, and an output layer.

An input layer is a layer related to input values input to an artificial intelligence model.

In the hidden layer, a feature map may be output by performing a multiply-accumulate (MAC) operation and an activation operation on an input value.

The MAC operation may be an operation of multiplying an input value by a corresponding weight and summing the multiplied values.

The activation operation may be an operation of inputting a result of the MAC operation to an activation function and outputting a resultant value. Activation functions can be of various types. For example, the activation function may include, but is not limited to, a sigmoid function, a tangent function, a relu function, a ricky relu function, a maxout function, and/or an elu function.

The hidden layer may consist of at least one layer. For example, when the hidden layer is composed of a first hidden layer and a second hidden layer, the first hidden layer outputs a feature map by performing a MAC operation and an activation operation based on the input value of the input system, and A feature map, which is a result value of the layer, may be an input value of the second hidden layer. The second hidden layer may perform a MAC operation and an activation operation based on the feature map resulting from the first hidden layer.

An output layer may be a layer related to a result value of an operation performed in a hidden layer.

In one embodiment, the learning model automatically learns the boundaries of compound words and object names by learning syllable (letter) patterns that are frequently combined and used in a given corpus, and object information of the first UI source and object information rendered by the browser Integrating to create an object information file for learning, generating learning data for learning a deep learning network using the object information file for learning, receiving data from various domains of the support system, and corresponding to each of the various domains Based on at least one standardization method, standardizes the data of the various domains in an integrated format, learns and infers data of a specific domain, determines information to be transmitted for the standardization in the data of the specific domain, Post processing may be performed on data from the various domains. The first UI source includes an XML file, and the object information file for learning includes an input JSON file for learning features and an output JSON file that is label data during learning, and the output JSON file is a web standard. and the various domains include at least one of RAN (radio access network), transport, or core, and the post-processing is (correlation) function can be included.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (5)

In electronic devices,
communication module; and
A processor electrically connected to the model intellect and the model tool;
The model carcass is
It consists of a plurality of parts, includes a magnetic object at at least one point connecting the first part and the second part, and is connected using magnetism,
The remaining non-magnetic points are physically connected using at least one of a rubber band, thread, or Velcro, and the first and second portions can be separated or coupled to each other,
The modeling tool is
An electronic control device that provides vibration when a specific point containing a magnetic object on the model land is approached within a specified level,
The processor
Detecting that the model tool approaches at least one point connecting the first part and the second part on the model land within a specified level;
Generating a first vibration on the model tool in response to the approach of the model tool;
Displays information indicating that the model tool has been contacted at the correct point to dismantle the model carcass;
Detecting that the model tool approaches a point corresponding to a bone, which is not at least one point connecting the first part and the second part on the model carcass within a specified level;
In response to the approach of the model tool, a second vibration is generated on the model tool;
An electronic device displaying information indicating that the model tool has been contacted at an inappropriate point to dismantle the model carcass.
According to claim 1,
The processor
Detecting the position of the model tool on the model intellect;
Based on the position of the model tool, the name of a specific part of the model education is displayed on a display,
Provide a control method of the model tool including information on at least one of the angle, depth, position of the model tool or the intensity of force applied to the specific part based on the name of the specific part,
Detecting that the model tool approaches at least one point connecting the first part and the second part on the model land within a specified level;
Maintaining the direction of magnetism and controlling the intensity of magnetism to less than a specified level, providing information indicating that the model is located at the right place to dismantle the model and that an appropriate level of force is being applied,
Detecting that the model tool approaches a point corresponding to a bone, which is not at least one point connecting the first part and the second part on the model carcass within a specified level;
Controlling the model tool to push it by changing the direction of the magnetism to provide information indicating that it is located at an incorrect location to dismantle the carcass,
The intensity of the magnetism is controlled to exceed a specified level to provide information indicating that an inappropriate level of force is being applied,
Setting a designated time to provide a user experience to develop the model education to the user;
Giving a score based on at least one of the angle, depth, position of the model tool or the intensity of force applied to a specific part using an artificial intelligence learning model,
Provide an analysis of why the score is low for users whose scores are below a specified level;
By setting a specified time again, the user is provided with a user experience to develop the model education,
An electronic device that learns the user's learning experience, score, and analysis of problems with the artificial intelligence learning model, and provides an analysis of the wrong bone bone method for each user.
According to claim 1,
The processor
Displaying an actual cognitive education image corresponding to the model cognitive education together with the model cognitive education;
By synchronizing the cognitive education image and the model education,
When the shape of the model education is changed by the model tool, the education image is also changed to a shape corresponding to the shape of the model education,
When the model carcass is cut into a first part and a second part by the model tool,
Displaying information on at least one of intramuscular fat, meat color, fat color, texture, maturity, loin cross-section area, or back fat thickness of the first part;
Displays information on at least one of intramuscular fat map, meat color, fat color, texture, maturity, loin cross-section area, or back fat thickness of the second part,
The modeling tool is
An electronic device manufactured to be more flexible than a specified level or have less strength than a specified level.
According to claim 1,
The processor
Performing a task that includes at least one of a head tracking task, a hand tracking task, an eye tracking task, a scene analysis (eg, scene understanding), or a rendering task;
Acquiring performance data or AR usage data of the electronic device using a sensor while performing an augmented reality service;
Based on the performance data of the electronic device or the AR usage data, performance evaluation corresponding to the AR context is performed, performance evaluation scores are calculated for each AR performance evaluation element,
Analyzing the performance evaluation to provide information about vulnerabilities to users;
Based on the performance evaluation result, create a personalized performance preset in which at least some of the configuration parameters of the basic performance preset are adjusted;
Applying the generated personalized performance preset to the augmented reality service;
The performance data of the electronic device is
GPU clock and CPU load of the electronic device, GPU clock and GPU load of the electronic device, network type, tethering network type between the electronic device and AR device, tethering network signal strength , rendering FPS (frame per second), including at least one of a surface temperature of the electronic device, a limit temperature of the electronic device, and a resolution of the electronic device,
The above AR usage data is
AR device display frequency, AR device operating temperature, AR device refresh rate, AR device screen brightness, AR device threshold temperature, network traffic, tracking frequency, tracking coordinates, tracking latency, Or tracking accuracy, tracking input image quality, scene analysis result, scene analysis accuracy, location information of the electronic device and GPS-based location information of the AR device, error process information in augmented reality service, An electronic device that includes at least one AR app usage time.
According to claim 1,
The processor
Capture the movement of the model tool using a camera,
After analyzing the physical properties related to the movement of the model tool from the captured video frame images, determining the size of the model tool to fit the user's body part;
Render augmented content objects for bones and set interactions,
The model education and the bone augmentation content object are superimposed and displayed in a see through manner,
Determining information about the bones of the model intellect to be provided to the user based on at least one of the position, moving direction, or strength of the moving force of the model tool on the model intellect;
Provides information on the bone of the model carcass using a display,
The display is an electronic device including any one of a Head Mounted Display (HMD) and smart glasses.

Images