TW202240344A - Method of generating user-interactive object, multimedia system, and non-transitory computer-readable medium - Google Patents

Abstract

A method of generating user-interactive object is provided. The method includes the following operations: receiving a picture of a physical environment; identifying a target surface and multiple target objects located on the target surface from the picture to generate an identifying result; generating a virtual surface and multiple virtual three-dimensional (3D) objects located on the virtual surface according to the identifying result, in which the multiple virtual 3D objects are user-interactive objects; and setting multiple operational behaviors of the multiple 3D virtual objects according to a configuration file. The multiple operational behaviors correspond to multiple input operations, respectively. The virtual surface and the multiple virtual objects are for being displayed and manipulated in a virtual environment.

Description

Method for generating user-interactive objects, multimedia system, and non-transitory computer-readable medium

The disclosed document relates to a method for generating an object, a multimedia system and a non-transitory computer-readable medium for realizing the method, especially a method for generating a user-interactive object, a multimedia system for realizing the method, and a non-transitory computer-readable medium media.

Virtual reality (virtual reality, referred to as VR) technology is the use of computers to simulate a fully immersive three-dimensional virtual world. Augmented reality (AR) technology is not fully immersive, but superimposes virtual objects on the real-time images captured by the camera. As long as the user wears the device using the above technology, he can experience various situations personally. Therefore, virtual reality and augmented reality technology can reduce the limitation of venue and money costs on the training content of personnel operating machines, and help to improve the efficiency and flexibility of training courses. However, currently there is no method for efficiently and quickly setting training content for operating machine personnel in virtual reality and augmented reality.

This disclosure document provides a method for generating user-interactive objects, which includes the following processes: receiving an image of a physical environment; identifying a target surface in the image and multiple target objects on the target surface to generate a recognition result; based on the recognition result, establishing A virtual surface and multiple virtual three-dimensional objects located on the virtual surface, wherein the multiple virtual three-dimensional objects are user-interactive (user-interactive) objects; and setting multiple operation outputs of the multiple virtual three-dimensional objects according to the configuration file, among which multiple An operation output corresponds to a plurality of command inputs respectively, and the virtual surface and the plurality of virtual three-dimensional objects are used for displaying and operating in the virtual reality environment.

The disclosed document provides a multimedia system including a storage device, a display device and a computing device. The computing device is coupled to the storage device and the display device, and is used for: receiving an image of the physical environment; identifying a target surface in the image and a plurality of target objects on the target surface to generate a recognition result; according to the recognition result, establishing a virtual surface and A plurality of virtual three-dimensional objects located on the virtual surface, wherein the plurality of virtual three-dimensional objects are user-interactive objects; multiple operation outputs of the plurality of virtual three-dimensional objects are set according to the configuration file, wherein the plurality of operation outputs correspond to a plurality of commands respectively Input, and the virtual surface and multiple virtual three-dimensional objects are used to display and operate in the virtual reality environment.

The present disclosure provides a non-transitory computer-readable medium containing one or more computer-executable instructions for controlling a multimedia system. The multimedia system includes a storage device, a display device and a computing device. When the computing device executes one or more computer-executable instructions, the computing device performs the following operations: receiving an image of the physical environment; identifying a target surface in the image and multiple target objects on the target surface to generate a recognition result; according to the recognition result , create a virtual surface and a plurality of virtual 3D objects located on the virtual surface, wherein the plurality of virtual 3D objects are user-interactive objects; set a plurality of operation outputs of the plurality of virtual 3D objects according to the configuration file, wherein the plurality of operation outputs are respectively Corresponding to multiple command inputs, the virtual surface and multiple virtual three-dimensional objects are used to display and operate in the virtual reality environment.

One of the advantages of the above-mentioned embodiment is that customized virtual training content can be quickly generated.

Another advantage of the above-described embodiment is that it enables the trainee to experience the situation himself.

Embodiments of the disclosed document will be described below in conjunction with related figures. In the drawings, the same reference numerals represent the same or similar elements or method flows.

FIG. 1 is a simplified diagram of a multimedia system 100 according to an embodiment of the disclosure. The multimedia system 100 includes a display device 110 and a computing device 120 . The display device 110 is used for displaying virtual reality (virtual reality, referred to as VR) images or augmented reality (augmented reality, referred to as AR) images. In some embodiments, the display device 110 can also play audio and/or provide vibration tactile feedback. In practice, the display device 110 can be realized by a head-mounted device.

The computing device 120 is coupled to the display device 110, and the computing device 120 is used to transmit display data (such as RGB signals, vertical synchronization signals and horizontal synchronization signals) to the display device 110 to control the display content of the display device 110. The display device 110 is, for example, PC VR device. In some embodiments, the computing device 120 may be implemented by a personal computer, a server, a notebook computer or other programmable electronic devices. In some embodiments, the display device 110 has the function of the computing device 120 at the same time, that is, the display device 110 and the computing device 120 are integrated into an all-in-one type, and there is no need to connect an external computing device 120 by wire or wirelessly, such as an all-in-one computer VR (Standalone VR) device.

The computing device 120 can be connected to the electronic device 103 through network communication to receive the image 105 obtained by the electronic device 103 shooting the physical environment, but this disclosure is not limited thereto. In some embodiments, the electronic device 103 can transfer the image 105 to the computing device 120 through an external memory (such as a memory card or a flash drive). In practice, the electronic device 103 can be realized by a smart phone or a digital camera. In some embodiments, the electronic device 103 may be implemented with a camera on the display device 110 .

In some embodiments, the electronic device 103 is used to photograph the operation panel of the machine 107 in the physical environment, and transmit the obtained image 105 to the computing device 120 through the network. The computing device 120 is used to extract the appearance of the operation panel from the image 105 , and the computing device 120 creates a user-interactive virtual operation panel according to the appearance of the operation panel. In some embodiments, the aforementioned virtual operation panel can be displayed by the display device 110 in a virtual reality environment. That is, the computing device 120 can recognize a specific object from the received image, and automatically generate a corresponding user-interactable virtual object according to the recognition result.

FIG. 2 is a flow chart of a method 200 for generating user-interactive objects according to an embodiment of the disclosed document. One or more computer-executable instructions (not shown) are stored in the memory 122 of the computer device 120 . When the computing device 120 executes the computer-executable instructions, the computer-executable instructions cause the computing device 120 to execute the method 200 for generating user-interactive objects. In some embodiments, one or more setting files 124 are also stored in the memory 122, and the computer device 120 is used to set multiple operation outputs of the virtual object (such as generating light, sound, picture, vibration, etc.) according to the setting files 124, The detailed setting method will be described in the following paragraphs.

In the process S210, the computing device 120 receives an image 105 of a physical environment, such as the image 105 obtained by photographing the physical environment. FIG. 3 is an enlarged schematic view of the image 105 . In this embodiment, the image 105 includes a control panel 300 , and the control panel 300 includes a surface 310 and a plurality of objects 320 on the surface 310 . The plurality of objects 320 can be various movable elements or indicating elements on the control panel 300, such as a plurality of knobs 322, a plurality of square buttons 324, round buttons 326 and indicator lights 328, and for example levers, pull rods, joysticks, etc. ( not shown in the figure).

In the process S220 , the computing device 120 is configured to recognize the control panel 300 from the image 105 . In detail, the computing device 120 can recognize the shape, contour, scale, text and other information of the surface 310 and the objects 320 , and the computing device 120 can also recognize the position and arrangement of the objects 320 on the surface 310 .

In the process S230 , the computing device 120 generates depth information representing the height of each object 320 on the surface 310 . For example, the computing device 120 can analyze the image 105 through a suitable algorithm to obtain depth information. For another example, the computing device 120 may additionally receive (eg, from the electronic device 103 ) an image of different viewing angles of the control panel 300 to calculate the depth information by using the disparity between the two images. Relevant calculation methods are well known to those with ordinary knowledge in the technical field, and will not be repeated here. In some embodiments, the height of each object 320 on the surface 310 can also be a system default value, or can be set or adjusted by the user.

In the process S240, please refer to FIG. 3 and FIG. 4 at the same time. The computing device 120 generates a plurality of two-dimensional geometries according to the contours of the surface 310 and the plurality of objects 320, and according to the arrangement of the plurality of objects 320 on the surface 310. Graphics 410~418. In some implementations, the contour of the geometric figure 410 is the same as or corresponds to the contour of the surface 310, and the contours of the geometric figures 412-418 are the same as or correspond to the contours of the knob 312, the square button 314, the round button 316 and the indicator light 318, respectively. . In other implementations, the arrangement of the geometric figures 412 - 416 in the geometric figure 410 is the same or corresponds to the arrangement of the knob 312 , the square button 314 , the round button 316 and the indicator light 318 on the surface 310 . In some embodiments, the computing device 120 stores the geometric figures 410-418 as vector graphics.

In the process S250, please refer to FIG. 4 and FIG. 5 at the same time. The computing device 120 generates the virtual content 500 according to the geometric figures 410-418 and the depth information, wherein the geometric figures 410-418 are used to determine the outline or shape of the virtual content 500, and The depth information is used to determine the height or thickness of the virtual content 500 . Specifically, the virtual content 500 includes a virtual surface 510 and a plurality of virtual three-dimensional objects 522 - 528 . The computing device 120 sets the contour of the virtual surface 510 to be the same as or corresponding to the contour of the geometric figure 410 , and sets the contours of the virtual 3D objects 522 - 528 to be the same or correspond to the contours of the geometric figures 412 - 416 . The computing device 120 also determines the heights of the virtual three-dimensional objects 522-528 on the virtual surface 510 according to the depth information, and sets the arrangement of the virtual three-dimensional objects 522-528 on the virtual surface 510 to be the same or correspond to the geometric figures 412-418 on the virtual surface 510. Arrangement in geometry figure 410 . In some embodiments, the heights of the virtual three-dimensional objects 522-528 on the virtual surface 510 can also be a system default value, or can be set or adjusted by the user.

In summary, in the process S210-S250, the multimedia system 100 recognizes the surface 310 and the plurality of objects 320 in the image 105, and generates the virtual content 500 having the same or similar appearance as the surface 310 and the plurality of objects 320.

Next, in the process S260-270, the computing device 120 sets the user interaction function of the virtual content 500. In the process S260, the computing device 120 sets the types of the virtual three-dimensional objects 522-528 according to the contours of the plurality of objects 320, wherein when the virtual three-dimensional objects 522-528 respond to user input, different types of virtual three-dimensional objects will have different motions Types of. The virtual three-dimensional objects 522-528 can form virtual movable elements such as virtual knobs, virtual buttons, virtual levers, virtual pull rods, virtual joysticks, etc. through the aforementioned different motion types.

For example, please refer to FIG. 3 and FIG. 5 at the same time, the virtual 3D object 524 is generated according to the object 314 with a square outline, so the computing device 120 sets the virtual 3D object 524 as a virtual button. For another example, the virtual three-dimensional object 522 is generated according to the object 312 having a circular outline, so the computing device 120 sets the virtual three-dimensional object 522 as a virtual knob. For another example, the virtual three-dimensional object 526 is generated according to the object 316, and the object 316 has a circular edge but a non-circular outline. Therefore, the computing device 120 sets the virtual three-dimensional object 526 as a virtual button. However, the disclosed document is not limited thereto.

Therefore, in an embodiment as shown in FIG. 6, when the display device 110 displays the virtual content 500 in the virtual reality environment 600, the virtual three-dimensional objects 524 and 526 will respond to the user input 610 and move perpendicular to the virtual surface. Move back and forth in the direction of 510 to simulate a button. In addition, the virtual three-dimensional object 522 generates a rotational motion in response to the user input 610 to simulate a knob. In a word, the movement types of the virtual three-dimensional objects 522 - 528 correspond to the contours of one of the plurality of objects 320 .

In some other embodiments, the computing device 120 sets motion events generated when the virtual three-dimensional objects 522 - 528 respond to user input according to one or more symbols or characters on or around the target object 320 . For example, please refer to FIG. 3 and FIG. 5 at the same time. The virtual three-dimensional object 522 is generated according to the knob 322, and the computing device 120 recognizes that there are multiple scale marks around the knob 322, so the computing device 120 sets the virtual three-dimensional object 522 to It is a virtual knob, and the scale symbol can be displayed at the corresponding position of the virtual three-dimensional object 522 at the same time, for example, displayed around the virtual three-dimensional object 522 .

As another example, the virtual three-dimensional object 524 is generated according to the square button 324, and the computing device 120 recognizes that the square button 324 has identifiable characters "-X", "-Y", "F1" and "F2", etc., thus The computing device 120 sets the virtual three-dimensional object 524 as a square virtual button, and simultaneously displays text on a corresponding position of the virtual three-dimensional object 324 , such as the upper surface of the virtual three-dimensional object 324 .

For another example, the virtual three-dimensional object 526 is generated according to the circular button 326, and the computing device 120 recognizes that the circular button 326 is surrounded by recognizable characters "EMERGENCY STOP", so the computing device 120 sets the virtual three-dimensional object 526 as a circle A virtual button can be formed, and the identification text can be displayed on the corresponding position of the virtual three-dimensional object 526, such as the upper surface of the virtual three-dimensional object 526. However, the disclosed document is not limited thereto.

In some embodiments, the computing device 120 can set the user interaction functions of the virtual three-dimensional objects 522 - 528 according to the outline of the object 320 and surrounding characters or symbols. In this way, it is convenient for the user to set the interactive function, and the probability of the computing device 120 misjudging the type of the object 320 is reduced.

Next, in the process S270 , the computing device 120 sets multiple operation outputs of the virtual content 500 according to the setting file 124 to simulate various reactions of the machine 107 under real operation conditions. In some embodiments, the operation output refers to the corresponding predetermined light, predetermined sound, predetermined screen, predetermined vibration, predetermined virtual machine action or two or more of these generated when the virtual content 500 receives an input operation from the user. A combination of these, wherein the command input can be composed of the aforementioned one or more user inputs 610, and a plurality of different operation outputs can correspond to a plurality of different command inputs respectively. For example, please refer to FIG. 6, when the user presses a plurality of virtual three-dimensional objects 524 in a specific order to generate command input, the virtual three-dimensional object 528 generated according to the indicator light 328 will generate predetermined light and other prompts to remind the user whether the operation correct. For another example, when the user rotates the virtual three-dimensional object 522 to generate a command input, and the virtual three-dimensional object 522 rotates beyond a predetermined angle or fails to reach the predetermined angle, the virtual content 500 will cause the display device 110 to play a predetermined sound effect to prompt the user. However, the disclosed document is not limited thereto.

In some embodiments, the computing device 120 can identify the model of the machine 107 according to the image 105 , and then automatically select one of multiple candidate setting files stored in advance as the setting file 124 according to the model of the machine 107 . In other embodiments, the user can manually select one of multiple candidate configuration files as the configuration file 124, or the user can input the configuration file on the computing device 120 through the input interface (not shown) of the computing device 120. A configuration file 124 is written and saved.

In some embodiments, the computing device 120 may provide the configured virtual content 500 to the display device 110 after the process S270 ends. The display device 110 displays the virtual content 500 in a virtual reality environment (such as the virtual reality environment 600 ). Therefore, the user of the display device 110 can operate a plurality of virtual three-dimensional objects 522-528 in the virtual reality environment.

To sum up, the method 200 for generating user-interactive objects can quickly generate customized virtual training content, which helps reduce the time and money costs for enterprises to set up virtual training content and train employees. When the method 200 for generating user-interactive objects is used in combination with the multimedia system 100, the trainee can experience the training content personally, which is helpful to improve the training effect.

The execution sequence of the above-mentioned processes in FIG. 2 is only an exemplary embodiment, rather than limiting the actual implementation of the disclosed document. For example, in FIG. 2 , the process 230 and the process 240 can be performed simultaneously, or the order can be reversed.

Certain terms are used in the specification and claims to refer to particular elements. However, those skilled in the art should understand that the same element may be called by different terms. The description and the scope of the patent application do not use the difference in the name as the way to distinguish the components, but the difference in the function of the components as the basis for the distinction. The term "comprising" mentioned in the specification and scope of patent application is an open term, so it should be interpreted as "including but not limited to". In addition, "coupling" here includes any direct and indirect connection means. Therefore, if it is described that the first element is coupled to the second element, it means that the first element can be directly connected to the second element through electrical connection or signal connection means such as wireless transmission or optical transmission, or through other elements or connections. The means are indirectly electrically or signally connected to the second element.

The description of "and/or" used herein includes any combination of one or more of the listed items. In addition, unless otherwise specified in the specification, any singular term also includes plural meanings.

The above are only preferred embodiments of this disclosure document, and all equivalent changes and modifications made according to the requirements of this disclosure document shall fall within the scope of this disclosure document.

100:Multimedia system 103: Electronic device 105: Image 107: machine 110: display device 120: Computing device 122: Memory circuit 124: Setting file 200: User-interactive object generation method S210~S270: process 300: control panel 310: target surface 320: target object 322: Knob 324: square button 326: round button 410~418: Geometry 500: virtual content 510: virtual surface 522~528: Virtual three-dimensional objects 600: virtual environment 610: Virtual environment

FIG. 1 is a simplified diagram of a multimedia system according to an embodiment of the disclosure. FIG. 2 is a flow chart of a method for generating user-interactive objects according to an embodiment of the disclosed document. Fig. 3 is an enlarged schematic view of the image in Fig. 1 . FIG. 4 shows a schematic diagram of multiple 2D geometric figures generated from the image in FIG. 1 . FIG. 5 shows a schematic diagram of virtual content generated according to the image in FIG. 1 . Fig. 6 shows part of the virtual environment in the user's view.

100:Multimedia system

103: Electronic device

105: Image

107: machine

110: display device

120: Computing device

122: Memory circuit

124: Setting file

Claims (20)

A method for generating user-interactive objects, comprising: receiving an image of a physical environment; identifying a target surface and a plurality of target objects on the target surface in the image to generate an identification result; Establishing a virtual surface and a plurality of virtual three-dimensional objects located on the virtual surface according to the identification result, wherein the plurality of virtual three-dimensional objects are user-interactive objects; and A plurality of operation outputs of the plurality of virtual three-dimensional objects are set according to a setting file, wherein the plurality of operation outputs correspond to a plurality of command inputs respectively, and the virtual surface and the plurality of virtual three-dimensional objects are used for displaying and operating on a virtual realistic environment. The method according to claim 1, wherein each of the plurality of operating outputs includes one or more of the following: a predetermined light, a predetermined sound, a predetermined image, a predetermined vibration, and a predetermined virtual machine action. The method as described in claim 1, wherein setting the plurality of operation outputs of the plurality of virtual three-dimensional objects according to the configuration file includes: Selecting a corresponding one from multiple candidate configuration files as the configuration file according to the identification result. The method according to claim 1, wherein when a first virtual three-dimensional object among the plurality of virtual three-dimensional objects responds to a user input, the first virtual three-dimensional object has a motion type, and the motion type corresponds to An outline corresponding to one of the plurality of target objects, or corresponding to one or more characters or symbols on or around the corresponding one of the plurality of target objects. The method as described in claim 4, wherein, if the contour of the corresponding one of the plurality of target objects is cylindrical, the first virtual three-dimensional object forms a virtual knob or a virtual button, If the outline shape of the corresponding one of the plurality of target objects is a square column, the first virtual three-dimensional object forms the virtual button. The method as described in claim 4, wherein, if the one or more characters or symbols include multiple scales, the first virtual three-dimensional object forms a virtual knob, If the one or more characters or symbols include one or more recognizable characters, the first virtual three-dimensional object forms a virtual button. The method as described in Claim 1, wherein creating a virtual surface and a plurality of virtual three-dimensional objects located on the virtual surface comprises: generating a plurality of two-dimensional geometric figures according to a plurality of outlines and an arrangement of the plurality of target objects; generating a depth information of the plurality of target objects according to the image; determining a plurality of shapes of the plurality of virtual three-dimensional objects according to the plurality of two-dimensional geometric figures; and The heights of the virtual three-dimensional objects on the virtual surface are determined according to the depth information. A multimedia system comprising: a storage device for storing a configuration file; a display device; and A computing device, coupled to the storage device and the display device, and used for: receiving an image of a physical environment; identifying a target surface and a plurality of target objects on the target surface in the image to generate an identification result; Establishing a virtual surface and a plurality of virtual three-dimensional objects located on the virtual surface according to the identification result, wherein the plurality of virtual three-dimensional objects are user-interactive objects; Setting a plurality of operation outputs of the plurality of virtual three-dimensional objects according to a setting file, wherein the plurality of operation outputs correspond to a plurality of command inputs respectively, and the virtual surface and the plurality of virtual three-dimensional objects are used for display and operation on one virtual reality environment. The multimedia system as described in claim 8, wherein each of the plurality of operation outputs includes one or more of the following: a predetermined light, a predetermined sound, a predetermined picture, a predetermined vibration, a predetermined virtual machine action, or A combination of two or more of these. The multimedia system as described in Claim 8, wherein when the computing device configures the plurality of operation outputs of the plurality of virtual three-dimensional objects according to the configuration file, the computing device is used for: Selecting a corresponding one from multiple candidate configuration files as the configuration file according to the identification result. The multimedia system as described in claim 8, wherein, when a first virtual three-dimensional object among the plurality of virtual three-dimensional objects responds to a user input, the first virtual three-dimensional object has a motion type, and the motion type An outline corresponding to a corresponding one of the plurality of target objects, or corresponding to one or more characters or symbols on or around the corresponding one of the plurality of target objects. The multimedia system as described in claim 11, wherein, if the contour of the corresponding one of the plurality of target objects is cylindrical, the first virtual three-dimensional object forms a virtual knob or a virtual button, If the outline shape of the corresponding one of the plurality of target objects is a square column, the first virtual three-dimensional object forms the virtual button. The multimedia system as described in claim 11, wherein, if the one or more characters or symbols include multiple scales, the virtual three-dimensional object forms a virtual knob, If the one or more characters or symbols include one or more recognizable characters, the virtual three-dimensional object forms a virtual button. The multimedia system as described in claim 8, wherein, when the computing device creates the virtual surface and the plurality of virtual three-dimensional objects located on the virtual surface, the computing device is used for: generating a plurality of two-dimensional geometric figures according to a plurality of outlines and an arrangement of the plurality of target objects; generating a depth information of the plurality of target objects according to the image; determining a plurality of shapes of the plurality of virtual three-dimensional objects according to the plurality of two-dimensional geometric figures; and The heights of the virtual three-dimensional objects on the virtual surface are determined according to the depth information. A non-transitory computer-readable medium comprising one or more computer-executable instructions for controlling a multimedia system, wherein the multimedia system comprises a storage device, a display device, and a computing device, and when the computing device executes When the one or more computers are executable instructions, the computing device performs the following operations: receiving an image of a physical environment; identifying a target surface and a plurality of target objects on the target surface in the image to generate an identification result; Establishing a virtual surface and a plurality of virtual three-dimensional objects located on the virtual surface according to the identification result, wherein the plurality of virtual three-dimensional objects are user-interactive objects; Setting a plurality of operation outputs of the plurality of virtual three-dimensional objects according to a setting file, wherein the plurality of operation outputs correspond to a plurality of command inputs respectively, and the virtual surface and the plurality of virtual three-dimensional objects are used for display and operation on one virtual reality environment. The non-transitory computer-readable medium as described in Claim 15, wherein each of the plurality of operational outputs includes one or more of the following: a predetermined light, a predetermined sound, a predetermined picture, a predetermined vibration, A predetermined virtual machine action or a combination of two or more of them. The non-transitory computer-readable medium as described in claim 15, wherein setting the plurality of behaviors of the plurality of virtual three-dimensional objects according to the configuration file includes: Selecting a corresponding one from multiple candidate configuration files in the storage device as the configuration file according to the identification result. The non-transitory computer readable medium of claim 15, wherein when a first virtual three-dimensional object among the plurality of virtual three-dimensional objects responds to a user input, the first virtual three-dimensional object has a motion type, and the motion type corresponds to a contour of a corresponding one of the plurality of target objects, or corresponds to one or more characters or symbols on or around the corresponding one of the plurality of target objects. The non-transitory computer readable medium as described in claim 18, wherein if the one or more characters or symbols include multiple scales, the first virtual three-dimensional object forms a virtual knob, If the one or more characters or symbols include one or more recognizable characters, the first virtual three-dimensional object forms a virtual button. The non-transitory computer-readable medium as described in Claim 15, wherein creating a virtual surface and a plurality of virtual three-dimensional objects located on the virtual surface comprises: generating a plurality of two-dimensional geometric figures according to a plurality of outlines and an arrangement of the plurality of target objects; generating a depth information of the plurality of target objects according to the image; determining a plurality of shapes of the plurality of virtual three-dimensional objects according to the plurality of two-dimensional geometric figures; and The heights of the virtual three-dimensional objects on the virtual surface are determined according to the depth information.

Images