TWI747294B - Remote virtual scene building system, method and non-transitory computer-readable storage medium - Google Patents

Abstract

The present invention provides a remote virtual scene building system comprising a first optical label device, a second optical label device, and an optical label recognition device. The first optical label device comprises a first optical label which is corresponding to a device posture-position information. The second optical label device comprises a second optical label. The optical label recognition device comprises a camera unit, a display, and a processor. The optical label recognition device captures the image of the second optical label by the camera unit and obtains a posture information via the processor. The processor constructs a scene information corresponding to the device posture-position information of the first optical label device according to the posture information and the user’s chosen first optical label device, and outputs the scene information to the display to provide a virtual object placement function.

Description

System and method for remotely constructing virtual scene and non-temporary computer readable recording medium

The present invention relates to a system and method for constructing a virtual scene, and a non-transitory computer-readable recording medium, in particular to a system and method for remotely constructing a virtual scene, and a non-transitory computer-readable recording medium.

Augmented Reality (Augmented Reality, AR), also called virtual reality or expanded reality, refers to the position and angle of the camera image through the actuarial calculation and image analysis technology, so that the virtual world on the screen can be compared with The technology of combining and interacting with real world scenes.

When discussing the virtual world purely, all objects in the virtual world (for example: sky, ground, people, animals, tables and chairs, high-rise buildings, etc.) can be called virtual objects. The key to the augmented reality system is how to combine the virtual objects in the virtual world with the actual environment. Therefore, the augmented reality algorithm software must obtain the coordinates of the real world, and then superimpose the virtual objects on the coordinates.

In order to obtain real-world coordinates and establish accurate coordinate positions, it is usually necessary for a dedicated photographer to use a dedicated camera to take pictures and go to the real-world environment to obtain corresponding images. Only then can the real-world accurate coordinate positions be obtained and added. Combining virtual objects has met the needs of augmented reality.

The invention provides a system for remotely constructing a virtual scene, which includes at least a first optical label device, a second optical label device, and an optical label identification device. The first optical tag device has a first optical tag, and the first optical tag at least corresponds to a device pose information. The second optical label device has a second optical label. The optical label identification device includes a photographing unit, a display, and a processor. The optical label identification device photographs the image of the second optical label through the photographing unit and calculates the position information by the processor, and according to The pose information and the first optical tag device selected by the user establish scene information relative to the device pose information of the first optical tag device, display the scene information on the display, and provide a virtual object setting function.

The present invention also provides a method for remotely constructing a virtual scene, which includes a photographing unit of an optical label recognition device to take an image of a second optical label device. The image is calculated by a processor of the optical tag recognition device to obtain position information. The processor creates scene information relative to a device pose information of the first optical tag device based on the pose information and a first optical tag device selected by the user. The scene information is displayed on a display of the optical label identification device. And the processor sets the virtual object in the scene information.

The present invention also provides a non-transitory computer-readable recording medium, which is used to store a program. When the program is loaded into a processing chip, the aforementioned method can be executed.

Therefore, compared with the conventional technology, the present invention does not need to personally go to the real environment to obtain the image, and can obtain the position in the three-dimensional space through the light tag and set the virtual object at the remote end.

The detailed description and technical content of the present invention will now be described in conjunction with the drawings as follows. Furthermore, for the convenience of description, the figures in the present invention are not necessarily drawn according to actual proportions. These figures and their proportions are not intended to limit the scope of the present invention, and are described here first.

Please refer to "Figure 1" below, which is a block diagram of the system for remotely constructing virtual scenes according to the present invention, as shown in the figure:

This embodiment provides a system 100 for remotely constructing a virtual scene, which mainly includes at least one first optical label device 10, a second optical label device 20 having a second optical label, and an optical label identification device 30. The dotted line shown in "FIG. 1" refers to the photographing relationship of the optical label identification device 30 to the second optical label device 20, which is described here first.

The first optical tag device 10 has a first optical tag, and the first optical tag at least corresponds to the pose information of the device.

The aforementioned first optical label device 10 and second optical label device 20 are devices that can transmit information through different light emitting methods. Unlike traditional two-dimensional codes, the optical label has a long recognition distance, strong directivity, and is not subject to visible light conditions. Therefore, the optical tag can provide a longer recognition distance and a stronger information exchange capability. Generally, the optical label usually includes a controller or at least one light source, and the controller can drive the light source in different modes, so that the optical label can transmit different information to the outside. Each optical tag can be assigned a piece of label information as the identification information of the optical tag, and the device pose information of the optical tag can be obtained through the identification information. It should be noted here that although only two sets of optical label devices (the first optical label device 10 and the second optical label device 20) are disclosed in this embodiment, three optical label devices can be selected in practice. Or more than three, the user can select the corresponding optical label device through the service provided by the application, and edit based on the scene information corresponding to the optical label device, which is not limited in the present invention.

The aforementioned device pose information includes position information (coordinates) and posture information (the posture information refers, for example, to the orientation of the light tag, for example, toward true north). The aforementioned posture information refers to the orientation information of the first optical label device 10 and the second optical label device 20 in a certain coordinate system (for example, a world coordinate system or an optical label coordinate system). When the first optical label device 10 and the second optical label device 20 are translated without rotating, the position information (coordinates) changes, but the posture information of the first optical label device 10 and the second optical label device 20 will remain unchanged. When the first optical label device 10 and the second optical label device 20 only rotate without translation, the position information of the first optical label device 10 and the second optical label device 20 remain unchanged, but the posture information will change.

In another embodiment, the aforementioned location information may be, in a feasible embodiment, the address information of the device itself in world coordinates (for example, World Geodetic System (WGS), latitude and longitude coordinate system, etc.) . In other feasible embodiments, the device location information can also be a spatial coordinate system pre-established based on the user setting a relatively stable anchor point, or any other spatial coordinate information that can be used as an absolute or relative position reference. In the present invention There is no restriction in it.

Please refer to “FIG. 2”. The optical label identification device 30 includes a photographing unit 32, a display 34, and a processor 36. The optical label recognition device 30 can be (but is not limited to) a mobile phone (Smart Phone), a tablet computer (Tablet), smart glasses (Smart Glasses), a wearable device (Wearable Devices), etc., which have shooting functions, display functions, and computing functions. Or other devices with sensors and camera lenses, display screens, computing processors and networking functions that can transmit the captured images and/or their information to other devices (such as servers) via the Internet. The optical tag identifies The choice of the device 30 is not limited in the present invention. In one embodiment, the processor 36 described in the present invention may further cooperate with a storage unit to execute, store a program through the storage unit, and thereby perform corresponding functions by loading the storage unit; in another feasible embodiment In this case, the processor 36 can be co-constructed with the storage unit to be implemented as a single chip, which is not limited in the present invention. The storage unit can be, but is not limited to, cache memory, dynamic random access memory (DRAM), persistent memory, etc., which can be used as a device for storing and fetching data. The combination is not limited in the present invention.

In one of the embodiments, the present invention can be used with a server having a storage unit. The server (Server) includes a central processing unit, a hard disk, a memory (storage unit), etc., and the hardware cooperates to execute the corresponding software (Software) to realize the functions and algorithms described in the present invention The cooperative relationship between the software and hardware on the electrical signal is not within the scope of the present invention. In one of the embodiments of the present invention, part of the program can be executed by the optical label identification device 30 or the server, which is not limited in the present invention.

The hardware architecture of the present invention has been roughly described above. The algorithm and functions executed by the hardware in conjunction with the present invention will be further described later. Please also refer to "Figure 3" to disclose the remote end of the present invention. The flow diagram of the method of setting virtual objects is shown in the figure:

The scene information for each optical label device (for example, the first optical label device 10, the second optical label device 20) is pre-stored in the optical label in the form of a database according to the corresponding optical label code (or coordinate position) as an index The identification device 30 may be a server connected or coupled to the optical label identification device 30.

When the optical label recognition device 30 starts the service provided by the application, first, the user operates the optical label recognition device 30, and takes a picture of the second optical label on the second optical label device 20 through the photographing unit 32 of the optical label recognition device 30 Image (step S201).

Subsequently, after obtaining the image of the second optical label, the processor 36 obtains the pose information of the optical label identification device 30 through the image calculation (step S202).

The pose information (including position information and posture information) refers to the relative positional relationship and relative posture of the optical tag identification device 30 and the second optical tag device 20.

In one embodiment, the posture information (position information and posture information) of the optical tag identification device 30 relative to the second optical tag device 20 can be determined by the following method. First, a coordinate system is established based on the optical label of the second optical label device 20, and the coordinate system may be called an optical label coordinate system. Some points on the optical label may be determined as some spatial points in the optical label coordinate system, and the coordinates of these spatial points in the optical label coordinate system may be determined according to the physical size information and/or physical shape information of the optical label. Some points on the optical label may be, for example, the corners of the housing of the optical label, the end of the light source in the optical label, some identification points in the optical label, and so on. According to the physical structure feature or geometric structure feature of the optical tag, the image points corresponding to these spatial points can be found in the image taken by the photographing unit 32, and the position of each image point in the image can be determined. According to the coordinates of each spatial point in the optical label coordinate system and the position of each corresponding image point in the image, combined with the internal reference information of the optical label recognition device 30, it can be calculated that the optical label recognition device 30 is in the optical label when the image is taken. The pose information (R, t) in the coordinate system, where R is the rotation matrix, which can be used to represent the posture information (also called orientation information) of the optical tag recognition device 30 in the optical tag coordinate system, and t is the displacement The vector can be used to represent the position information of the optical label identification device 30 in the optical label coordinate system. The aforementioned method of calculating R and t can be calculated using known existing technology, for example, the PnP (Perspective-n-Point) method used in 3D-2D technology to calculate R and t.

Subsequently, after obtaining the pose information, the processor 36 further establishes scene information relative to the device pose information of the first optical tag device 10 according to the pose information and the first optical tag device 10 selected by the user (step S203).

Specifically, after the processor 36 has obtained the pose information, the user can use the software built into the processor 36 to select the required first optical label device 10, and the processor 36 serves as an index from the database according to the user's selection. Find the scene information corresponding to the first optical tag device 10, and anchor it to the coordinates corresponding to the first optical tag device 10 according to the acquired pose information of the optical tag identification device 30 and the second optical tag device 20, The user's corresponding position and posture in the space of the first optical label device 10 are set virtually, and the user's scene information is established based on the information.

的第二光標籤裝置20A，並運算光標籤識別裝置30A與第二光標籤裝置20A的向量

、roll 角(

)（圖未示）、pitch 角(

)（圖未示）、yaw 角(

)（圖未示）後錨定至第一光標籤裝置10A所對應的坐標

。所述的向量

與

之間

軸坐標變化的純量為

，

軸坐標變化的純量為

，

軸坐標變化的純量為

。所述的roll 角(

)為在yz平面逆時針的旋轉方向。所述的pitch 角(

)為在zx平面逆時針的旋轉方向。yaw 角(

)為在xy平面逆時針的旋轉方向。所述的第一光標籤裝置10A於圖4中為虛線，該虛線係指對於用戶P來說，第一光標籤裝置10A並非實際存在用戶P周圍。 The following lists a specific embodiment for the foregoing content, please refer to "FIG. 4". The user P holds the optical tag recognition device 30A, and photographs with coordinates

The second optical label device 20A, and calculate the vector of the optical label identification device 30A and the second optical label device 20A

, Roll angle (

) (Not shown), pitch angle (

) (Not shown), yaw angle (

) (Not shown) and anchored to the coordinates corresponding to the first optical label device 10A

. Said vector

and

between

The scalar of the axis coordinate change is

,

The scalar of the axis coordinate change is

,

The scalar of the axis coordinate change is

. The roll angle (

) Is the counterclockwise rotation direction in the yz plane. The pitch angle (

) Is the counterclockwise direction of rotation in the zx plane. yaw angle (

) Is the counterclockwise direction of rotation in the xy plane. The first optical label device 10A is shown as a dashed line in FIG.

In addition to the above methods, the scene information of the first optical label device 10 can also be downloaded from the server by the processor 36 via the Internet, local area network or physical connection connected to the server, which is not limited in the present invention. The medium where the scene information is stored. In addition to the three-dimensional information corresponding to the scene space where the optical label device is located, the scene information described therein may also include a virtual object that has been created in advance in conjunction with the optical label device. The virtual objects described here are, for example, 3D models, 2D plane pictures, transparent masked videos, videos or virtual text messages, etc., which are not limited in the present invention.

Furthermore, in a preferred embodiment, the scene information around the first optical label device 10 includes a plurality of scene information corresponding to individual time points. The foregoing includes a plurality of scene information corresponding to individual time points, please refer to "Figure 5". For example, the scene information includes scene information at nine o'clock in the morning (see Figure 5 (A)) and scene information at eight o'clock in the evening (see Figure 5). 5(B)). The time point of the aforementioned scene information is not within the scope of the present invention.

In other embodiments, when the scene information surrounding the first optical label device 10 includes a plurality of scene information corresponding to individual time points, the processor 36 can create the scene information corresponding to the time point according to the time point selected by the user P. In one of the feasible application embodiments, for example, the information of different stores can be displayed according to the business hours of the store, or the ambient brightness (day or night) can be established according to the time point, as shown in "Figure 6", Figure 6(A) is Scene information under the business conditions of the restaurant and the bank at nine o'clock in the morning; Figure 6(B) shows the scene information when the restaurant turns into a bar at eight o'clock in the evening, and the bank is not open at night. The aforementioned scene information is created. It is not within the scope of the present invention. Among them, the diagonal line in the figure refers to the night sky.

After the processor 36 confirms the scene information, the processor 36 displays the scene information on the display 34 of the optical label identification device 30 (step S204).

In this step, the scene information constructed by the processor 36 is based on the device pose information of the first light tag device 10, and the image displayed on the display 34 will correspond to the scene information of the first light tag device 10 at this time, and The orientation and distance of the user relative to the first optical label device 10 seen by the user P in the display 34 are the same as the orientation and distance of the optical label identification device 30 and the second optical label device 20.

In other embodiments, the database may further include a plurality of scene information corresponding to individual time points, and the user P can select the corresponding information according to the required time point (or according to the current time point) through the service provided by the application The scene information at the time point is also displayed on the display 34 of the optical label recognition device 30. As shown in "Figure 7", when the optical tag recognition device 30 is a smart phone 30B and the user selects the scene information at 8 pm, the display of the smart phone 30B displays the scene information at 8 pm selected by the user P (Figure 6(B)).

Finally, the processor 36 sets the virtual object in the scene information according to the setting (step S205).

In this step, the function of the processor 36 to set the virtual object is a scene creation tool executed by the application program provided by the processor 36. The scene creation tool can, for example, calibrate the coordinates in a three-dimensional space, and according to the calibration Coordinates create three-dimensional objects (such as extended reality objects, dialog boxes, images, etc.), and modify scene information by creating three-dimensional objects. The edited scene information can be stored in the storage unit of the optical label identification device 30 or directly uploaded to the server to update the scene information, so that a third party can access the server and download the edited scene information.

In this embodiment, the processor 36 has software that can convert the scene information displayed on the display 34 into a virtual object, so that the user P can set a virtual object to the scene information. The said software includes (but is not limited to) ARKit, ARCore, Unity, Vufori, HP Reveal, MAKAR and other software development kits (Software Development Kit, SDK) or other algorithm software that can combine virtual objects with the actual environment. There is no limitation in the invention.

In other embodiments, the processor 36 can set the scene information that already has the virtual object displayed on the display 34, so that the user P can adjust the virtual object in the scene information. The processor 36 has software capable of adjusting the virtual object, and the software is the same as the software for setting the virtual object, so it will not be repeated here.

The method of the present invention can also be recorded on a computer-readable recording medium. The "computer-readable recording medium" includes (but is not limited to) portable or non-portable storage devices, optical storage devices, and capable of storing and containing Or various other media that carry instructions and/or information. The computer-readable recording medium may include a non-transitory medium in which data can be stored and does not include carrier waves and/or temporary electronic signals transmitted wirelessly or via wired connections.

Examples of non-transitory media may include (but are not limited to) magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices.

The computer-readable recording medium may have codes and/or machine-executable instructions stored thereon, and these codes and/or machine-executable instructions may represent programs, functions, subroutines, programs, routines, subroutines, Modules, software packages, categories, or any combination of commands, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuit by transmitting and/or receiving information, data, parameters, parameters, or memory content. Information, parameters, parameters, data, etc. may be transferred, transferred or transmitted via any suitable method including memory sharing, message transfer, token transfer, network transfer or the like.

In addition, the embodiment can be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the code or code segments (such as computer program products) used to perform the necessary tasks can be stored in a computer-readable or machine-readable medium. The processor can perform the necessary tasks.

In summary, the present invention does not need to personally go to the real environment to obtain the image, and can obtain the position in the three-dimensional space through the light tag and set the virtual object at the remote end.

The present invention has been described in detail above, but what is described above is only a preferred embodiment of the present invention, and should not be used to limit the scope of implementation of the present invention, that is, everything made in accordance with the scope of the patent application of the present invention is equal Changes and modifications should still fall within the scope of the patent of the present invention.

:向量

:坐標

:坐標

:純量

:純量

:純量 S201-205:步驟 100: System for remotely constructing virtual scenes 10: First optical label device 10A: First optical label device 20: Second optical label device 20A: Second optical label device 30: Optical label identification device 30A: Optical label identification device 30B : Smart phone 32: Camera unit 34: Display 36: Processor P: User

:vector

:coordinate

:coordinate

: Scalar

: Scalar

: Scalar S201-205: Step

Fig. 1 is a block diagram of a system for remotely constructing a virtual scene according to the present invention.

Figure 2 is a block diagram of the optical label identification device of the present invention.

Fig. 3 is a schematic flowchart of a method for remotely constructing a virtual scene of the present invention.

Fig. 4 is a schematic block diagram of anchoring by the optical label identification device of the present invention.

Fig. 5 is a schematic diagram of scene information at different time points of the present invention.

FIG. 6 is a schematic diagram of the business situation of the store at different time points in the scene information of the present invention.

FIG. 7 is a schematic diagram of the present invention displaying scene information at 8 o'clock in the evening on a smart phone.

100: System for remotely constructing virtual scenes

10: The first optical label device

20: The second optical label device

30: Optical label recognition device

Claims (11)

A system for remotely constructing a virtual scene includes: at least one first optical tag device with a first optical tag, the first optical tag at least corresponding to a device pose information; a second optical tag device with a first optical tag Two optical labels; and an optical label identification device including a photographing unit, a display, and a processor. The processor calculates and obtains the pose information, and creates scene information relative to the device pose information of the first light tag device based on the pose information and the first light tag device selected by the user, and displays the scene information on the display And provide a virtual object setting function. The scene information includes three-dimensional information corresponding to the scene space where the first optical label device is located, and the position of the user relative to the first optical label device seen in the display The distance is the same as the orientation and distance of the optical label identification device and the second optical label device. In the system for remotely constructing a virtual scene as described in the first item of the scope of patent application, the scene information around the first optical label device is obtained by a server or a storage unit in the processor. For example, the system for remotely constructing a virtual scene as described in item 1 or item 2 of the scope of patent application, wherein the scene information around the first optical label device includes the established The processor can adjust the created virtual object. For example, in the system for remotely constructing a virtual scene as described in item 3 of the scope of patent application, the scene information around the server or the first optical label device of the storage unit includes a plurality of scene information corresponding to individual time points and the scene information Displayed on the display for the user to choose. For the system for remotely constructing a virtual scene as described in item 4 of the scope of patent application, the processor creates the scene information corresponding to the time point according to the selected time point. A method for remotely constructing a virtual scene, including: a photographing unit of an optical label recognition device takes an image of a second optical label device; the image is calculated by a processor of the optical label recognition device to obtain position information The processor establishes a scene information relative to a device pose information of the first optical label device based on the pose information and a first optical tag device selected by the user; the scene information is displayed on a piece of the optical tag identification device And the processor sets a virtual object in the scene information, the scene information includes three-dimensional information corresponding to the scene space where the first light tag device is located, and the user seen in the display relative to the First light label The orientation and distance of the positioning are the same as the orientation and distance of the optical tag identification device and the second optical tag device. In the method for remotely constructing a virtual scene as described in item 6 of the scope of patent application, the scene information around the first optical label device is obtained by a server or a storage unit in the processor. In the method for remotely constructing a virtual scene as described in item 6 or item 7 of the scope of patent application, the processor can adjust the virtual object that has been created in the scene information around the first optical label device. In the method for remotely constructing a virtual scene as described in item 8 of the scope of patent application, the display can display scene information around the first light label device including a plurality of scene information corresponding to individual time points. In the method for remotely constructing a virtual scene as described in item 9 of the scope of patent application, the scene information at the corresponding time point can be selected by the processor to select the time and establish the corresponding scene information. A non-temporary computer-readable recording medium is used to store a program. When a processing chip is loaded into the program, the method described in any one of items 6 to 10 of the scope of patent application can be executed.

Images