TWI691932B - Image processing system and image processing method - Google Patents

Abstract

A image processing system comprises a camera, a positioning device, a processor, and a display. The camera captures a real view image. The positioning device positions a camera position of the camera. The processor receives a high-precision map and a virtual object and calculates a camera posture. According to a three-dimensional information of the camera posture and the high-precision map, the processor projects a depth image, superimposes the depth image and the real view image to generate a stack image, and superimposes the virtual object to the stack image according to a virtual coordinate of the virtual object to produce a rendered image. The display displays the rendered image.

Description

Image processing system and image processing method

This case relates to an image processing system and image processing method, and particularly to an image processing system and image processing method applied to augmented reality.

Generally speaking, augmented reality technology requires a camera that can shoot the depth of the image to obtain the distance between the real object and the camera, and then the virtual object is placed on the screen according to the depth of the image. However, the general depth camera can only obtain the image depth of the environment within a distance of about 2~3 meters from the camera. When the user wants to render a large virtual object on a large object outdoors, such as rendering a huge virtual octopus In the real scene of a high-rise building, and presenting this picture on the display, the depth camera will not be able to obtain the correct depth of large outdoor objects, resulting in the inappropriate placement of the rendered picture between the objects (such as errors The shadowing effect).

Therefore, how to accurately combine virtual images with large objects in augmented reality has become one of the problems to be solved.

According to one aspect of the present case, an image processing system is provided, including: a camera, a positioning device, a processor, and a display. The camera is used to shoot a real image. The positioning device is used to detect a camera position of the camera. The processor is coupled to the camera and the positioning device for receiving a high-precision map and a virtual object. The processor uses a simultaneous localization and mapping (Simultaneous localization and mapping, SLAM) algorithm and the camera position to calculate a camera of the camera Pose, and project a depth image based on the camera pose and a three-dimensional information of the high-precision map. The processor superimposes the depth image and the real image to generate a superimposed image. Based on a virtual coordinate of the virtual object, the virtual The object is superimposed on the superimposed image to produce a rendered image. The display is coupled to the processor for displaying rendered images.

According to another aspect of the case, there is provided an image processing method including: shooting a real image with a camera; positioning a camera position of the camera with a positioning device; building an algorithm and camera position calculation camera with a simultaneous positioning and map A camera posture; a depth image is projected by the processor according to the camera posture and a three-dimensional information of a high-precision map; a depth image and a real image are superimposed by the processor to generate a superimposed image; by The processor superimposes the virtual object on the superimposed image according to a virtual coordinate of the virtual object to generate a rendered image; and displays the rendered image through a display.

In summary, the image processing system and image processing method of this case use high-precision maps, camera poses and other information to project the real image into a depth image after conversion, and accurately overlay the depth image and the real image to produce a superimposed image Each point in the superimposed image contains depth information, so that the virtual object can be mapped to the position in the superimposed image according to its virtual coordinates to be rendered onto the superimposed image, thereby enabling large objects and objects in the real image Virtual objects are precisely combined.

Please refer to FIGS. 1-2. FIG. 1 is a block diagram of an image processing system 100 according to an embodiment of the present invention. FIG. 2 is a flowchart of an image processing method 200 according to an embodiment of the present case. In one embodiment, the image processing system 100 includes a camera 10, a positioning device 20, a processor 30, and a display 40. The processor 30 is coupled to the camera 10 and the positioning device 20. The display 40 is coupled to the processor 30.

In an embodiment, the camera 10 may be a charge coupled device (Charge Coupled Device, CCD) or a complementary metal oxide semiconductor (Complementary Metal-Oxide Semiconductor, CMOS), and the positioning device 20 may be a global positioning system (Global Positioning System, GPS ) Locator, the processor 30 may be implemented as a micro controller, a microprocessor, a digital signal processor, an application specific integrated circuit (ASIC) or a Logic circuit.

In an embodiment, the display 40 may be a display device of a handheld electronic device (such as a mobile phone or a tablet) or a display device of a head-mounted device.

In one embodiment, please refer to Figures 2 and 3A~3D. Figures 3A~3D are schematic diagrams of an image processing method according to an embodiment of the present invention. The flowchart of the image processing method 200 in this case is described in detail below. The components mentioned in the image processing method 200 may be implemented by the components described in FIG. 1.

In step 210, the camera 10 shoots a real image (for example, FIG. 3C).

In step 220, the positioning device 20 detects a camera position of the camera 10. In an embodiment, the positioning device 20 in the image processing system 100 is a global positioning system locator, which can be used to locate the camera position of the camera 10.

In step 230, the processor 30 receives a high-precision map and a virtual object.

In an embodiment, a high-precision map may be prepared in advance. For example, the processor 30 may obtain a high-precision map stored in a remote storage device through a network, or read the high-resolution map from the storage device of the image processing system 100 itself. Accuracy map. In one embodiment, the high-precision map contains three-dimensional information, for example, as shown in FIG. 3A, the three-dimensional information includes one of the three-dimensional composition of the real image or the height information of the objects in the real image (such as buildings OBJ1, OBJ2).

In one embodiment, the virtual object may be prepared in advance, which may be a three-dimensional virtual object, and the virtual coordinates of each point in the virtual object may also be defined. For example, the virtual object shown in FIG. 3B is an octopus VI, which is used The processor draws the octopus VI in advance through the processor 30 and defines the virtual coordinates of each point on the image of the octopus VI.

In step 240, the processor 30 calculates a camera posture of the camera 10.

In one embodiment, the processor 30 calculates the camera posture of the camera 10 by using a simultaneous localization and mapping (SLAM) algorithm and camera position. In one embodiment, after the camera 10 continuously shoots multiple images from different perspectives in a scene, it uses simultaneous positioning and map construction algorithms to compare the corresponding positions of feature points on the same object between these images by By superimposing the corresponding positions of these feature points in different images, a three-dimensional map of this scene can be generated, and the position where the camera 10 is placed can be located, thereby inferring the camera posture of the camera 10 according to the feature points (for example, horizontal, tilt Or placed vertically) and/or shooting angle of view (for example, the relative position of the camera 10 and a building).

In step 250, the processor 30 projects a depth image based on the camera pose and the three-dimensional information of the high-precision map.

In one embodiment, the processor 30 maps each point coordinate in the high-precision map to a camera coordinate through a coordinate conversion function, so that it can be projected onto the display 40.

For example, the processor 30 maps the three-dimensional information in the high-precision map, such as the world coordinate (marked as a symbol wc in the coordinate conversion function) by the following coordinate conversion function, and converts it into a camera coordinate (camera coordinate, marked as the symbol cc in the coordinate conversion function: [xyz] ^T _cc = [R|t][XYZ 1] ^T _wc where each coordinate (X, Y, Z) in the high-precision map Substitute this coordinate conversion function, and by adjusting the rotation parameter R and the translation parameter t, each point in the high-precision map can be projected to the camera coordinates; in other words, the high-precision map with three-dimensional information can be converted through the coordinate conversion function The conversion is performed so that it can be projected on the display 40 (for example, a mobile phone screen).

In one embodiment, the processor 30 can apply a known image location tracking technology or perspective projection to project a depth image.

In an embodiment, the depth image is, for example, shown in FIG. 3B, which is a gray-scale image, and the object closer to the camera 10 is brighter (for example, the gray-scale block a, whose position corresponds to the real image, as shown in FIG. 3C Building image a'), the further away from the camera 10, the darker the object (for example, gray scale block c, its position corresponds to the real image, such as the architectural image c'in Figure 3C), the position of gray scale block b ( It corresponds to the real image, such as the architectural image b'in FIG. 3C, between the gray-scale block a and the gray-scale block c. In addition, the depth image may include a pre-drawn virtual image VI, for example, the octopus VI in FIG. 3B, which may be a three-dimensional image and includes the defined virtual coordinates and colors.

In step 260, the processor 30 superimposes the depth image and the real image to generate a superimposed image. For example, the processor 30 superimposes the depth image shown in FIG. 3B and the real image shown in FIG. 3C to generate a superimposed image.

In an embodiment, the processor 30 compares a gray-scale edge information in the gray-scale image (for example, the edge L in FIG. 3B) and a real edge information in the real image (for example, the edge L in the 3C image) ') for comparison, and by rotating or translating the grayscale image or one of the real images (such as rotating or translating the grayscale image to align the edge L with the edge L'in the real image), the depth image and Real images overlap. In one embodiment, multiple edge information in the depth image and the real image can be compared, and by rotating and translating one of the depth image and the real image, the depth image and the real image can be superimposed more accurately.

In one embodiment, each point in the superimposed image includes depth information, which may be expressed in a coordinate manner or a depth numerical manner, for example.

In one embodiment, the superimposed image includes the coordinate information of each point in the real image.

In step 270, the processor 30 superimposes the virtual object on the superimposed image according to a virtual coordinate of the virtual object to generate a rendered image.

In one embodiment, rendering is defined as adding real images to virtual images or virtual objects.

In one embodiment, since the superimposed image includes the coordinate information of each point in the real image, the processor 30 superimposes the virtual object on the superimposed image according to the coordinate information and the virtual coordinate of the virtual object.

In one embodiment, the processor 30 superimposes a virtual object (such as the octopus VI in Figure 3B) according to the virtual coordinates of each point to the superimposed image, thereby generating a rendered image as shown in Figure 3D. Among them, because the superimposed image contains the coordinate information of each point in the image, and each virtual coordinate point of the virtual object (such as octopus VI) is also defined in advance, you can render the octopus VI to the superimposed image according to the virtual coordinates on. In this example, since the octopus VI only has the coordinates of two feet in front of the building a'and the coordinates of the other feet are behind the building a', it can only be seen that it is not obscured by the building a'in Figure 3D From the two feet, it can be seen that each point of the octopus VI is rendered to the position after the superimposed image can be calculated correctly.

In step 280, the display 40 displays the rendered image.

In summary, the image processing system and image processing method of this case use high-precision maps, camera poses and other information to project the real image into a depth image after conversion, and accurately overlay the depth image and the real image to produce a superimposed image Each point in the superimposed image contains depth information, so that the virtual object can be mapped to the position in the superimposed image according to its virtual coordinates to be rendered onto the superimposed image, thereby enabling large objects and objects in the real image Virtual objects are precisely combined.

Although this case has been disclosed above with examples, it is not intended to limit this case. Anyone who is familiar with this skill can make various changes and retouching without departing from the spirit and scope of this case, so the scope of protection of this case should be considered The scope of the attached patent application shall prevail.

100: image processing system 10: camera 20: positioning device 30: processor 40: display 200: image processing method 210-280: steps OBJ1, OBJ2: building VI: octopus a, b, c: gray-scale block a' , B', c': architectural image

In order to make the above and other purposes, features, advantages and embodiments of the present disclosure more obvious and understandable, the attached drawings are explained as follows: Figure 1 is a block diagram of an image processing system according to an embodiment of the present case Figure 2 is a flowchart of an image processing method according to an embodiment of the case; and Figures 3A~3D are schematic diagrams of an image processing method according to an embodiment of the case.

200: Image processing method 210~280: Step

Claims (10)

An image processing system includes: a camera for shooting a real image; a positioning device for positioning a camera position of the camera; a processor coupled to the camera and the positioning device and receiving a high-precision map And a virtual object, the processor calculates a camera pose of the camera by a synchronous positioning and map building algorithm and the camera position, and projects a depth based on the camera pose and a three-dimensional information of the high-precision map Image, the processor superimposes the depth image and the real image to generate a superimposed image, and the processor superimposes the virtual object to the superimposed image according to a virtual coordinate of the virtual object to generate a rendered image And a display coupled to the processor to display the rendered image. The image processing system according to claim 1, wherein the depth image is a grayscale image, and the processor compares a grayscale edge information in the grayscale image with an actual edge information in the real image, And by rotating or translating one of the grayscale image or the real image, the depth image and the real image are superimposed. The image processing system according to claim 1, wherein the superimposed image includes coordinate information of each point in the real image, and the processor uses the coordinate information and the virtual coordinates of the virtual object to select the virtual object Superimpose to the superimposed image. The image processing system according to claim 1, wherein the three-dimensional information of the high-precision map includes a three-dimensional composition of the real image or height information of objects in the real image. The image processing system according to claim 1, wherein the processor maps each point coordinate in the high-precision map to a camera coordinate displayed by the display through a coordinate conversion function. An image processing method, including: shooting a real image with a camera; locating a camera position of the camera with a positioning device; calculating with a processor based on a simultaneous localization and mapping (Simultaneous localization and mapping, SLAM) Method and the camera position to calculate a camera posture of the camera; by the processor according to the camera posture and a three-dimensional information of a high-precision map to project a depth image; by the processor superimposing the depth image and Generating a superimposed image by the real image; superimposing the virtual object on the superimposed image according to a virtual coordinate of a virtual object by the processor to generate a rendered image; and displaying the rendering by a display image. The image processing method according to claim 6, wherein the depth image is a grayscale image, and the image processing method further includes: a grayscale edge information in the grayscale image and an actual edge information in the real image Make a comparison, and rotate or translate the grayscale image or the real image to make the depth image and the real image overlap. The image processing method according to claim 6, wherein the superimposed image includes coordinate information of each point in the real image, and the image processing method further includes: based on the coordinate information and the virtual coordinates of the virtual object, to Superimpose the virtual object to the superimposed image. The image processing method according to claim 6, wherein the three-dimensional information of the high-precision map includes a three-dimensional composition of the real image or a height information of an object in the real image. The image processing method according to claim 6, further comprising: mapping each point coordinate in the high-precision map to a camera coordinate displayed by the display through a coordinate conversion function.

Images