TW201913292A - Mobile device and method for blending display content with environment scene - Google Patents

Abstract

A mobile device capable of blending a display content with an environment scene is provided. The mobile device includes: a sensing unit, configured to sense a distance information between a user and the mobile device; an image capturing unit, configured to capture an environment image and a depth information of the environment image from the environment scene; a display unit, configured to display the environment image; a processing unit coupled to the sensing unit and the image capturing unit, respectively, based on the distance information obtained by the sensing unit and the depth information obtained by the image capturing unit, configured to adjust the display size of the environment image in the display unit so that the environment image is blended with the environment scene that the user sees outside of the display unit.

Description

 Mobile device capable of integrating display content into environmental reality and method thereof  

The present invention relates to an augmented reality mobile device and method thereof, and more particularly to a mobile device and method thereof for integrating display content into an environment.

Augmented Reality (AR) is a way to combine the virtual information generated by computers into real-world images. The virtual world can be combined and interacted with real-world scenes through the screen. This method has been used in many different life applications, such as advertising, games, travel, navigation, education, sports, entertainment, etc., are the areas of augmented reality. In recent years, with the improvement of the computing power of portable electronic products, the use of augmented reality has become more and more widespread.

At present, most of the augmented reality solutions on the market are mobile cameras or computers that add new virtual objects through the screen to enter the device's camera angle range. However, if the user wants to use the augmented reality, the display content of the device can In line with the perspective of the human eye, it is integrated into the real environment, and the current general products still cannot meet this demand. For example, the first figure shows the display content 110 of the general mobile device 100 in a preview mode. If the magnification is not manually adjusted, the display content 110 and the environment in the background can be seen 120. The trees cannot be merged.

The present invention provides a mobile device and method thereof that can integrate display content into an environment.

The present invention provides a mobile device that can integrate display content into an environment, including: a sensing unit for sensing a distance information between a user and the mobile device; and an image capturing unit for The environment image captures an environment image and one of the environment image depth information; a display unit displays the environment image; a processing unit is coupled to the sensing unit and the image capturing unit, respectively, and according to the sensing unit Obtaining the distance information and the depth information obtained by the image capturing unit, for adjusting a display size of the environment image in the display unit, and causing the environment image and the environment view that the user sees outside the display unit Convergence.

The present invention further provides a method for integrating display content into an environment, comprising the steps of: sensing a distance information between a user and the mobile device by using a sensing unit of the mobile device; The capturing unit is configured to capture an environmental image and a depth information of the environmental image from the real scene; display the environmental image by using a display unit; and obtain the image according to the sensing unit by using a processing unit The distance information and the depth information obtained by the image capturing unit are used to adjust the display size of the environment image in the display unit, so that the environment image is merged with the environment view seen by the user outside the display unit.

100‧‧‧Mobile devices

110‧‧‧Display content

120‧‧‧Environmental scene

200‧‧‧Mobile devices

210‧‧‧Image capture unit

220‧‧‧Processing unit

221‧‧Augmented Reality Unit

222‧‧‧Computation unit

230‧‧‧Sensor unit

240‧‧‧ display unit

250‧‧‧ Positioning unit

300‧‧‧Mobile devices

310‧‧‧ human eyes

320‧‧‧Environmental scene

400‧‧‧Mobile devices

410‧‧‧Display content

420‧‧‧Environmental scene

501, 502, 503, 504‧ ‧ steps

601, 602, 603, 604, 605 ‧ ‧ steps

Figure 1 is a schematic diagram showing the display contents of a general mobile device in a camera preview mode.

2A is a schematic diagram of a mobile device that can integrate display content into an environmental reality scene in accordance with a first embodiment of the present invention.

2B is a schematic diagram of a mobile device that can integrate display content into an environmental reality scene in accordance with a second embodiment of the present invention.

Figures 3A and 3B are schematic diagrams of the relative positions of the human eye, the mobile device, and the actual environment.

4A, 4B, 4C, and 4D are schematic diagrams showing the correspondence between content and distance information.

Figure 5 is a flow chart of a method for integrating display content into an environmental reality scene in accordance with a first embodiment of the present invention.

Figure 6 is a flow chart of a method for integrating display content into an environment according to a second embodiment of the present invention.

The above and other objects, features and advantages of the present invention will become more <RTIgt;

2A is a schematic diagram of a mobile device 200 that can integrate display content into an environment according to a first embodiment of the present invention. The mobile device 200 mainly includes an image capturing unit 210, a processing unit 220, a sensing unit 230, and a display unit 240. In this embodiment, the mobile device 200 is, for example, a tablet computer, a smart phone, a mobile phone, a mobile device without a bezel or a narrow bezel, or an electronic device such as a notebook computer, but is not limited thereto.

The image capturing unit 210 is disposed in the rear lens area of the mobile device 200, that is, the main lens area of the general camera, and is mainly used to capture an environment image for an environment, and the environment image may be a continuous motion image or a static image, and Have a depth of information. The image capturing unit 210 can be any device or device that can capture image and depth information, for example, a dual lens camera/camera, a stereo camera/camera, a laser stereo camera/camera (a laser device with a laser depth value) ), an infrared stereo camera/camera (photographic device with infrared measurement depth value) or a three-dimensional scanner. The three-dimensional scanner may be a laser scanner, a structured light scanner or a time of flight scanner. Furthermore, the image capturing unit 210 can use a zoom lens including optical zoom or digital zoom to change the magnification or use the zoom lens to push or pull, so that the environment image has zoom in or zoom out (zoom out )Features.

The display unit 240 is coupled to the image capturing unit 210 through the processing unit 220 for displaying the environmental image captured by the image capturing unit 210 or the environmental image adjusted by the processing unit 220. The display unit 240 can be any device that can display images, such as a liquid crystal display, a plasma display, an LED display, an OLED display, a transparent display, etc., and the display unit 240 can further have a touch function, so that the user and the mobile device 200 can be displayed through the display. Unit 240 has an interactive function.

The sensing unit 230 is disposed in the front lens area of the mobile device 200, that is, the general user uses the secondary lens area from the shooting, and is mainly used to measure or sense a distance information between the user and the mobile device 200. Sensing unit 230 can be any device or device that can sense the length of the distance between the user and mobile device 200, such as a proximity sensor, an infrared range sensor, a dual lens camera/camera, or a stereo camera/camera. When the user changes the distance between the mobile device 200 through the hands or one hand, the sensing distance may range from 10 cm to 70 cm, which is approximately the average user's hands or one hand holding the mobile device and the face. The variable distance between the parts, but the embodiment of the invention does not limit the sensing range.

The processing unit 220 is coupled to the image capturing unit 210, the sensing unit 230, and the display unit 240, and the environment image captured by the image capturing unit 210, the depth information of the environment image, and the sensing unit 230. The distance information between the user and the mobile device 200 is used to adjust the display size of the environmental image in the display unit 240 to integrate the environmental image with the environmental reality seen by the user outside the display unit. The processing unit 220 may be a processing unit having a computing capability such as a central processing unit (CPU), a microprocessor, or a microcontroller.

2B is a schematic diagram of a mobile device 200 that can integrate display content into an environment according to a second embodiment of the present invention. In this embodiment, elements having the same names as those in the first embodiment have functions as described above, and are not described herein again. The main difference between the 2B and 2A is that the processing unit 220 further includes an augmented reality unit 221 and a computing unit 222, and the processing unit 220 is coupled to the positioning unit 250. The augmented reality unit 221 provides an operation interface for the user to enable the Augmented Reality (AR) function to take a pre-established virtual image or take a specific object image from an image and place the image capture image. The integrated image captured by the unit 210 is post-integrated and displayed, and an augmented reality image is displayed through the display unit 240, and the virtual image of the augmented reality is expanded or non-equalized by the calculation unit 222, and the new image is superimposed. The added object enters the environmental image. Further, the positioning unit 250 provides a mobile device positioning function, and the positioning unit 250 can be a three-dimensional spatial positioning unit or an inertial measurement unit (IMU), such as a gyroscope, to provide a user-operated amplification. The rotation angle of the image is zoomed in or out.

In various embodiments of the present invention, the mobile device 200 includes the image capturing unit 210, the processing unit 220, the sensing unit 230, and the display unit 240, and is used by other hardware devices, operating systems, application software, and communication networks. You can turn on photography, photography, internet and location.

Next, please refer to the relative position diagrams of the human eye 310, the mobile device 300 and the environment real scene 320 in FIGS. 3A-3B. When the user turns on the preview mode, the operation interface (not shown) provides a The function of the display content is integrated into the real environment. After the user turns on the function, the image capturing unit of the mobile device 300 captures the current environment image and the depth information of the environment image, that is, the environment environment 320 (eg, tree) and the mobile device. The depth value of 300, the closer the mobile device 300 is to the ambient scene 320, the smaller the depth value, and vice versa. In addition, the sensing unit senses distance information between the user and the mobile device 300, such as the distance between the user's face and the mobile device 300. The sensing unit 230 may sense one point in the middle of the user's eyes (the human eye 310) as a reference point for sensing the distance information, or sense the face plane of the user as a reference surface for sensing the distance information. . The sensing distance may range from 10 cm to 70 cm, and the user may hold the variable distance between the mobile device 300 and the face through both hands or one hand. Further, the processing unit receives the depth information of the environment image and the distance information of the user, and zooms in or zooms out the environment image according to the value of the distance information to match the display content to the environment. Real scene 320. Wherein, when the value of the distance information becomes small, the processing unit scales down the environment image proportionally. Conversely, when the value of the distance information becomes larger, the processing unit enlarges the environment image proportionally to maintain the environment image in the environment. . The processing unit ranges the environment image magnification by a factor of 1 to 4, but the embodiment of the present invention does not limit the range.

For example, please refer to the corresponding relationship between the display content 410 and the distance information in the 4A-4D figure. Before the user opens the function of displaying the content into the environment, the display content 410 will be similar to the first figure of the drawing, and the content is displayed. The tree with the real environment in the background cannot be merged, and after the user turns on the function, the display content 410 will gradually align with the real scene 420, as shown in FIG. 4A. The figure 4A~4D shows the display content 410 of the mobile device 400 as seen by the user's human eye angle corresponding to different distance information after the user turns on the display content into the environment. 4A is a depth information of the environmental image when the human eye and the mobile device 400 are separated by a predetermined distance (for example, 10 cm) (for example, the depth value of the mobile device 400 and the environment real scene 420 is 5 meters). When the distance between the environment 420 (eg, the tree) is substantially constant, the mobile device 400 aligns the display content 410 with the environmental reality 420, at which time the environment image magnification is set to 1 time; then, when the mobile device 400 Gradually away from the human eye, as shown in Fig. 4B, when the distance between the human eye and the mobile device 400 is 30 cm, the environmental image is gradually enlarged to 2 times; when the mobile device is farther away from the human eye, as shown in Fig. 4C, the human eye and When the distance of the mobile device 400 is 50 cm, the environmental image is further enlarged by 3 times, and the mobile device 400 still aligns the display content 410 with the environmental reality scene 420; finally, as shown in FIG. 4D, the distance between the human eye and the mobile device 400 is 70 cm. The environmental image magnification is 4 times to maintain the environmental image in the ambient scene 420; conversely, when the mobile device 400 gradually approaches the human eye, the environmental image is scaled down. As described above, when the distance is changed, the mobile device 400 instantly senses the distance between the human eye and the device while zooming in or out the display size of the environmental image so that the display of the mobile device 400 is seen in the user's human eye. The content 410 is merged with the environmental reality 420 seen outside of the mobile device.

Figure 5 is a flow chart of a method for integrating display content into an environmental reality scene in accordance with a first embodiment of the present invention. Referring to FIG. 2A of the first embodiment of the present invention, in step 501, a distance information between a user and the mobile device 200 is sensed by a sensing unit 230 of the mobile device 200 of FIG. In step 502, an image capturing unit 210 of FIG. 2A is used to extract an environment image and one depth information of the environment image from the environment. In step 503, the environmental image is displayed by a display unit 240 of FIG. 2A. In step 504, the distance information obtained by the sensing unit and the depth information obtained by the image capturing unit are used to adjust the environmental image in the display unit by using a processing unit 220 of FIG. The display size is such that the environmental image is merged with the environment view that the user sees outside of the display unit. In step 504, the processing unit 220 zooms in or out of the environment image according to the magnitude of the distance information and the depth information. Wherein, when the value of the distance information becomes smaller, the environment image is scaled down. Conversely, when the value of the distance information becomes larger, the environment image is enlarged according to a ratio to maintain the environment image in the environment. Real scene. In addition, in step 501, the range of the distance information is 10 to 70 cm. In step 504, the processing unit 220 ranges the environment image magnification by 1 to 4 times.

Figure 6 is a flow chart of a method for integrating display content into an environment according to a second embodiment of the present invention. In the method flow chart, the same steps as the method flow chart of the first embodiment of Fig. 5 are as described above, and are not described herein again. Referring to FIG. 2B of the second embodiment of the present invention, the main difference between FIG. 6 and FIG. 5 is that the method further includes step 605. In step 605, the processing unit 220 further includes an augmented reality. The unit 221 and a computing unit 222 are configured to provide an augmented reality function, and display an augmented reality image through the display unit 240. In step 605, the processing unit 220 adjusts the augmented reality image according to the adjusted display size.

Therefore, with the mobile device and method of the present invention that can integrate the display content into the real environment, the mobile device automatically enlarges or reduces the display size of the environmental image without manually adjusting the magnification by the user, so as to achieve the user's human eye angle of view. The effect of seeing the display content of the mobile device and the environmental reality seen outside the mobile device adds a better user experience to the mobile device in the augmented reality.

While the present invention has been described in its preferred embodiments, the present invention is not intended to limit the invention, and the present invention may be modified and modified without departing from the spirit and scope of the invention. The scope of protection is subject to the definition of the scope of the patent application.

Claims (20)

 A mobile device capable of integrating display content into an environment, comprising: a sensing unit for sensing a distance information between a user and the mobile device; and an image capturing unit for using the environment from the environment An environmental image and a depth information of the environmental image; a display unit for displaying the environmental image; a processing unit coupled to the sensing unit and the image capturing unit, respectively, and obtained according to the sensing unit The distance information and the depth information obtained by the image capturing unit are used to adjust the display size of the environment image in the display unit, so that the environment image is merged with the environment view seen by the user outside the display unit.    The mobile device of claim 1, wherein the processing unit zooms in or zooms out the environmental image according to the magnitude of the distance information and the depth information.    The mobile device of claim 2, wherein, when the value of the distance information becomes smaller, the processing unit scales the environmental image proportionally, and conversely, when the value of the distance information becomes larger, the processing The unit magnifies the environmental image in proportion to maintain the environmental image in the environment.    The mobile device of claim 3, wherein the range of the distance information is 10 to 70 cm, and the processing unit ranges the environment image magnification by 1 to 4 times.    The mobile device of claim 1, wherein the mobile device is a mobile device having no border or narrow border.    The mobile device of claim 1, wherein the sensing unit is a proximity sensor, an infrared distance sensor, a dual lens camera or a stereo camera.    The mobile device of claim 1, wherein the image capturing unit is a two-lens camera, a stereo camera or a three-dimensional scanner.    The mobile device of claim 7, wherein the three-dimensional scanner is a laser scanner, a structured light scanner or a time of flight scanner.    The mobile device of claim 1, wherein the processing unit further comprises an augmented reality unit and a computing unit for providing an augmented reality (AR) function, and displaying an expansion through the display unit Increase the reality image.    The mobile device of claim 9, wherein the processing unit adjusts the augmented reality image in an equal proportion according to the adjusted display size.    The mobile device of claim 9 or claim 10, wherein the mobile device further comprises a positioning unit coupled to the processing unit for positioning the augmented reality image.    The mobile device of claim 11, wherein the positioning unit is a three-dimensional spatial positioning unit or an inertial measurement unit.    The mobile device of claim 1, wherein the sensing unit senses a point in the middle of the user's eyes as a reference point for sensing the distance information.    The mobile device of claim 1, wherein the sensing unit senses a face plane of the user as a reference surface for sensing the distance information.    A method for integrating display content into an environment, for a mobile device, comprising the steps of: sensing a distance information between a user and the mobile device by using a sensing unit of the mobile device; An image capturing unit is configured to capture an environment image and a depth information of the environment image from the real scene; display the environment image by using a display unit; and obtain, according to the sensing unit, a processing unit The distance information and the depth information obtained by the image capturing unit are used to adjust the display size of the environment image in the display unit, so that the environment image is similar to the environment view seen by the user outside the display unit. Fusion.    The method of claim 15, wherein the step of adjusting the display size of the environmental image in the display unit by the processing unit is based on the numerical value of the distance information and the depth information, The environment image zooms in or zoom out.    The method of claim 16, wherein when the value of the distance information becomes smaller, the environmental image is scaled down, and conversely, when the value of the distance information becomes larger, the environmental image is proportionally Zoom in to maintain the environment image into the real scene.    The method of claim 17, wherein the range of the distance information is 10 to 70 cm, and the processing unit ranges the environment image magnification by 1 to 4 times.    The method of claim 15, further comprising the step of: the processing unit further comprising an augmented reality unit and a computing unit for providing an augmented reality function, displaying a display through the display unit Amplify the real-world image.    The method of claim 19, wherein the processing unit adjusts the augmented reality image in an equal proportion according to the adjusted display size.