WO2017133160A1

WO2017133160A1 - Smart eyeglass perspective method and system

Info

Publication number: WO2017133160A1
Application number: PCT/CN2016/086347
Authority: WO
Inventors: 艾韬; 蔡铁峰
Original assignee: 深圳市易瞳科技有限公司
Priority date: 2016-02-03
Filing date: 2016-06-20
Publication date: 2017-08-10
Also published as: CN107037584B; CN107037584A

Abstract

A smart eyeglass perspective method, comprising the following steps: step a (S500): assembling smart eyeglasses so that an optical axis of human eyes (5), an optical axis of a magnifier (4), and an optical axis of a camera (1) are collinear, a normal direction of a display screen (3) is parallel to the optical axis of the magnifier, and a connection line between the center of a half-screen display area corresponding to a single human eye and an optical center of the magnifier is collinear with the optical axis of the human eyes; step b (S600): operating the smart eyeglasses, generating an image of an external scene by using the camera, and displaying the image on the display screen according to a given mapping relation; and step c (S700): wearing the smart eyeglasses, magnifying the image displayed on the display screen by using the magnifier, and displaying the image after zooming out till it is comfortable for the human eyes to view the image. Also disclosed is a smart eyeglass perspective system. The smart eyeglass perspective method and perspective system enable a human wearing smart eyeglasses to accurately sense pose and size information of an external scene as if sensing with naked eyes, thereby improving comfort and viewing accuracy of a wearer.

Description

Intelligent glasses perspective method and system

Technical field

The present invention relates to the field of smart glasses, and in particular, to a smart glasses perspective method and system.

Background technique

For most glasses, ambient light can be received by the human eye through the glasses, which are optically transparent. When a person wears smart glasses, the scenery light can not be received by the human eye, and the human eye indirectly perceives the external scene by watching the image on the display screen. At this time, the external scene pose, size, etc. which are indirectly perceived by the human eye may appear. The information is inconsistent with the external scene information directly perceived by the naked eye of the human eye, and thus the human eye cannot accurately perceive the external scene.

Summary of the invention

The invention provides a method and a system for fluoroscopy of an intelligent glasses, which are intended to solve the problem that the information, such as the position and size of the external scene, which is indirectly perceived by the human eye during the wearing process of the existing smart glasses, is inconsistent with the external scene information directly perceived by the naked eye of the human eye. Technical problem.

In order to solve the above problems, the technical solution adopted by the present invention is: a smart glasses perspective method, comprising the following steps:

Step a: Assembling the smart glasses so that the human eye axis, the magnifier optical axis, and the camera optical axis are on the same straight line, the normal direction of the display screen is parallel to the optical axis of the magnifying glass, and the center of the half-screen display area corresponding to a single human eye and the light source of the magnifying glass The line is on the same line as the human eye axis;

Step b: running the smart glasses, and generating an image on the external scene through the camera, so that the image is displayed on the display screen according to a certain mapping relationship;

Step c: Wear smart glasses and use the magnifying glass to display the image displayed on the display Zoom in and zoom out the image to a distance that the human eye can comfortably view.

The technical solution adopted by the embodiment of the present invention further includes: before the step a, the method further comprises: selecting a size and a resolution of the smart glasses display screen; setting a focal length of the magnifying glass; and respectively setting a distance of the human eye to the optical center of the magnifying glass, The distance from the center of the magnifier to the display, and the distance from the center of the magnifier to the center of the camera.

The technical solution adopted by the embodiment of the present invention further includes: before the a, the method further comprises: calculating a required camera angle of view for avoiding a missing field of view of the human eye, and calculating a required camera resolution with a matching display resolution as a target. .

The technical solution adopted by the embodiment of the present invention further includes: calculating the field of view of the camera by:

The camera's horizontal field of view is:

The vertical field of view of the camera is:

In the above formula, d _{em is} the distance from the human eye to the optical center of the magnifying glass, d _mc is the distance from the optical center of the magnifying glass to the optical center of the camera, d _md is the distance from the optical center of the magnifying glass to the display screen, and f _m is the focal length of the magnifying glass, q _x And q _y is the half-screen display area width and height of the display corresponding to a single camera, d _{eo is} the distance from the human eye to the scene, d _co is the distance from the camera to the scene, f _c is the camera focal length, and dX is the camera single pixel Horizontal physical size, dY is the vertical dimension of the camera's single pixel, d _{ev is} the distance from the human eye to the virtual image, d _mv is the distance from the optical center of the magnifying glass to the virtual image of the scene, d _vo is the distance from the virtual image of the scene to the scene, and s _d is the scene in the display Size, s _v is the corresponding virtual image size, s _o is the size of the scene, s _c is the size of the scene in the camera, dx is the horizontal physical size of a single pixel on the display, and dy is the vertical size of a single pixel on the display.

The technical solution adopted by the embodiment of the present invention further includes: the calculation formula of the calculation camera resolution is:

In the above formula, N _x is the horizontal maximum pixel number corresponding to the half-screen display area of one eye, and N _y is the maximum vertical pixel number.

The technical solution adopted by the embodiment of the present invention further includes: in the step b, the displaying the image on the display screen according to a certain mapping relationship is to display the image to the human eye at a certain scaling ratio; The solution method includes: when satisfying the perspective requirement, the ratio of the number of pixels that the scene should occupy in the display screen to the number of pixels occupied by the scene in the camera is the image scaling ratio p=(p _x , p _y ), where p _x is the scaling in the horizontal direction, and p _y is the scaling in the vertical direction; the ratio of the size of the scene on the display to the physical size of a single pixel on the display is the number of pixels occupied by the scene on the display, and the scene is in The ratio of the image size on the camera to the physical size of a single pixel on the camera's imaging surface is the number of pixels the scene occupies on the camera's imaging surface, ie:

The image scaling p _x in the landscape direction is:

The image scaling p _y in the portrait direction is:

Another technical solution adopted by the embodiment of the present invention is: a smart glasses perspective system, including a camera, a display screen, and a magnifying glass; the camera corresponds to a position of a human eye, and the display screen is located between the camera and the magnifying glass. The magnifying glass is located in the smart glasses near the human eye. Side, and the human eye axis, the magnifier optical axis and the camera optical axis are on the same line, the normal direction of the display screen is parallel to the optical axis of the magnifying glass, and the connection between the center of the half-screen display area corresponding to a single human eye and the optical center of the magnifying glass and the human eye The axes are on the same line; the camera is used to generate an image on the external scene, and the image is displayed on the display screen in a certain mapping relationship, and the magnifying glass is used to enlarge the image displayed on the display screen, and pull the image Displayed as far as the human eye can comfortably view.

The technical solution adopted by the embodiment of the present invention further includes: an operation processing unit, wherein the operation processing unit is respectively connected to the camera and the display screen signal, and is configured to perform distortion correction, polar line correction processing, and display by the camera-generated image. The screen displays the processed image.

The technical solution adopted by the embodiment of the present invention further includes: calculating a field of view of the camera:

The camera's horizontal field of view is:

The vertical field of view of the camera is:

The calculation formula of the camera resolution is:

In the above formula, d is the distance light magnifier _em human eye heart to heart, d is the distance _MC magnifying the optical center of the optical center of the camera, d the distance _MD is the optical center display screen magnifier, f _m is the focal length of the magnifying glass, q _x And q _y is the half-screen display area width and height of the display corresponding to a single camera, d _{eo is} the distance from the human eye to the scene, d _co is the distance from the camera to the scene, f _c is the camera focal length, and dX is the camera single pixel Horizontal physical size, dY is the vertical dimension of the camera's single pixel, d _{ev is} the distance from the human eye to the virtual image, d _mv is the distance from the optical center of the magnifying glass to the virtual image of the scene, d _vo is the distance from the virtual image of the scene to the scene, and s _d is the scene in the display Size, s _v is the corresponding virtual image size, s _o is the size of the scene, s _c is the size of the scene in the camera, dx is the horizontal physical size of a single pixel on the display, dy is the vertical size of a single pixel on the display; N _x is a single eye Corresponding to the horizontal maximum number of pixels in the half-screen display area, N _y is the maximum number of pixels in the vertical direction.

The technical solution adopted by the embodiment of the present invention further includes: displaying the image on the display screen according to a certain mapping relationship, specifically: displaying the image to the human eye at a certain scaling ratio; and the method for solving the image scaling ratio includes: The ratio of the number of pixels that should be occupied in the display to the number of pixels occupied by the scene in the camera is the image scaling p=(p _x , p _y ), where p _x is the scaling in the horizontal direction and p _y is vertical The scaling in the direction; the ratio of the size of the scene on the display to the physical size of a single pixel on the display is the number of pixels the scene occupies on the display, and the size of the scene on the camera and the physics of a single pixel on the camera's imaging surface The size ratio is the number of pixels the scene occupies on the camera's imaging surface, namely:

The image scaling p _x in the landscape direction is:

The image scaling p _y in the portrait direction is:

Compared with the prior art, the invention has the beneficial effects that the smart glasses perspective method and system of the embodiment of the invention image the external scene through the camera, and the generated image is displayed on the display screen after processing, and is displayed on the display screen through the magnifying glass. Image is magnified so that The image is far away to the distance that the human eye can comfortably watch, so that when the human eye wears the smart glasses, it accurately senses the pose and size information of the external scene as the naked eye, and improves the wearer's comfort and viewing accuracy.

DRAWINGS

1 is a flow chart of a method for seeing a smart glasses according to an embodiment of the present invention;

2 is a parameter identification diagram of smart glasses according to an embodiment of the present invention;

3 is a schematic perspective view of a smart glasses according to an embodiment of the present invention;

4(a), 4(b), and 4(c) are schematic diagrams showing a method of calculating a field of view of a camera according to an embodiment of the present invention;

FIG. 5 is a schematic structural view of a smart eyeglass perspective system according to an embodiment of the present invention.

detailed description

In order to facilitate the understanding of the present invention, the present invention will be described more fully hereinafter with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention may be embodied in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the understanding of the present disclosure will be more fully understood.

All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. The terminology used in the description of the present invention is for the purpose of describing particular embodiments and is not intended to limit the invention.

Please refer to FIG. 1 , which is a flowchart of a method for seeing a smart glasses according to an embodiment of the present invention. The smart glasses perspective method of the embodiment of the invention comprises the following steps:

Step 100: Select the size and resolution of the smart glasses display screen;

In step 100, the smart glasses of the embodiments of the present invention include a camera, a display, and a display. The large mirror, the display is located between the camera and the magnifying glass, the magnifying glass is located on the side of the smart glasses close to the human eye, the camera is used to generate images of the external scene, and the generated image is processed by the arithmetic processing unit (such as distortion correction, polar line school) Correction) Displayed by the display screen, the magnifying glass is used to enlarge the image displayed on the display, and the image is pulled far to the distance that the human eye can comfortably watch, that is, the human eye's clear vision distance (about 250mm), so that the human eye is wearing The smart glasses of the embodiment of the invention can accurately sense the position and size of the external scene as the naked eyes.

Step 200: setting a focal length of the magnifying glass;

Step 300: respectively set the distance from the human eye to the optical center of the magnifying glass, the distance from the optical center of the magnifier to the display screen, and the distance from the optical center of the magnifying glass to the optical center of the camera;

Step 400: Calculate a camera angle of view and calculate a camera resolution;

In step 400, selecting the camera field angle can avoid the lack of field of view such as black borders in the field of view of the human eye, and selecting the camera resolution can prevent the image from being interpolated and then zoomed in to the human eye.

Step 500: assembling the smart glasses, and making the human eye axis, the magnifier optical axis, and the camera optical axis on the same straight line, the normal direction of the display screen is parallel to the optical axis of the magnifying glass, and the center of the half-screen display area corresponding to a single human eye and the magnifying glass light center The line is on the same line as the human eye axis;

In the step 500, as shown in FIG. 2 and FIG. 3, FIG. 2 is a smart glasses parameter identification diagram according to an embodiment of the present invention, and FIG. 3 is a schematic perspective view of the smart glasses according to an embodiment of the present invention. In Fig. 2, a is the distance d _{em of the} human eye to the optical center of the magnifying glass, b is the distance d _md from the optical center of the magnifying glass to the display screen, and c is the distance d _mc from the optical center of the magnifying glass to the optical center of the camera. In Fig. 3, the horizontal axis of the solid line represents the human eye, the magnifying glass, the optical axis of the camera, and the normal of the display screen, wherein the human eye is at the point e of the horizontal axis, the magnifying glass is at the m point, and the display is at the d point, f1 is The distance of m is the focal length of the magnifying glass. The camera is at point c, the distance from f2 to c is the focal length of the camera, the v point is the position of the image displayed on the display in the magnifying glass corresponding to the virtual image, and the o point is the position given by the outside world. . The imaging of the scene in the human eye and the imaging of the naked eye in the human eye will be the same, thus achieving perspective.

Under the premise that the assembly requirements of the smart glasses have been met, it is further assumed that the display screen, the magnifying glass, and the distance d _md of the magnifying glass center and the display screen, the magnifying glass center and the camera light have been selected according to factors such as human visual characteristics. heart of the distance d _mc, heart to the human eye from the optical center of the magnifier d _em, such as the glasses known parameters: the distance magnifying optical center of the camera optical center d _mc, magnifying optical center of the display screen the distance d _md, human The distance between the eye of the eye and the center of the magnifier d _em , the focal length of the magnifier f _m , the half-screen display area of the display corresponding to a single camera is q _x , the height is q _y , and the half-screen display area of the single-eye display is the maximum horizontal number of pixels. _x , the maximum number of pixels in the vertical direction is N _y , so that the camera angle of view and resolution are calculated according to the above parameters;

Among them, the way to calculate the camera's field of view is:

Known: heart to the human eye from the optical center of the magnifier d _em, the distance d _mc magnifying the optical center of the camera optical center, a distance d _md magnifying the optical center of the display, magnifying the focal length f _m, corresponding to a single camera display half The screen display area width q _x and height q _y ;

Find: Under the premise of perspective, the minimum field of view angle α=(α _x ,α _y ) required by the camera, where α _x is the horizontal field of view angle and α _y is the vertical field of view angle;

Solution: When the image of the external scene on the display just fills the display, the field of view of the scene corresponding to the camera is the minimum angle of view required by the camera. Therefore, it is known that the image size on the display screen is the size of the half-screen image display area of the display screen, and the angle of view of the scene object at the camera is the value of the field of view to be solved. For convenience of explanation, let d _eo be the distance from the human eye to the scene, d _co is the distance from the camera's optical center to the scene, f _c is the camera focal length, the camera's single pixel horizontal physical size is dX, the vertical size is dY, and d _{ev is the} human eye. The distance to the virtual image, d _mv is the distance from the magnifying glass to the virtual image of the scene, d _vo is the distance from the virtual image to the scene, and the size of the scene in the display is s _d , the corresponding virtual image size is s _v , and the size of the scene is s _o . The image size of the scene in the camera is s _c , and the horizontal physical size of a single pixel on the display screen is dx and the vertical size is dy. among them:

d _co =d _eo -d _em -d _mc ,

d _vo =d _eo -d _em -d _mv ,d _ev =d _em +d _mv .

When the camera's field of view is too small, under the premise of satisfying the perspective, it may happen that the image obtained by the camera cannot cover the full display (in a certain sense, it is to cover the field of view of the full eye), resulting in wearing the smart glasses perspective method. In the system, the field of view of the human eye has a black field and other fields of view missing. Under the premise of satisfying the perspective, in order to ensure that the image obtained by the camera can cover the full display, the camera field of view needs to be smaller than a certain value, as shown in Figure 4(a), Figure 4(b), Figure 4(c). Shown as a schematic diagram of the camera field of view. Among them, the solid horizontal axis represents the common optical axis of the human eye, the magnifying glass, and the camera, and the normal of the display screen. On the horizontal axis of Fig. 4(a), a is the human eye, b is the magnifying glass, c is the display screen, d is the virtual image of the image displayed on the display screen in the magnifying glass, and the length of the vertical line at c represents the half-screen image display of the display screen. The size of the area is, under the premise of perspective, the field of view of the human eye obtained by viewing the image on the display screen outside the d distance is V. On the horizontal axis of Fig. 4(b), one more camera than Fig. 4(a), when the camera angle of view is θ, the field of view of the camera is W. Try to avoid the lack of field of view such as black borders in the human eye field. It is necessary to have as much as possible the field of view of the camera to include the field of view obtained by the human eye by viewing the image of the display screen. In Figure 4(c), the camera field angle θ is large enough to satisfy

Therefore, when the scene is at a distance of d, under the premise of perspective, it can ensure that there is no missing field of view such as black circles.

In Figure 3, the field of view of the scene relative to the camera is

The procedure for calculating α for a given s _{d is} as follows:

1) Find the virtual image size from s _d

2) Find the scene size s _o from s _v :

3) Find the field of view of the scene relative to the camera by s _o :

Therefore, by substituting the size of the display area corresponding to the display screen of a single camera into the formula (2), the minimum angle of view required by the camera can be obtained.

The camera's horizontal field of view is:

The vertical field of view of the camera is:

among them

The smaller the angle of view required for the smaller d _{eo, the} smaller the d _eo minimum is defined as the human eye clear vision distance of 250 mm, and the minimum value of this d _eo is substituted into the formula (3) and the formula (4) to obtain the camera field of view. The corner needs that specific value that cannot be less than.

According to the visual resolution of the human eye and other factors, the resolution of the display screen is selected. When the resolution of the camera is too low, the camera resolution will not match the resolution of the display. The image captured by the camera is interpolated and displayed to the human eye, that is, the image is enlarged. Under the premise of perspective, in order to match the resolution of the upper display (in a certain sense, it is to match the resolution of the human eye), the camera resolution can not be less than a certain specificity without interpolating the image and displaying it to the human eye. value. The method for calculating the resolution of the camera in the embodiment of the present invention is:

Known: heart to the human eye from the optical center of the magnifier d _em, the distance d _mc magnifying the optical center of the camera optical center, a distance d _md magnifying the optical center of the display, magnifying the focal length f _m, corresponding to a single camera display half Screen display area width q _x and height q _y , display screen half screen horizontal maximum number of pixels N _x , vertical maximum number of pixels N _y ;

Seeking: Under the premise of perspective, if the image cannot be interpolated and magnified when the image is displayed on the display, the minimum resolution of the camera is required; the resolution of the camera on the horizontal axis is used.

Representation, resolution on the vertical axis

Express

Solution: The minimum resolution of the camera requires that the camera captures the image and the image on the display is 1:1. The number of pixels in the image captured by the camera is the same as the number of pixels occupied on the display. Therefore, first, the given scene is imaged on the display screen, thereby obtaining the corresponding number of pixels L _d , and then obtaining the corresponding scene size s _o , and then the number of pixels L _c (L _{c )} by the scene size and the scene in the camera imaging. =L _d ) Find the minimum resolution of the camera. The calculation steps are as follows:

1) Given the number of pixels L _d given s _d , first distinguish the horizontal direction from the vertical direction, so that the display

The physical size of a single pixel on the screen is d' _XY , then:

It can also be seen that the number of pixels in the camera is also L _d .

2) The actual size of the scene is s _o ;

Known by formula (1)

3) From the size of the scene s _o and the number of pixels occupied in the camera L _{d to} obtain the camera resolution:

among them,

The distance d _{em of the} human eye and the magnifying glass can be measured when the user wears it. The smaller the d _eo is, the larger the resolution is required. The present invention proposes to define the minimum value of d _eo as the human eye clear distance 250 mm, and this d The minimum value of _eo is substituted into equations (5) and (6) to obtain the specific value that the camera resolution needs to be less than.

Step 600: Power on the smart glasses, generate images by the camera of the smart glasses, perform distortion correction correction on the generated images, and correct the polar lines, so that the images are displayed on the display screen according to a certain mapping relationship;

In step 600, the mapping relationship between the image captured by the camera and the image displayed on the display screen includes: mapping the image captured by the camera to the display screen needs to solve the following problems:

1. The camera lens has certain radial distortion and tangential distortion. The generated image needs to be distorted. The image distortion can be obtained by camera calibration, and the camera internal parameters including the distortion coefficient are obtained, and then the image is dedistorted in the camera; The image distortion elimination method is prior art, and the present invention will not be described again.

2. The two cameras on the smart glasses are often not precisely aligned. There is vertical parallax between the images obtained by the two cameras. If there is a parallax image that is displayed to the human eye, it will give a binocular stereo vision band. It is difficult, so it is necessary to eliminate vertical parallax by polar line correction; this method of eliminating vertical parallax is a prior art, and the present invention will not be described again.

3. After selecting the camera's field of view and resolution, it is often impossible to map the image captured by the camera to the display screen to meet the perspective requirements, but to reduce the image display to the eye. That is, when the camera captures an image, the mapping relationship between the captured image and the image on the display screen is required to achieve perspective, that is, what scale the image is displayed to the human eye. The solution to this scaling value is as follows:

Known: d _em , de _eo , d _mc , d _md , f _m , q _x , q _y , N _x , N _y ,

Seeking: Under the premise of perspective, the camera captures the scaling of the image when the image is displayed on the display.

solution:

Under the premise of satisfying the perspective, the ratio of the number of pixels that the scene should occupy in the display to the number of pixels occupied by the scene in the camera is the image scaling ratio p=(p _x , p _y ), where p _x is horizontal scaling, p _y is the scaling in the vertical direction. The solution process is as follows:

1) Given the size of the scene s _o Find the size of the scene on the display

As shown in Figure 1, the virtual image size of the scene seen by the human eye in the magnifying glass should satisfy:

thereby

Therefore, the size of the scene on the display screen can be further determined as follows:

2) Calculate the image size of the scene on the camera:

3) Calculate the image scaling p

The ratio of the size of the scene on the display to the physical size of a single pixel on the display is the number of pixels occupied by the scene on the display, and the ratio of the size of the scene on the camera to the physical size of a single pixel on the camera's imaging surface is the scene in the camera. The number of pixels occupied on the imaging surface. So: image scaling in the landscape direction

Substituting formulas (8) and (9):

Similarly, the image scaling p _{y is} obtained in the longitudinal direction:

among them

Step 700: Wearing smart glasses, magnifying the image displayed on the display screen through the smart glasses magnifying glass, and extending the image to a distance that the human eye can comfortably watch, thereby realizing the perspective of the smart glasses.

Please refer to FIG. 5 , which is a schematic structural diagram of a smart glasses perspective system according to an embodiment of the present invention. The smart glasses fluoroscopy system of the embodiment of the present invention comprises a camera 1, an arithmetic processing unit 2, a display screen 3 and a magnifying glass 4; wherein the camera 1 comprises two, the two cameras 1 respectively corresponding to the position of the human eye 5; The processing unit 2 is separately connected to the camera 1 and the display screen 3, and the display screen 3 is located between the camera 1 and the magnifying glass 4, and the magnifying glass 4 is located on the side of the smart glasses close to the human eye 5. Camera 1 is used to generate an image of an external scene, and an arithmetic processing unit 2 is used for performing distortion correction, polar line correction and the like on the image generated by the camera 1, and displaying the processed image on the display screen 3 in a certain mapping relationship, and the magnifying glass 4 is used for performing the image displayed on the display screen 3. Zooming in, thereby extending the image to a distance that the human eye 5 can comfortably view, that is, the human eye's clear vision distance, so that the human eye can accurately sense the posture of the external scene as the naked eye when wearing the smart glasses of the embodiment of the present invention. Information such as size.

Specifically, the assembling manner of the smart glasses perspective system of the embodiment of the present invention includes: selecting the size and resolution of the smart glasses display screen 3; setting the focal length of the magnifying glass 4; respectively setting the optical center of the human eye 5 to the optical center of the magnifying glass 4 Distance, the distance from the magnifying glass 4 light to the display 3, and the distance from the magnifying glass 4 to the optical center of the camera 1; after calculating the field of view and resolution of the camera 1, the smart glasses are assembled, and the human eye axis and the magnifying glass are lighted. The optical axis of the axis and the camera 1 are on the same straight line, and the normal direction of the display screen 3 is parallel to the optical axis of the magnifying glass 4, and the center of the half-screen display area corresponding to the single human eye 5 is connected with the optical center of the magnifying glass 4 and the optical axis of the human eye 5 On the same straight line; the smart glasses are powered on, and the camera 1 of the smart glasses generates an image for the external scene, and the distortion correction and the polar line correction are performed on the generated image by the arithmetic processing unit 2, so that the image is displayed in a certain mapping relationship. On the display screen 3; wearing smart glasses, the image displayed on the display screen 3 is enlarged by the smart glasses magnifier 4, and the image is pulled far to the distance that the human eye can comfortably display, and the smart is realized. A perspective view mirror. Wherein, the field of view angle of the selected camera 1 can avoid the lack of field of view such as black border in the field of view of the human eye 5, and the resolution of the selected camera 1 can prevent the image from being interpolated and then enlarged to the human eye 5 for viewing.

In the embodiment of the present invention, as shown in FIG. 2 and FIG. 3, FIG. 2 is a smart glasses parameter identification diagram according to an embodiment of the present invention, and FIG. 3 is a schematic perspective view of the smart glasses according to an embodiment of the present invention. In Fig. 2, a is the distance d _{em of the} human eye 5 light center to the optical center of the magnifying glass 4, b is the distance d _md of the magnifying glass 4 to the display screen 3, and c is the distance d from the optical center of the magnifying glass 4 to the optical center of the camera 1. _Mc . In FIG. 3, the horizontal axis of the solid line represents the human eye 5, the magnifier 4, the optical axis of the camera 1, and the normal of the display screen 3, wherein the human eye 5 is at the e point of the horizontal axis, the magnifying glass 4 is at the m point, and the display screen 3 At point d, the distance from f1 to m is the focal length of magnifier 4, camera 1 is at point c, the distance from f2 to c is the focal length of camera 1, and point v is the image displayed on display screen 3 corresponding to the virtual image in magnifier 4. Position, point o is the location of the scene given by the outside world. The imaging of the scene in the human eye 5 and the imaging of the naked eye scene in the human eye 5 will be the same, thereby achieving fluoroscopy.

Under the premise that the assembly requirements of the smart glasses have been met, it is further assumed that the display screen 3, the magnifying glass 4, the distance d _{md of the} optical center of the magnifying glass 4 and the display screen 3, and the magnifying glass have been selected according to factors such as the visual characteristics of the human eye 5. 4 The distance between the optical center and the optical center of the camera 1 d _mc , the distance from the human eye 5 to the optical center of the magnifying glass 4 d _em , so that the following smart glasses parameters are known: the distance d _{mc of the} optical center of the magnifying glass 4 and the optical center of the camera 1 The distance d _{md of the} magnifying glass 4 and the display screen 3, the distance d _{em of the} human eye 5 light center and the optical center of the magnifying glass 4, the focal length f _{m of the} magnifying glass 4, the half screen display area width of the display screen 3 corresponding to the single camera 1 is q _x The height is q _y , the maximum number of pixels in the horizontal half of the screen 3 is N _x , and the maximum number of pixels in the vertical direction is N _y , so that the field of view and resolution of the camera 1 are calculated according to the above parameters;

Wherein, the way of calculating the angle of view of the camera 1 is:

Known: the distance from the human eye 5 to the magnifying glass 4 light center d _em , the distance from the magnifying glass 4 to the optical center of the camera 1 d _mc , the distance from the magnifying glass 4 to the display 3 d _md , the magnifying glass 4 focal length f _m The display screen 3 corresponding to the single camera 1 displays the area width q _x and the height q _{y in} half screen;

Find: Under the premise of perspective, the minimum angle of view α required by camera 1 is α=(α _x , α _y ), where α _x is the horizontal field of view angle and α _y is the vertical field of view angle;

Solution: When the image of the external scene on the display screen 3 just fills the display screen 3, the field of view of the scene corresponding to the camera 1 is the minimum angle of view required by the camera 1. Therefore, it is known that the image size on the display screen 3 is the size of the half screen image display area of the display screen 3, and the object is at the angle of view corresponding to the camera 1, and the value of the obtained angle of view is the value to be solved. For convenience of explanation, let d _eo be the distance from the human eye 5 to the scene, d _co is the distance from the center of the camera 1 to the scene, f _c is the focal length of the camera 1 , and the horizontal physical size of the camera 1 is dX and the vertical dimension is dY , d _{ev is} the distance from the human eye 5 to the virtual image, d _mv is the distance from the optical center of the magnifying glass 4 to the virtual image of the scene, d _vo is the distance from the virtual image of the scene to the scene, and the size of the scene in the display 3 is s _d , the corresponding virtual image size is s _v , the size of the scene is s _o , the size of the scene in the camera 1 is s _c , the horizontal physical size of the single pixel on the display 3 is dx, and the vertical size is dy. Where: d _co =d _eo -d _em -d _mc ,

d _vo =d _eo -d _em -d _mv ,d _ev =d _em +d _mv .

When the field of view of the camera 1 is too small, under the premise of satisfying the perspective, it may happen that the image obtained by the camera 1 cannot cover the full display 3 (in a certain sense, it is to cover the field of view of the full eye 5), resulting in wearing In the perspective method and system of the smart glasses, the field of view of the human eye 5 appears with a missing field of view such as black edges. Under the premise of satisfying the perspective, in order to ensure that the image obtained by the camera 1 can cover the full display 3, the field of view of the camera 1 needs to be smaller than a certain value, as shown in Fig. 4(a), Fig. 4(b), Fig. 4 (c) is a schematic diagram of the calculation method of the camera 1 field of view. The solid horizontal axis represents the common optical axis of the human eye 5, the magnifying lens 4, and the camera 1, and the normal of the display screen 3. On the horizontal axis of Fig. 4(a), a is the human eye 5, b is the magnifying glass 4, c is the display screen 3, d is the virtual image of the image displayed on the display screen 3 in the magnifying glass 4, and the length of the vertical line at c represents the display The size of the half-screen image display area of the screen 3 is, under the premise of the perspective, the field of view of the human eye 5 obtained by viewing the image on the display screen 3 outside the d distance is V. On the horizontal axis of Fig. 4(b), one more camera 1 is compared to Fig. 4(a), and when the field of view of the camera 1 is θ, the field of view of the camera 1 is W. To avoid the loss of the field of view such as the black side of the human eye 5 field of view, it is necessary to include as much as possible the field of view of the camera 1 by viewing the image of the display screen 3 by the human eye 5. In Fig. 4(c), the camera 1 angle of view θ is large enough to satisfy

Therefore, when the scene is at a distance d, under the premise of perspective, it can be ensured that the human eye 5 does not see the black field and other fields of view missing.

In FIG. 3, the field of view of the scene relative to the camera 1 is

The procedure for calculating α for a given s _{d is} as follows:

1) Find the virtual image size from s _d

2) Find the scene size s _o from s _v :

3) Find the field of view of the scene relative to the camera 1 by s _o :

Therefore, the minimum angle of view required by the camera 1 can be obtained by substituting the display size of the display unit 3 corresponding to the display area of the single camera 1 into the formula (2).

The horizontal field of view of camera 1 is:

The vertical field of view of camera 1 is:

among them

The smaller the field of view required for the smaller d _{eo, the} smaller the d _eo minimum is the human eye clear vision distance of 250 mm, and the minimum value of this d _eo is substituted into the formula (3) and the formula (4) to obtain the camera 1 view. The field angle needs that specific value that cannot be less than.

After the resolution of the display screen 3 is selected according to factors such as the visual resolving power of the human eye, when the resolution of the camera 1 is too low, the resolution of the camera 1 does not match the resolution of the display screen 3, and the camera 1 needs to be The captured image is displayed and displayed to the human eye, that is, the enlarged image. Under the premise of perspective, in order to match the resolution of the upper display 3 (in a certain sense, it is to match the resolution of the human eye), if the image is not displayed and displayed to the human eye, the resolution of the camera 1 cannot be less than A specific value.

The manner of calculating the resolution of the camera 1 in the embodiment of the present invention is:

Known: the distance from the human eye 5 to the magnifying glass 4 light center d _em , the distance from the magnifying glass 4 to the optical center of the camera 1 d _mc , the distance from the magnifying glass 4 to the display 3 d _md , the magnifying glass 4 focal length f _m The display screen 3 corresponding to the single camera 1 has a half screen display area width q _x and a height q _y , a display screen 3 half screen horizontal maximum pixel number N _x , and a vertical maximum pixel number N _y ;

Seeking: Under the premise of perspective, if the image cannot be interpolated and magnified when the image is displayed on the display 3, the minimum resolution of the camera 1 is required; the resolution of the camera 1 on the horizontal axis is used.

Representation, resolution on the vertical axis

Express

Solution: The minimum resolution of camera 1 requires that the image captured by camera 1 and the image on display 3 be displayed 1:1. The number of pixels in the image captured by camera 1 is the same as the number of pixels occupied on display screen 3. Therefore, first, the given scene is imaged on the display screen 3, thereby obtaining the corresponding number of pixels L _d , and then obtaining the corresponding scene size s _o , and then the number of pixels L _c (the number of pixels in the image of the scene 1 and the subject in the camera 1) L _c = L _d ) Find the minimum resolution of the camera 1. The calculation steps are as follows:

The physical size of a single pixel on screen 3 is d' _XY , then:

Therefore, it is also known that the number of pixels in the camera 1 is also L _d .

2) The actual size of the scene is s _o ;

Known by formula (1)

3) From the scene size s _o and the number of pixels occupied in the camera 1 L _{d to} obtain the resolution of the camera 1:

among them,

The distance d _{em of the} human eye 5 light center and the magnifying glass 4 light center can be measured when the user wears, and the smaller the d _eo is, the larger the resolution is required. The present invention proposes to define the minimum value of d _eo as the human eye 5 clear vision distance 250 mm. , the minimum value of d _eo this into equation (5) and (6) can be obtained that the camera need not be less than a specific value of a resolution.

In the embodiment of the present invention, the mapping relationship between the image captured by the camera 1 and the image displayed on the display screen 3 includes: mapping the image captured by the camera 1 to the display screen 3 needs to solve the following problems:

1. There is a certain radial distortion and tangential distortion of the camera 1 lens. The generated image needs to be distorted. The image distortion can be corrected by the camera 1 to obtain the internal parameters of the camera 1 including the distortion coefficient, and then the camera 1 is de-distorted. The image distortion elimination method is prior art, and the present invention will not be described again.

2. The two cameras 1 on the smart glasses are often not precisely aligned. There is vertical parallax between the images obtained by the two cameras 1. If the image with parallax is displayed to the human eye 5, it will give a double Stereoscopic vision brings difficulties, so it is necessary to eliminate vertical parallax by polar line correction; this method of eliminating vertical parallax is a prior art, and the present invention will not be described again.

3. After selecting the camera 1 field of view and resolution, it is often impossible to make the image captured by camera 1 1:1 mapped to display 3 to satisfy the perspective, but to reduce the image display to the human eye 5 Do perspective. That is, after the image is acquired by the camera 1, the mapping relationship between the captured image and the image on the display screen 3 is required to achieve the perspective, that is, what scale the image is displayed to the human eye. The solution to this scaling value is as follows:

Known: d _em , de _eo , d _mc , d _md , f _m , q _x , q _y , N _x , N _y ,

Seeking: Under the premise of perspective, the camera 1 captures the zoom ratio of the image when the image is displayed on the display screen 3.

solution:

Under the premise of satisfying the perspective, the ratio of the number of pixels that the scene should occupy in the display screen 3 to the number of pixels occupied by the scene in the camera 1 is the image scaling ratio p=(p _x , p _y ), where p _x is horizontal The scaling in the direction, p _y is the scaling in the vertical direction. The solution process is as follows:

1) Given the size of the scene s _o Find the scene in the size of the display 3

The virtual image size of the scene seen by the human eye 5 in the magnifying glass 4 as shown in Fig. 1 should satisfy:

Thus

Therefore, the size of the scene on the display screen 3 can be further determined as follows:

2) Calculate the image size of the scene on the camera 1 as:

3) Calculate the image scaling p

The ratio of the size of the scene on the display screen 3 to the physical size of a single pixel on the display screen 3 is the number of pixels occupied by the scene on the display screen 3, and the size of the scene on the camera 1 and the physical size of a single pixel on the imaging surface of the camera 1 The ratio is the number of pixels the scene occupies on the imaging surface of camera 1. and so:

Image scaling in landscape orientation

Substituting formulas (8) and (9):

Similarly, the image scaling p _{y is} obtained in the longitudinal direction:

among them

The smart glasses perspective method and system of the embodiment of the invention image the external scene through the camera, and the generated image is displayed through the display screen after processing, and the display screen is displayed through the magnifying glass. The image on the image is enlarged to extend the image to a distance that the human eye can comfortably view, so that when the human eye wears the smart glasses, it accurately senses the pose and size information of the external scene as the naked eye, and improves the wearer's comfort and Watch the accuracy.

The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and combinations thereof may be made without departing from the spirit and scope of the invention. Simplifications should all be equivalent replacements and are included in the scope of the present invention.

Claims

A smart glasses perspective method includes the following steps:

Step a: Assembling the smart glasses so that the human eye axis, the magnifier optical axis, and the camera optical axis are on the same straight line, the normal direction of the display screen is parallel to the optical axis of the magnifying glass, and the center of the half-screen display area corresponding to a single human eye and the light source of the magnifying glass The line is on the same line as the human eye axis;

Step b: running the smart glasses, and generating an image on the external scene through the camera, so that the image is displayed on the display screen according to a certain mapping relationship;

Step c: Wear smart glasses, magnify the image displayed on the display through a magnifying glass, and zoom out the image to a distance that the human eye can comfortably view.
The method for fluoroscopy of the smart glasses according to claim 1, wherein the step a further comprises: selecting a size and a resolution of the smart glasses display screen; setting a focal length of the magnifying glass; and respectively setting a human eye to a magnifying glass The distance of the optical center, the distance from the magnifying glass to the display, and the distance from the optical center of the magnifier to the optical center of the camera.
The method for fluoroscopy of the smart glasses according to claim 2, wherein the front of the a further comprises: calculating a required camera angle of view for avoiding a missing field of view of the human eye, and calculating the target resolution of the display screen. The required camera resolution.
The method for fluoroscopy of a smart glasses according to claim 3, wherein the calculating the angle of view of the camera is:

The camera's horizontal field of view is:

The vertical field of view of the camera is:

In the above formula, d em is the distance from the human eye to the optical center of the magnifying glass, d mc is the distance from the optical center of the magnifying glass to the optical center of the camera, d md is the distance from the optical center of the magnifying glass to the display screen, and f m is the focal length of the magnifying glass, q x And q y is the half-screen display area width and height of the display corresponding to a single camera, d eo is the distance from the human eye to the scene, d co is the distance from the camera to the scene, f c is the camera focal length, and dX is the camera single pixel Horizontal physical size, dY is the vertical dimension of the camera's single pixel, d ev is the distance from the human eye to the virtual image, d mv is the distance from the optical center of the magnifying glass to the virtual image of the scene, d vo is the distance from the virtual image of the scene to the scene, and s d is the scene in the display Size, s v is the corresponding virtual image size, s o is the size of the scene, s c is the size of the scene in the camera, dx is the horizontal physical size of a single pixel on the display, and dy is the vertical size of a single pixel on the display.
The method for fluoroscopy of a smart glasses according to claim 4, wherein: The calculation formula for calculating the camera resolution is:

In the above formula, N x is the horizontal maximum pixel number corresponding to the half-screen display area of one eye, and N y is the maximum vertical pixel number.
The method for fluoroscopy of a smart glasses according to any one of claims 1 to 5, wherein in the step b, the image is displayed on the display screen in a certain mapping relationship, that is, the image is scaled at a certain scale. Displayed to the human eye; the method for solving the image scaling ratio includes: the ratio of the number of pixels that the scene should occupy in the display screen to the number of pixels occupied by the scene in the camera is the image scaling ratio p= (p x , p y ), where p x is the scaling in the horizontal direction and p y is the scaling in the vertical direction; the ratio of the size of the scene on the display screen to the physical size of a single pixel on the display screen is that the scene is displayed The number of pixels occupied on the screen, and the ratio of the image size of the scene on the camera to the physical size of a single pixel on the image plane of the camera is the number of pixels occupied by the scene on the imaging surface of the camera, namely:

The image scaling p x in the landscape direction is:

The image scaling p y in the portrait direction is:
A smart glasses fluoroscopy system, comprising: a camera, a display screen and a magnifying glass; the camera corresponding to a position of a human eye, the display screen being located between the camera and the magnifying glass, the magnifying glass being located in the smart glasses close to the human eye On one side, and the human eye axis, the magnifier optical axis and the camera optical axis are on the same line, the normal direction of the display screen is parallel to the optical axis of the magnifying glass, and the center of the half-screen display area corresponding to a single human eye is connected with the optical center of the magnifying glass. The human eye axis is on the same straight line; the camera is used to generate an image on the external scene, and the image is displayed on the display screen in a certain mapping relationship, and the magnifying glass is used to enlarge the image displayed on the display screen, The image is zoomed out to the distance that the human eye can comfortably view.
The smart glasses fluoroscopy system according to claim 7, further comprising an arithmetic processing unit, wherein the arithmetic processing unit is respectively connected to the camera and the display signal. It is used to perform distortion correction and polar line correction processing on the image generated by the camera, and display the processed image through the display screen.
The smart glasses fluoroscopy system according to claim 8, wherein the camera angle of view is calculated as:

The camera's horizontal field of view is:

The vertical field of view of the camera is:

The calculation formula of the camera resolution is:

In the above formula, d em is the distance from the human eye to the optical center of the magnifying glass, d mc is the distance from the optical center of the magnifying glass to the optical center of the camera, d md is the distance from the optical center of the magnifying glass to the display screen, and f m is the focal length of the magnifying glass, q x And q y is the half-screen display area width and height of the display corresponding to a single camera, d eo is the distance from the human eye to the scene, d co is the distance from the camera to the scene, f c is the camera focal length, and dX is the camera single pixel lateral physical size, dY longitudinal camera single pixel size, d ev man from the eye to the virtual image, d mv of magnifying the optical center from the scene of the virtual image, d vo is the scene of the virtual image to the scene distance, s d to display the scene Size, s v is the corresponding virtual image size, s o is the size of the scene, s c is the size of the scene in the camera, dx is the horizontal physical size of a single pixel on the display, dy is the vertical size of a single pixel on the display; N x is a single eye Corresponding to the horizontal maximum number of pixels in the half-screen display area, N y is the maximum number of pixels in the vertical direction.
The smart glasses fluoroscopy system according to any one of claims 7 to 9, wherein the image is displayed on the display screen according to a certain mapping relationship, specifically: displaying the image to the human eye at a certain scaling ratio; The method for solving the image scaling ratio includes: the ratio of the number of pixels that the scene should occupy in the display screen to the number of pixels occupied by the scene in the camera is the image scaling ratio p=(p x , p y ), where p x is horizontal The scaling in the direction, p y is the scaling in the vertical direction; the ratio of the size of the scene on the display to the physical size of a single pixel on the display is the number of pixels the scene occupies on the display, and the scene is imaged on the camera. The ratio of the size to the physical size of a single pixel on the camera's imaging surface is the number of pixels the scene occupies on the camera's imaging surface, ie:

The image scaling p x in the landscape direction is:

The image scaling p y in the portrait direction is: