CN106846311B - Positioning and AR methods and systems and applications based on image recognition - Google Patents

Abstract

The present invention relates to the field of image recognition and positioning, and in particular to a positioning and AR method, system and application based on image recognition. It is characterized in that: the position at the viewing angle is determined by two viewing angles of at least two determined feature objects, or the position at the viewing angle is determined by the ratio of a viewing angle of at least one determined object and its viewing angle direction projection surface of the object. The beneficial effects are: the purpose of the present invention is to provide an angle positioning method, which can be applied to navigation positioning, AR navigation positioning, AR game positioning, etc. Positioning can be achieved in any unfamiliar environment. Suitable for indoor positioning.

Description

Positioning and AR methods and systems and applications based on image recognition

Technical field

The present invention relates to the field of image recognition and positioning, and in particular to a positioning and AR method, system and application based on image recognition.

Background technique

Currently, positioning navigation based on image recognition generally compares the photo taken at the positioning point with one of the preset photos to obtain the location; or establishes a 3D model map and compares the photo taken at the positioning point with the 3D model map. Get the location. Both methods require a pre-set photo database or 3D model database, and positioning cannot be performed in any location without a database.

Augmented Reality (AR) is a technology that calculates the position and angle of camera images in real time and adds corresponding images, videos, and 3D models. The goal of this technology is to put a virtual world on the screen. real world and interact with it.

For AR technology applied to positioning, only the angle relationship can be considered and the distance relationship can be ignored.

For indoor positioning, only the angle relationship can be considered and the distance relationship can be ignored.

The present invention is based on image recognition, including existing mature feature recognition, object contour recognition, color recognition, action recognition, face recognition, text recognition, etc., and adopts corresponding recognition strategies in different environments.

Contents of the invention

The purpose of the present invention is to provide an angle positioning method, which can be applied to navigation positioning, AR navigation positioning, AR game positioning, etc.

The idea of the present invention is to determine the position of the viewing angle based on two viewing angles of two determined objects (feature objects), or one viewing angle of a determined object and the projection of two line segments or two surfaces in the object in the viewing angle direction. The line segment ratio or area ratio determines the position at the viewing angle.

Referring to Figure 2, the position of O at the viewing angle is determined based on the two viewing angles β1 and β2 of the two determined objects (features) A and B.

Or referring to Figure 3, a line segment ratio D2E2/E2F2 that determines a viewing angle β3 of an object and the projection of two line segments DE and EF in the object in the viewing angle direction determines the position O at the viewing angle.

Or referring to Figure 4, a determined object's viewing angle β4 and the area ratio SO1/SO2 of the projections of two surfaces S1 and S2 in the object in the viewing angle direction determine the position O at the viewing angle.

Feature objects are objects that can be clearly identified through image recognition in the computer system, or certain objects that are artificially specified in the computer system. Feature objects do not refer to special objects. Feature objects can be any objects in the positioning space, but they must be identified through image recognition in the computer system. For example, two mobile phones shoot the number "5" in different locations, and the computer system Image recognition (number recognition strategy) determines that the two mobile phones captured the same number "5", so the number "5" is a characteristic object. Or a relatively stable and fixed object or pattern, which is discovered and identified by the image recognition software of the computer system.

Specific to the client, that is, the two clients respectively obtain the images around their respective locations and the direction and angle when shooting and send them to the server. The server determines the image of the common object captured by the two clients through image recognition and uses the difference in the image of the common object. and its direction angle to calculate the direction angle of the line connecting the two clients.

Further, the two position points obtain the direction angles of at least two feature objects respectively, and calculate the direction angle of the line connecting the two position points through the direction angles.

Or, two position points respectively obtain the ratio and direction angle of two line segments that are not on the same straight line, and calculate the direction angle of the line connecting the two position points through the ratio and direction angle.

Or, two position points respectively obtain the area ratio and direction angle of two surfaces of the same object (two surfaces that are not on the same plane), and calculate the direction angle of the line connecting the two position points through the area ratio and direction angle.

Further, provide a method and system for integrating the virtual world into the real world on the screen and interacting with it (friend screen positioning, merchant screen positioning, game target screen positioning, red envelope screen positioning, and virtual advertising screen positioning). The system can be Based on an independent instant messaging IM platform, or through third-party services (API), or embedded in existing IM platforms, such as QQ, WeChat, Momo, etc.

The technical solution adopted by the present invention is:

A positioning method based on image recognition, characterized in that: the position at the viewing angle is determined from at least two viewing angles of the determined object (feature object), or at least one viewing angle of the determined object and two of the objects. The line segment ratio or area ratio of the projection of a line segment or two surfaces in the viewing direction determines the position at the viewing angle.

The described positioning method based on image recognition is also characterized in that two position points obtain the direction angles of at least two objects (features) respectively, and the direction angle of the line connecting the two position points is calculated through the direction angles.

The described positioning method based on image recognition is also characterized in that: two position points respectively obtain the ratio and direction angle of two line segments that are not on the same straight line, and calculate the connection between the two position points through the ratio and the direction angle. The direction angle of the line.

The described positioning method based on image recognition is also characterized by: two position points obtain the area ratio and direction angle of two faces of the same object respectively, and calculate the line connecting the two position points through the area ratio and direction angle. direction angle.

The described positioning method based on image recognition also includes a client and a server, and is characterized in that it includes the steps:

(1) Determine the unified baseline through the client's direction device;

(2) Position C obtains the image A1 of object (feature) A and the angle α1 relative to the baseline. Position C obtains the image B1 of object (feature) B and the angle α2 relative to the baseline. Upload images A1 and α1. , images B1 and α2 to the server;

(3) Position O obtains the image A2 of object (feature) A and the angle β1 relative to the baseline, position O obtains the image B2 of object (feature) B and the angle β2 relative to the baseline, upload images A2 and β1 , images B2 and β2 to the server;

(4) The server determines that image A1 and image A2 are both images of object (feature) A through image recognition. The server determines that image B1 and image B2 are both images of object (feature) B through image recognition. Through α1, α2, β1 and β2 are calculated to obtain the angle γ of the OC straight line relative to the reference line.

That is: γ=F1 (α1, α2, β1, β2), F1 is the calculation function.

Of course, it is beneficial to the present invention to obtain more objects (features) for calculation and comprehensive calculation results.

or,

The described positioning method based on image recognition also includes a client and a server, and is characterized in that it includes the steps:

(1) Determine the unified baseline through the client's direction device;

(2) The line segment DE and the line segment EF of the line segment object DEF are not on the same straight line. Position C obtains the vertical projection image D1E1F1 of the line segment object DEF and the angle α3 between the vertical projection direction and the baseline, and uploads the images D1E1F1 and α3 to the server;

(3) Position O obtains the vertical projection image D2E2F2 of the line segment object DEF and the angle β3 between the vertical projection direction and the reference line, and uploads the images D2E2F2 and β3 to the server;

(4) The server determines through image recognition that both the image D1E1F1 and the image D2E2F2 are images of the line segment DEF, and calculates the angle γ of the OC straight line relative to the baseline through the D1E1/E1F1 ratio, D2E2/E2F2 ratio, α3, and β3.

That is: γ=F2 (D1E1/E1F1, α3, D2E2/E2F2, β3), F2 is the calculation function.

Of course, it is beneficial to the present invention to obtain more line segment objects for calculation and comprehensive calculation results.

or,

The described positioning method based on image recognition also includes a client and a server, and is characterized in that it includes the steps:

(1) Determine the unified baseline through the client's direction device;

(2) The areas of the two faces of the three-dimensional object G are S1 and S2. At position C, obtain the vertical projection image G1 of the three-dimensional object G and the angle α4 between the vertical projection direction and the reference line, and upload the images G1 and α4 to the server;

(3) Position O obtains the vertical projection image G2 of the three-dimensional object G and the angle β4 between the vertical projection direction and the reference line, and uploads the images G2 and β4 to the server;

(4) The server determines through image recognition that both image G1 and image G2 are images of the three-dimensional object G. In the G1 image, SC1 is the image area of S1, SC2 is the image area of S2, in the G2 image SO1 is the image area of S1, and SO2 is For the image area of S2, the angle γ of the OC straight line relative to the baseline is calculated through the SC1/SC2 ratio, SO1/SO2 ratio, α4, and β4.

That is: γ=F3 (SC1/SC2, α4, SO1/SO2, β4), F3 is the calculation function.

Of course, it is beneficial to the present invention to obtain more three-dimensional objects for calculation and comprehensive calculation results.

Of course, the above-mentioned feature objects, line segment objects, and three-dimensional objects can be comprehensively calculated.

Further, the positioning method based on image recognition is further characterized by comprising the steps:

The client obtains the angle γ calculated by the server, and locates or navigates on the client's guidance map based on the angle γ.

Or, the described positioning method based on image recognition is also characterized by including the steps: the client obtains the angle γ calculated by the server, the client turns on the camera to obtain the real-scene image, and superimposes the guide icon of the angle γ in the real-scene image. .

Furthermore, the angle obtained by the client also includes the horizontal inclination angle in the vertical direction, which is used to determine the direction angle of the vertical plane (the same principle can be demonstrated). In this way, combined with the above-mentioned direction angle of the horizontal plane, the direction angle of the three-dimensional space is determined.

An AR method based on image recognition positioning, characterized in that the angle γ guides mutual friends, merchants, game targets, red envelopes, or advertisements.

The described AR method based on image recognition and positioning is characterized by:

①Client 1 starts the red envelope publishing program, captures surrounding images and simultaneously obtains the direction angle and uploads it to the server;

②The server obtains the surrounding image and its direction angle taken by client 1;

③The server generates a virtual red envelope image based on the red envelope settings of client 1 and associates the image and its direction angle obtained by the server in step ②. The red envelope action assignment includes the direction angle, horizontal inclination angle, vertical height, and changes in its action assignment;

④Client 2 captures surrounding images and simultaneously obtains direction angles and uploads them to the server;

⑤ Based on the surrounding image and its direction angle taken by client 1 obtained in step ②, and the surrounding image and its direction angle taken by client 2 in step ④, the server calculates and obtains the location where client 1 releases the red envelope, that is, the location of the red envelope. The server Push the virtual red envelope image to client 2;

⑥Client 2 superimposes the virtual red envelope image and displays it on the real screen according to the position of the red envelope. At the same time, the virtual red envelope image moves according to the action assignment;

⑦Client 2 obtains red envelopes through the touch screen.

The described AR method based on image recognition and positioning is characterized by:

The steps for the server to publish advertisements are:

①The server obtains the surrounding image and its direction angle taken by any client;

②The server generates a virtual advertising image based on the advertiser's settings and associates the image and its direction angle obtained by the server in step ①. The advertiser's settings include placement, text, images, and image actions;

③The advertising receptor client captures surrounding images and simultaneously obtains the direction angle and uploads it to the server;

④The server calculates and obtains the advertising placement position based on the surrounding image and its directional angle captured by any client obtained in step ①, and the surrounding image and directional angle captured by the advertising receptor client in step ③;

⑤The advertising receptor client overlays and displays the virtual advertising image on the real screen according to the placement position of the advertisement. At the same time, the virtual advertising image moves according to the action assignment;

⑥The advertising receptor client obtains advertising content through the touch screen, such as links, jumps, collections, etc.

Further, the described positioning method based on image recognition is characterized in that it also includes the step of: determining the distance between two feature objects, or the size of a line segment object, or the size of a three-dimensional object, and calculating it through trigonometric functions. The distance between two locations.

A positioning system based on image recognition, characterized by: including a server and a client,

Servers include:

Image recognition unit, used to determine feature objects, line segment objects or three-dimensional objects;

The calculation unit calculates the direction angle of the line connecting the two position points based on the direction angles of the two features; or calculates the direction angle of the line connecting the two position points based on the ratio of two line segments that are not on the same straight line and their direction angle; Or calculate the direction angle of the line connecting the two position points based on the area ratio of the two surfaces of the three-dimensional object and its direction angle;

The calculation result push unit pushes the calculated direction angle data of the line connecting the two position points to the client;

The client includes at least a direction acquisition unit, an image overlay unit, and a camera.

The feature of the positioning system based on image recognition is that the system is embedded in an existing IM system, payment system or game system.

The described positioning system based on image recognition is characterized in that: the client is a mobile phone or a tablet computer.

The present invention is based on the specific implementation of a mobile phone. Currently, mobile phone sensors include acceleration sensors, direction devices, gyroscopes, thermometers, etc. The information that can be obtained includes acceleration, magnetic field, rotation vector, gyroscope, light induction, pressure, temperature, proximity ,gravity.

The beneficial effects of the present invention are: The purpose of the present invention is to provide an angle positioning method, which can be applied to navigation positioning, AR navigation positioning, AR game positioning, etc. Positioning can be achieved in any unfamiliar environment. Suitable for indoor positioning.

Description of the drawings

Figure 1 is a schematic diagram of the present invention using two features to determine the angle of the positioning point.

Figure 2 is a geometric diagram of the present invention using two features to determine the angle of the positioning point.

Figure 3 is a schematic diagram of the present invention using line segments to determine the angle of the anchor point.

Figure 4 is a schematic diagram of the present invention using three-dimensional objects to determine the angle of the positioning point.

Figure 5 is a schematic interface diagram of AR positioning according to the present invention.

Figure 6 is a hardware configuration diagram of the present invention.

Figure 7 is a flow chart of the present invention.

Figure 8 is an AR application flow chart of the present invention.

Figure 9 is an example of angle calculation according to the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and examples.

Figure 1 is a schematic diagram of the present invention using two features to determine the angle of the positioning point. 101 is a mobile phone, 101 is at point O, 102 is another mobile phone, 102 is at point C, in the positioning space, 103 is the feature A, and 104 is the feature Object B, the characteristic object is an object that can be clearly determined by the server through image recognition, or an artificially set marker (the marker is registered in the server for identification, such as a certain advertising image, text, etc.), 105 is the server. Assume that mobile phone 101 is looking for the position of mobile phone 102 (that is, the angle at which OC is found). First, mobile phone 102 can rotate and take pictures of the positioning space. Specifically, for example, 360-degree continuous camera shooting along the horizontal line can simultaneously record the direction angle and upload it to the server at the same time. The image of feature A must be obtained. and the direction angle (viewing angle) when shooting feature A. At this time, the direction of feature A is perpendicular to the screen of mobile phone 102 (for simplicity, only the horizontal plane is considered for the time being, and the vertical plane is calculated using the inclination angle. The calculation method can be verified in the same way), That is, the image of feature A falls in the center of the screen of mobile phone 102, and the direction and baseline of feature A are obtained according to the direction sensor (magnetic element) in the mobile phone (the baseline is determined by the determined north-south angle line. For simplicity, in the figure The east-west direction is the angle between the baseline (that is, the WE direction, or the OX direction) (that is, the direction angle when shooting the image). In the same way, feature object B is shot, and the shot image and its direction angle are uploaded to the server; the mobile phone 101 pairs the positioning space Rotate and take photos to find the 102 mobile phone, and also obtain the images of the two features and their direction angles and upload them to the server. The server calculates the unique OC direction angle γ (and the reference angle) based on the direction angles of the two features at the two locations through geometric mathematics. The angle between line OX), refer to Figure 2. The present invention uses two features to determine the geometric diagram of the angle of the positioning point. The steps are:

(1) Determine the unified baseline through the client's direction device;

(2) Position C obtains the image A1 of feature A and the angle α1 relative to the reference line. Position C obtains the image B1 of feature B and the angle α2 relative to the reference line. Upload images A1 and α1, images B1 and α2 to server;

(3) Position O obtains the image A2 of feature A and the angle β1 relative to the reference line. Position O obtains the image B2 of feature B and the angle β2 relative to the reference line. Upload images A2 and β1, images B2 and β2 to server;

(4) The server determines that image A1 and image A2 are both images of feature A through image recognition. The server determines that image B1 and image B2 are both images of feature B through image recognition. The OC is obtained through α1, α2, β1, and β2 calculations. The angle γ of the straight line relative to the datum line.

That is: γ=F1 (α1, α2, β1, β2), F1 is the calculation function.

Furthermore, the distance between the two features is determined, and the distance between the two mobile phones can be calculated through trigonometric functions.

The following provides a calculation process with reference to Figure 9:

In triangle ABC:

AB/sin(α1+α2)=AC/sin(δ-α2)=BC/sin( -α1-δ)=BC/sin(α1+δ), deriving AC=ABsin(δ-α2)/sin(α1+α2);

In triangle AOB:

AB/sin(β1-β2)=OB/sin( -δ-β1)=OB/sin(δ+β1)=OA/sin(δ+β2), deriving OA=ABsin(δ+β2)/sin(β1-β2);

In triangle AOC:

OA/sin( -γ+α1)=OA/ sin(γ-α1)=AC/sin(γ-β1),

Substitute to find: ABsin(δ+β2)/ sin(β1-β2)sin(γ-α1)= ABsin(δ-α2)/sin(α1+α2)sin(γ-β1), sin(γ-α1) /sin(γ-β1)= sin(α1+α2)sin(δ+β2)/sin(δ-α2) sin(β1-β2);

Let K= sin(α1+α2) sin(δ+β2)/sin(δ-α2)sin(β1-β2), then sin(γ-α1)/ sin(γ-β1)=K;

sinγcosα1-cosγsinα1=Ksinγcosβ1-Kcosγsinβ1;

(Ksinβ1-sinα1)cosγ=(Kcosβ1-cosα1)sinγ;

cosγ= sinγ(Kcosβ1-cosα1)/(Ksinβ1- sinα1);

Let (Kcosβ1-cosα1)/(Ksinβ1- sinα1)=t, then cosγ= tsinγ;

Since cos ² γ+sin ² γ=1, so t ² sin ² γ+sin ² γ=1;

sin ² γ=1/(1+t ² )，γ∈[0, ] when sinγ≥0, sinγ=root (1/(1+t ² ));

Finally, find the γ angle.

Figure 3 is a schematic diagram of the present invention using line segment objects to determine the angle of the positioning point. 101 is a mobile phone, 101 is at point O, 102 is another mobile phone, and 102 is at point C. In the positioning space, DEF is a line segment object, and line segment DE and line segment EF are not there. On the same straight line, it can specifically be a line segment on an object, which can be recognized and determined by computer software, that is, it is an object that the server can clearly determine through image recognition, or an artificially set marker (the marker is registered in the server and is used for identification), 105 is the server. Assume that mobile phone 101 is looking for the position of mobile phone 102 (that is, the angle at which OC is found). First, mobile phone 102 can rotate and take pictures of the positioning space, obtain the image of the line segment object DEF and the direction angle (viewing angle) when shooting the line segment object DEF. At this time, the line segment object DEF The direction is perpendicular to the screen of the mobile phone 102 (for simplicity, only the horizontal plane will be considered for now, the calculation of the vertical plane can be verified in the same way), that is, the image of the line segment object DEF falls in the center of the screen of the mobile phone 102. According to the direction sensor (magnetic) in the mobile phone component) to obtain the angle between the line segment object DEF and the baseline (the baseline is determined by the determined north-south angle line. For simplicity, the east-west direction is used as the baseline in the figure, that is, the WE direction, or the OX direction) (that is, the direction when the image is taken) angle), the shooting is completed and the captured image and its direction angle are uploaded to the server; the mobile phone 101 rotates and takes pictures in the positioning space to find the mobile phone 102, and also obtains the image of the line segment object DEF and its direction angle and uploads it to the server, and the server uses the same line segment in the two locations The line segment ratio and direction angle of the object DEF image are calculated through geometric mathematics to calculate the unique direction angle γ of OC (the angle between the baseline OX and the baseline OX). The steps are:

(1) Determine the unified baseline through the client's direction device;

(2) The line segment DE and the line segment EF of the line segment object DEF are not on the same straight line. Position C obtains the vertical projection image D1E1F1 of the line segment object DEF and the angle α3 between the vertical projection direction and the baseline, and uploads the images D1E1F1 and α3 to the server;

(3) Position O obtains the vertical projection image D2E2F2 of the line segment object DEF and the angle β3 between the vertical projection direction and the reference line, and uploads the images D2E2F2 and β3 to the server;

(4) The server determines through image recognition that both the image D1E1F1 and the image D2E2F2 are images of the line segment DEF, and calculates the angle γ of the OC straight line relative to the baseline through the D1E1/E1F1 ratio, D2E2/E2F2 ratio, α3, and β3.

That is: γ=F2 (D1E1/E1F1, α3, D2E2/E2F2, β3), F2 is the calculation function.

Line segment objects can be understood as line segments in an object that are not on the same straight line.

Further, determine the size of the line segment and calculate the distance between the two mobile phones through trigonometric functions.

Figure 4 is a schematic diagram of the present invention using a three-dimensional object to determine the angle of the positioning point. 101 is a mobile phone, 101 is at point O, 102 is another mobile phone, and 102 is at point C. In the positioning space, G is a three-dimensional object, and the two sides of the three-dimensional object It is S1 and S2, specifically it can be an object that can be recognized and determined by computer software, that is, it is an object that the server can clearly determine through image recognition, or an artificially set marker (the marker is filed in the server and used for identification ), 105 is the server. Assume that mobile phone 101 is looking for the position of mobile phone 102 (that is, the angle at which the OC is found). First, mobile phone 102 can rotate and take pictures in the positioning space to obtain the image of the three-dimensional object G and the direction angle (viewing angle) when shooting the three-dimensional object G. At this time, the three-dimensional object G The direction is perpendicular to the screen of the mobile phone 102 (for the sake of simplicity, only the horizontal plane will be considered for the time being, and the calculation of the vertical plane can be verified in the same way), that is, the image of the three-dimensional object G falls in the center of the screen of the mobile phone 102. According to the direction sensor (magnetic element) in the mobile phone ) Obtain the angle between the direction of the three-dimensional object G and the datum line (the datum line is determined by the determined north-south angle line. For simplicity, the east-west direction is used as the datum line in the figure, that is, the WE direction, or the OX direction) (i.e., when capturing the image) direction angle), the shooting is completed and the captured image and its direction angle are uploaded to the server; the mobile phone 101 rotates the positioning space to take pictures and looks for the mobile phone 102, and also obtains the image of the three-dimensional object G and its direction angle and uploads it to the server, and the server determines the identity of the two locations according to the The area ratio and direction angle of the two faces of the image of the three-dimensional object G are calculated through geometric mathematics to calculate the unique direction angle γ of OC (the angle between the reference line OX). The steps are:

(1) Determine the unified baseline through the client's direction device;

(2) The areas of the two faces of the three-dimensional object G are S1 and S2. At position C, obtain the vertical projection image G1 of the three-dimensional object G and the angle α4 between the vertical projection direction and the reference line, and upload the images G1 and α4 to the server;

(3) Position O obtains the vertical projection image G2 of the three-dimensional object G and the angle β4 between the vertical projection direction and the reference line, and uploads the images G2 and β4 to the server;

(4) The server determines through image recognition that both image G1 and image G2 are images of the three-dimensional object G. In the G1 image, SC1 is the image area of S1, SC2 is the image area of S2, in the G2 image SO1 is the image area of S1, and SO2 is For the image area of S2, the angle γ of the OC straight line relative to the baseline is calculated through the SC1/SC2 ratio, SO1/SO2 ratio, α4, and β4.

That is: γ=F3 (SC1/SC2, α4, SO1/SO2, β4), F3 is the calculation function.

Further, determine the size of the three-dimensional object and calculate the distance between the two mobile phones through trigonometric functions.

Figure 5 is a schematic diagram of the AR positioning interface of the present invention. A real-life image of the positioning space is captured on the screen of the mobile phone 101, including feature objects A and B. Of course, it can include line segment objects or three-dimensional objects. A dynamic indicator icon 501 is set to guide the mobile phone 101 to find the location of the mobile phone 102. At position C, 502 is the position cursor of mobile phone 101.

The above method only considers horizontal plane calculations. In practical applications, it can be combined with vertical planes (the same principle can be demonstrated) to form 3D stereopositioning.

Figure 6 is a hardware configuration diagram of the present invention, including server and client:

The server includes: an image recognition unit, used for image recognition and determination of feature objects, line objects, three-dimensional objects, people, etc. The specific image recognition strategy includes existing mature feature recognition, object outline recognition, color recognition, action recognition, and face recognition. , text recognition, etc., corresponding recognition strategies are adopted in different environments.

Calculation unit, used for angle calculations, specifically trigonometric function calculations.

The calculation result pushing unit pushes the calculation result information or the calculation result information image to the client.

The client is the existing mobile phone, including: display screen and camera;

Image overlay unit for AR display;

The sensor unit includes a direction device for north-south direction determination, a horizontal sensor (two-dimensional) for measuring inclination (for horizontal positioning, or vertical angle measurement), and a gyroscope for motion measurement;

Network communication unit, including WIFI network or wireless communication network (2G, 3G, 4G, etc.);

GPS and LBS positioning units are used for geographical location positioning and implement comprehensive positioning strategies in combination with the present invention.

Figure 7 is a flow chart of the present invention, including steps:

Client 1:

Determine the baseline, that is, determine the unified baseline of the positioning system, which is determined by the client's direction device. The specific direction is not limited, but once determined, all clients and servers will use the determined baseline as the angle measurement baseline;

Shoot real-time scenes and simultaneously record the direction angle and upload it to the server. The client uses the camera and synchronously calls the direction device angle to shoot the real-time scene. It can further synchronously call the horizontal inclination angle (one-dimensional or two-dimensional) of the horizontal device;

Client 2:

Determine the baseline, that is, determine the unified baseline of the positioning system, which is determined by the client's direction device. The specific direction is not limited, but once determined, all clients and servers will use the determined baseline as the angle measurement baseline;

Shoot real-time scenes and simultaneously record the direction angle and upload it to the server. The client uses the camera and synchronously calls the direction device angle to shoot the real-time scene. It can further synchronously call the horizontal inclination angle (one-dimensional or two-dimensional) of the horizontal device;

server:

At least two features are determined. The server obtains the image uploaded by the client and performs image recognition, and determines the features according to the recognition strategy;

Based on the angles of client 1 and the two features relative to the baseline, and the angles of client 2 and the two features relative to the baseline, calculate the angle of the line connecting client 1 and client 2 relative to the baseline. .

or,

Determine at least one line segment object DEF, and the line segment DE and line segment EF of the line segment object DEF are not on the same straight line;

According to the angle of client 1 and line segment object DEF relative to the baseline and the DE/EF ratio of the image acquired by client 1, and the angle of client 2 and line segment object DEF relative to the baseline and the DE/EF ratio of the image acquired by client 1 DE/EF ratio, calculate the angle of the line connecting client 1 and client 2 relative to the baseline.

or,

Identify at least one three-dimensional object;

According to the angle between client 1 and the three-dimensional object relative to the reference line and the ratio of the areas of the two faces of the three-dimensional object image obtained by client 1, and, the angle between client 2 and the three-dimensional object relative to the reference line and the angle obtained by client 2 The ratio of the areas of the two faces of the three-dimensional object image is used to calculate the angle of the line connecting client 1 and client 2 relative to the reference line.

The server sends the position and angle information of client 2 to client 1, and client 1 superimposes the position icon and the real scene;

The server sends the location and angle information of client 1 to client 2, and client 2 superimposes the location icon and the real scene;

Figure 8 is an AR application flow chart of the present invention, including steps:

Client 1:

Determine the baseline, that is, determine the unified baseline of the positioning system, which is determined by the client's direction device. The specific direction is not limited, but once determined, all clients and servers will use the determined baseline as the angle measurement baseline;

Shoot real-time scenes and simultaneously record the direction angle and upload it to the server. The client uses the camera and synchronously calls the direction device angle to shoot the real-time scene. It can further synchronously call the horizontal inclination angle (one-dimensional or two-dimensional) of the horizontal device;

Client 2:

Determine the baseline, that is, determine the unified baseline of the positioning system, which is determined by the client's direction device. The specific direction is not limited, but once determined, all clients and servers will use the determined baseline as the angle measurement baseline;

Shoot real-time scenes and simultaneously record the direction angle and upload it to the server. The client uses the camera and synchronously calls the direction device angle to shoot the real-time scene. It can further synchronously call the horizontal inclination angle (one-dimensional or two-dimensional) of the horizontal device;

server:

At least two features are determined. The server obtains the image uploaded by the client and performs image recognition, and determines the features according to the recognition strategy;

Based on the angles of client 1 and the two features relative to the baseline, and the angles of client 2 and the two features relative to the baseline, calculate the angle of the line connecting client 1 and client 2 relative to the baseline. .

or,

Determine at least one line segment object DEF, and the line segment DE and line segment EF of the line segment object DEF are not on the same straight line;

According to the angle of client 1 and line segment object DEF relative to the baseline and the DE/EF ratio of the image acquired by client 1, and the angle of client 2 and line segment object DEF relative to the baseline and the DE/EF ratio of the image acquired by client 1 DE/EF ratio, calculate the angle of the line connecting client 1 and client 2 relative to the baseline.

or,

Identify at least one three-dimensional object;

According to the angle between client 1 and the three-dimensional object relative to the reference line and the ratio of the areas of the two faces of the three-dimensional object image obtained by client 1, and, the angle between client 2 and the three-dimensional object relative to the reference line and the angle obtained by client 2 The ratio of the areas of the two faces of the three-dimensional object image is used to calculate the angle of the line connecting client 1 and client 2 relative to the reference line.

The server locates the AR red envelope and pushes the AR red envelope to the client;

or,

The server locates the virtual advertisement and pushes the virtual advertisement to the client;

or,

The server locates the AR game object and pushes the AR game object to the client;

The client superimposes the AR image and displays it on the real scene of the client screen.

The specific application modes of positioning and AR methods based on image recognition are:

(1) At present, instant messaging clients have been widely used. Friends in the virtual space may not have met each other. The present invention is embedded in existing instant messaging software such as QQ, WeChat, etc. Friends on both sides can turn on the camera to find each other at any location. And display mutual indication icons on the mobile phone screen. When finding the other party, virtual images such as avatars can be superimposed on the real scene on the screen, and audio or vibration prompts can be provided.

(2) One party first shoots the positioning space image and uploads it to the server, and the other party shoots and searches for the positioning space within a certain effective time range.

(3) One party is the merchant, and the merchant takes the positioning space image and uploads it to the server, and the customer takes the positioning space image to find the merchant.

(4) Combined with China Patent Announcement 2016105984877, an AR method and system to provide red envelopes or virtual advertising positioning. The virtual image includes position and direction parameters, and the client hardware includes position and direction sensors. The virtual image displayed on the client changes with the orientation of the client hardware, which is VR technology.

The specific steps are:

①Client 1 starts the red envelope publishing program, captures surrounding images and simultaneously obtains the direction angle and uploads it to the server;

②The server obtains the surrounding image and its direction angle taken by client 1;

③The server generates a virtual red envelope image based on the red envelope settings of client 1 and associates the image and its direction angle obtained by the server in step ②. The red envelope action assignment includes the direction angle, horizontal inclination angle, vertical height, and changes in its action assignment;

④Client 2 captures surrounding images and simultaneously obtains direction angles and uploads them to the server;

⑤ Based on the surrounding image and its direction angle taken by client 1 obtained in step ②, and the surrounding image and its direction angle taken by client 2 in step ④, the server calculates and obtains the location where client 1 releases the red envelope, that is, the location of the red envelope. The server Push the virtual red envelope image to client 2;

⑥Client 2 superimposes the virtual red envelope image and displays it on the real screen according to the position of the red envelope. At the same time, the virtual red envelope image moves according to the action assignment;

⑦Client 2 obtains red envelopes through the touch screen.

Of course, the above virtual red envelope images can be replaced by merchant advertisements and virtual game object images.

The steps for the server to publish advertisements are:

①The server obtains the surrounding image and its direction angle taken by any client;

②The server generates a virtual advertising image based on the advertiser's settings (placement, text, images, image actions, etc.) and associates the image and its direction angle obtained by the server in step ①. The advertiser's settings include placement, text, images, images, etc. action;

③The advertising receptor client captures surrounding images and simultaneously obtains the direction angle and uploads it to the server;

④The server calculates and obtains the advertising placement position based on the surrounding image and its directional angle captured by any client obtained in step ①, and the surrounding image and directional angle captured by the advertising receptor client in step ③;

⑤The advertising receptor client overlays and displays the virtual advertising image on the real screen according to the placement position of the advertisement. At the same time, the virtual advertising image moves according to the action assignment;

⑥The advertising receptor client obtains advertising content through the touch screen, such as links, jumps, collections, etc.

(5) AR space positioning of game objects in the game system, such as virtual animals, virtual items, virtual treasures, etc. in the game. Specifically, in the embodiment, the mobile phone at O rotates to capture the positioning space and then issues AR red envelopes, virtual items, or virtual game objects at O, which are provided to other mobile phone clients to find within a certain period of time.

(6) At present, face recognition is very mature. In addition, one-pass scene recognition can be combined with clothing color, etc., so that both parties can obtain the position and angle of the other party based on the one-pass dynamic image recognition results.

The above application models and rules do not limit the basic features of the method and system of the present invention, nor do they limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (13)

1. A method of positioning based on image recognition, characterized by: the method comprises the steps that a server and a client are arranged, a first mobile phone is arranged at a point O, a second mobile phone is arranged at a point C, at least two characteristic object A and object B are determined in a positioning space, the first mobile phone searches for a second mobile phone position direction angle gamma, the second mobile phone photographs the positioning space, synchronously records the direction angle and uploads the same to the server, acquires an image of the characteristic object A and a direction angle when the characteristic object A is photographed, photographs the characteristic object B in the same way, and uploads a photographed image and the direction angle thereof to the server; the first mobile phone shoots the positioning space, also acquires images of two feature object and direction angles thereof, and uploads the images to the server, and the server calculates a unique OC direction angle gamma according to the direction angles of the two feature object at two places by geometric mathematics, wherein the steps are as follows:

(1) Determining a unified datum line through a direction device of the client;

(2) Acquiring an image A1 of an object A and an angle alpha 1 relative to a reference line at a position C, acquiring an image B1 of an object B and an angle alpha 2 relative to the reference line at the position C, and uploading the images A1 and alpha 1 and the images B1 and alpha 2 to a server;

(3) The position O acquires an image A2 of the object A and an angle beta 1 relative to the datum line, the position O acquires an image B2 of the object B and an angle beta 2 relative to the datum line, and the images A2 and beta 1 and the images B2 and beta 2 are uploaded to a server;

(4) The server determines that the image A1 and the image A2 are both images of the object A through image recognition, the server determines that the image B1 and the image B2 are both images of the object B through image recognition, and the direction angle gamma of the OC straight line relative to the datum line is obtained through calculation of alpha 1, alpha 2, beta 1 and beta 2.

2. A method of image recognition based positioning according to claim 1, further characterized by calculating a direction angle γ:

in triangle ABC:

AB/sin(α1+α2)=AC/sin(δ-α2)=BC/sin(- α1- δ) =bc/sin (α1+δ), deriving ac=absin (δ - α2)/sin (α1+α2);

in triangular AOB:

AB/sin(β1-β2)=OB/sin(- δ - β1) =ob/sin (δ+β1) =oa/sin (δ+β2), deriving oa=absin (δ+β2)/sin (β1- β2);

in triangular AOC:

OA/sin(-γ+α1)=OA/ sin(γ-α1)=AC/sin(γ-β1)，

substitution to obtain: ABsin (δ+β2)/sin (β1- β2) sin (γ - α1) =absin (δ - α2)/sin (α1+α2) sin (γ - β1), sin (γ - α1)/sin (γ - β1) =sin (α1+α2) sin (δ+β2)/sin (δ - α2) sin (β1- β2);

let k=sin (α1+α2) sin (δ+β2)/sin (δ - α2) sin (β1- β2), then sin (γ - α1)/sin (γ - β1) =k;

sinγcosα1-cosγsinα1=Ksinγcosβ1-Kcosγsinβ1；

(Ksinβ1-sinα1)cosγ=(Kcosβ1-cosα1)sinγ；

cosγ= sinγ(Kcosβ1-cosα1)/(Ksinβ1- sinα1)；

let (Kcos β1-cos α1)/(Ksin β1-sin α1) =t, then cos γ=tsin γ;

because of cos ² γ+sin ² Gamma=1, so t ² sin ² γ+sin ² γ=1；

sin ² γ=1/(1+t ² )，γ∈[0, ]The time sin gamma is not less than 0, sin gamma=root number (1/(1+t) ² )）；

Finally, the direction angle gamma is obtained.

3. A method of positioning based on image recognition, characterized by: the method comprises the steps that a first mobile phone is arranged at a point O, a second mobile phone is arranged at a point C, DEF is a line segment object in a positioning space, a line segment DE and a line segment EF are not on the same straight line, the first mobile phone searches for a second mobile phone position direction angle gamma, the second mobile phone photographs the positioning space, an image of the line segment object DEF and a direction angle when the line segment object DEF is photographed are obtained, at the moment, the direction of the line segment object DEF is perpendicular to a screen of the second mobile phone, namely, the image of the line segment object DEF falls in the center of the screen of the second mobile phone, an included angle between the line segment object DEF and a reference line, namely, a direction angle when the image is photographed, is obtained according to a direction sensor in the mobile phone, and the photographed image and the direction angle are uploaded to the server; the first mobile phone photographs the positioning space to find a second mobile phone, likewise obtains an image of the line segment object DEF and a direction angle thereof, and uploads the image to the server, and the server calculates a unique OC direction angle gamma according to the line segment ratio and the direction angle of the image of the same line segment object DEF at two places by geometric mathematics, wherein the method comprises the following steps:

(1) Determining a unified datum line through a direction device of the client;

(2) The method comprises the steps that a line segment DE and a line segment EF of a line segment object DEF are not on the same straight line, a vertical projection image D1E1F1 of the line segment object DEF and an angle alpha 3 between the vertical projection direction and a datum line are obtained at a position C, and the images D1E1F1 and alpha 3 are uploaded to a server;

(3) The position O acquires a vertical projection image D2E2F2 of the line segment object DEF and an angle beta 3 between the vertical projection direction and a reference line, and uploads the images D2E2F2 and beta 3 to a server;

(4) The server determines that the images D1E1F1 and the images D2E2F2 are images of line segments DEF through image recognition, and calculates and obtains the direction angle gamma of the OC straight line relative to the datum line through the ratio of D1E1/E1F1, the ratio of D2E2/E2F2, alpha 3 and beta 3.

4. A method of image recognition based positioning, further characterized by: the method comprises the steps that a first mobile phone is arranged at a point O, a second mobile phone is arranged at a point C, G is a three-dimensional object in a positioning space, two faces of the three-dimensional object are S1 and S2, the first mobile phone searches for a second mobile phone position direction angle gamma, the second mobile phone photographs the positioning space, an image of the three-dimensional object G and a direction angle when photographing the three-dimensional object G are obtained, at the moment, the direction of the three-dimensional object G is perpendicular to a screen of the second mobile phone, namely, the image of the three-dimensional object G falls in the center of the screen of the second mobile phone, and the direction angle when photographing the image is obtained according to a direction sensor in the mobile phone, namely, the direction angle when photographing the image is completed, and the photographed image and the direction angle are uploaded to the server; the first mobile phone photographs the positioning space to find a second mobile phone, also obtains the image of the solid object G and the direction angle thereof, and uploads the image to the server, and the server calculates the unique direction angle gamma of the OC according to the area ratio and the direction angle of two faces of the image of the same solid object G at two places by geometric mathematics, wherein the method comprises the following steps:

(1) Determining a unified datum line through a direction device of the client;

(2) The areas of two surfaces of the three-dimensional object G are S1 and S2, a vertical projection image G1 of the three-dimensional object G and an angle alpha 4 between the vertical projection direction and a datum line are acquired at a position C, and the images G1 and alpha 4 are uploaded to a server;

(3) The position O acquires a vertical projection image G2 of the three-dimensional object G and an angle beta 4 between the vertical projection direction and a datum line, and uploads the images G2 and beta 4 to a server;

(4) The server determines that the image G1 and the image G2 are images of the three-dimensional object G through image recognition, the image area of SC1 in the image G1 is the image area of S1, the image area of SC2 in the image G2 is the image area of S1, the image area of SO2 is the image area of S2, and the direction angle gamma of the OC straight line relative to the datum line is obtained through calculation through the ratio of SC1/SC2, the ratio of SO1/SO2, alpha 4 and beta 4.

5. A method of image recognition based positioning according to claim 1 or 2 or 3 or 4, further comprising the steps of: the client acquires the direction angle gamma calculated by the server, and positions or navigates on the client guiding map according to the direction angle gamma.

6. A method of image recognition based positioning according to claim 1 or 2 or 3 or 4, further comprising the steps of: the client acquires the direction angle gamma calculated by the server, the client opens the camera to acquire a live-action image, and the guide icon of the direction angle gamma is displayed in the live-action image in a superimposed mode.

7. A method of image recognition based positioning according to claim 1 or 2 or 3 or 4, characterized in that: the direction angle gamma is guided by friends, businesses, game targets, red packages and advertisements.

8. The method for positioning based on image recognition according to claim 7, wherein:

(1) the client 1 starts a red packet issuing program, shoots surrounding images and synchronously acquires direction angles to upload to a server;

(2) the server acquires a peripheral image shot by the client 1 and a direction angle thereof;

(3) the server generates a virtual red packet image according to red packet setting of the client 1 and associates the image acquired by the server in the step (2) and the direction angle thereof, wherein red packet action assignment comprises the direction angle, the horizontal inclination angle, the vertical height and the change of action assignment thereof;

(4) the client 2 shoots the peripheral image and synchronously acquires the direction angle and uploads the direction angle to the server;

(5) the server calculates and obtains the position where the client 1 releases the red packet, namely the position where the red packet is located, according to the peripheral image and the direction angle which are obtained by the client 1 and obtained by the step (2) and the peripheral image and the direction angle which are obtained by the step (4) and the client 2, and the server pushes the virtual red packet image to the client 2;

(6) the client 2 displays the virtual red package image in a screen real scene in a superposition manner according to the red package position, and meanwhile, the virtual red package image moves according to the action assignment;

(7) the client 2 acquires the red packet through the touch screen.

9. The method for positioning based on image recognition according to claim 7, wherein:

the server issues the advertisement steps as follows:

(1) the server acquires a peripheral image shot by any client and a direction angle thereof;

(2) the server generates a virtual advertisement image according to the setting of an advertiser and associates the image and the direction angle thereof acquired by the server in the step (1), wherein the setting of the advertiser comprises a putting position, characters, images and image actions;

(3) the advertisement receptor client shoots the peripheral image and synchronously acquires the direction angle and uploads the direction angle to the server;

(4) the server calculates and obtains the advertisement putting position according to the peripheral image and the direction angle of the peripheral image shot by any client side obtained in the step (1) and the peripheral image and the direction angle of the peripheral image shot by the advertisement receiver client side in the step (3);

(5) the advertisement receptor client displays the virtual advertisement image in the screen live-action according to the advertisement putting position, and meanwhile, the virtual advertisement image moves according to the action assignment;

(6) the advertisement acceptor client obtains advertisement content through a touch screen, wherein the advertisement content comprises links, jumps or collections.

10. A method of image recognition based positioning according to claim 1 or 2 or 3 or 4, further comprising the steps of: and determining the distance between the two connecting lines of the characteristic objects, or the size of the line segment object, or the size of the three-dimensional object, and calculating the distance between the two connecting lines of the positions through a trigonometric function.

11. A system for a method of image recognition based positioning according to claim 1 or 2 or 3 or 4, characterized in that: comprising a server and a client side,

the server comprises:

the image recognition unit is used for determining a characteristic object, a line segment object or a three-dimensional object;

a calculation unit for calculating the direction angle of the connecting line of the two position points according to the direction angles of the two feature objects; or calculating the direction angle of the connecting line of the two position points according to the ratio of the two line segments which are not on the same straight line and the direction angle thereof; or calculating the direction angle of the connecting line of the two position points according to the area ratio of the two surfaces of the three-dimensional object and the direction angle thereof;

the calculation result pushing unit is used for pushing the calculated direction angle data of the connecting line of the two position points to the client;

the client comprises at least a direction acquisition unit, an image superposition unit and a camera.

12. The system of a method of image recognition based positioning of claim 11, wherein: the system is embedded in an existing IM system or payment system or gaming system.

13. The system of a method of image recognition based positioning of claim 11, wherein: the client is a mobile phone or a tablet computer.