CN111462247A - Cursor position calibration method and device for screen interaction - Google Patents
Cursor position calibration method and device for screen interaction Download PDFInfo
- Publication number
- CN111462247A CN111462247A CN202010176903.0A CN202010176903A CN111462247A CN 111462247 A CN111462247 A CN 111462247A CN 202010176903 A CN202010176903 A CN 202010176903A CN 111462247 A CN111462247 A CN 111462247A
- Authority
- CN
- China
- Prior art keywords
- screen
- calibration
- coordinates
- image
- signal source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000003993 interaction Effects 0.000 title claims abstract description 37
- 238000013507 mapping Methods 0.000 claims abstract description 89
- 238000004364 calculation method Methods 0.000 claims abstract description 20
- 238000004458 analytical method Methods 0.000 claims abstract description 13
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B21/00—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant
Abstract
The embodiment of the invention discloses a cursor position calibration method for screen interaction, which comprises the following steps: displaying a calibration graph on a screen through a signal source, wherein a plurality of calibration points are arranged on the calibration graph, and the coordinates of the calibration points are screen coordinates of signals displayed on the screen by the signal source; acquiring an image of a screen where a calibration graph is located through a camera, and sending the image to a server for analysis and calculation to obtain image coordinates of the calibration point; establishing a first mapping relation between the image coordinates of each pixel point of the image acquired by the camera and the screen coordinates of each pixel point of the signal displayed on the screen by the signal source according to the screen coordinates and the image coordinates of the calibration points; and calibrating the cursor of the signal source on the screen according to the first mapping relation. The embodiment of the invention also discloses a cursor position calibration device for screen interaction, and the technical scheme disclosed by the embodiment of the invention greatly improves the accuracy of cursor position calculation.
Description
Technical Field
The embodiment of the invention relates to the field of screen interaction, in particular to a cursor position calibration method and device for screen interaction.
Background
In the screen interaction field, one of the methods is to directly perform interaction operation on a screen through a remote controller, hold a transmitter capable of transmitting laser/infrared with a certain wavelength by hand, generate corresponding light spots on the screen by light emitted by the transmitter, and perform image shooting on the screen by a camera according to a certain frequency; the method comprises the steps of processing a shot picture according to a certain algorithm, calculating the coordinate position of a light spot on the picture, and generating certain distortion and deformation on the shot picture because a certain distance and an angle exist between an image imaging surface and an actual screen surface. Therefore, it is important to calculate the accurate position of the light spot on the screen, so that the screen interaction can be accurately performed.
In the existing technology, only four vertexes of a screen are calibrated, and a screen area or a part of the screen area is interpreted as a plane defined by four points, namely the screen is assumed to be an ideal plane; the method can not well simulate the concave-convex part in the screen, and can not well solve the deviation of the cursor coordinate position caused by the curved surface of the screen if the curved surface screen or the special-shaped screen is adopted, so that the calculated cursor coordinate position deviation is large.
Disclosure of Invention
The technical solution disclosed in the embodiments of the present invention greatly improves the accuracy of the cursor position calculation.
In a first aspect, an embodiment of the present invention provides a cursor position calibration method for screen interaction, including:
displaying a calibration graph on a screen through a signal source, wherein a plurality of calibration points are arranged on the calibration graph, and the coordinates of the calibration points are screen coordinates of signals displayed on the screen by the signal source;
acquiring an image of a screen where a calibration graph is located through a camera, and sending the image to a server for analysis and calculation to obtain image coordinates of the calibration point;
establishing a first mapping relation between the image coordinates of each pixel point of the image acquired by the camera and the screen coordinates of each pixel point of the signal displayed on the screen by the signal source according to the screen coordinates and the image coordinates of the calibration points;
and calibrating the cursor of the signal source on the screen according to the first mapping relation.
Preferably, the establishing a first mapping relationship between the image coordinates of each pixel point of the image obtained by the camera and the screen coordinates of each pixel point of the signal displayed on the screen by the signal source according to the screen coordinates and the image coordinates of the plurality of calibration points includes:
acquiring a calibration equation, wherein the calibration equation is used for expressing the mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points, and the number of the calibration points is more than the number of calibration parameters of the calibration equation, so that an overdetermined equation can be obtained;
bringing the screen coordinates and the image coordinates of the calibration points into the overdetermined equations one by one to obtain a plurality of groups of coefficients of the overdetermined equations, and obtaining a plurality of calibration equations according to the plurality of groups of coefficients of the overdetermined equations;
the method comprises the steps of selecting an equation with the minimum calibration error from a plurality of calibration equations, and establishing a first mapping relation between image coordinates of each pixel point of an image acquired by a camera and screen coordinates of each pixel point of a signal displayed on a screen by a signal source.
Preferably, the calibration equation is:
XS(X,Y)＝K0*Xn+K1*Yn+K2*Xn-1*Y+K3*X*Yn-1+K4*Xn-1+K5*Yn-1+K6*Xn-2*Yn-2…+Kc-3X*Y+Kc-2X+Kc-1Y+Kc；
YS(X,Y)＝K00*Xn+K1*Yn+K02*Xn-1*Y+K03*X*Yn-1+K04*Xn-1+K05*Yn-1+K06*Xn-2*Yn-2…Kb- 3X*Y+Kb-2X+Kb-1Y+Kb；
wherein n is not less than 3, n is an integer, K0… Kc and K00…KbFor calibration parameters, (X, Y) are image coordinates and (Xs, Ys) are screen coordinates.
Preferably, the calibrating the cursor of the signal source on the screen according to the first mapping relationship includes:
emitting light with a certain wavelength to a screen and forming a light spot on the screen;
acquiring an image of a screen where a light spot is located, and sending the image to a server for analysis and calculation to obtain an image coordinate for displaying the light spot on the image;
bringing the image coordinates into a first mapping relation, and calculating screen coordinates of a cursor of the signal source on a screen;
and sending the calculated screen coordinates to a signal source, and moving a cursor to the calculated screen coordinates by the signal source.
Preferably, the calibration graph is a curved surface provided with 16 calibration points.
Preferably, the screen coordinates and the image coordinates are represented by pixel values.
Compared with the prior art, the embodiment of the invention provides a method for calibrating the position of a cursor on a screen, which comprises the steps of displaying a calibration graph with a plurality of calibration points on the screen through a signal source, acquiring screen coordinates of the calibration points, and acquiring an image of the screen where the calibration graph is located through a camera to acquire image coordinates of the calibration points displayed on the image; the method comprises the steps of establishing a first mapping relation between image coordinates of each pixel point of an image acquired by a camera and screen coordinates of each pixel point of a signal displayed on a screen by a signal source, and calibrating a cursor of the signal source on the screen according to the first mapping relation.
In a second aspect, an embodiment of the present invention further provides a method for calibrating a cursor position on a screen, including:
dividing the screen into a plurality of regions;
establishing a second mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the calibration graph in each area;
determining the position of a light spot formed on a screen by emitting certain wavelength light through an emitter, and determining the area of the light spot according to the position of the light spot;
and calibrating the cursor of the signal source on the screen according to the second mapping relation.
Preferably, the determining the position of the spot formed on the screen by the emitter emitting light of a certain wavelength comprises:
establishing a third mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the calibration graph in the whole screen;
acquiring an image of a screen where a light spot is located through a camera, and sending the image to a server for analysis and calculation to obtain an image coordinate for displaying the light spot on the image;
bringing the image coordinates into a third mapping relation, and calculating screen coordinates of a cursor of the signal source on a screen;
and determining the position of the light spot through the screen coordinates.
Preferably, the second mapping relationship and the third mapping relationship are determined by:
acquiring a calibration equation, wherein the calibration equation is used for expressing the mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the whole screen or each area in the whole screen, and the number of the calibration points is more than that of the calibration coefficients of the calibration equation, so that an overdetermined equation can be obtained;
screen coordinates and corresponding image coordinates of the whole screen or each area in the whole screen of the calibration points are brought into the over-determined equations one by one to obtain a plurality of groups of coefficients of the over-determined equations of the whole screen or each area in the whole screen, and a plurality of calibration equations of the whole screen or each area in the whole screen are obtained according to the coefficients of the plurality of groups of the over-determined equations;
an equation with the smallest calibration error is selected from the plurality of calibration equations as the calibration equation expressing the second mapping relationship and the third mapping relationship.
Preferably, the calibration equation of the whole screen is:
XS(X,Y)＝K0*Xn+K1*Yn+K2*Xn-1*Y+K3*X*Yn-1+K4*Xn-1+K5*Yn-1+K6*Xn-2*Yn-2…+Kc-3X*Y+Kc-2X+Kc-1Y+Kc
YS(X,Y)＝K00*Xn+K1*Yn+K02*Xn-1*Y+K03*X*Yn-1+K04*Xn-1+K05*Yn-1+K06*Xn-2*Yn-2…Kb- 3X*Y+Kb-2X+Kb-1Y+Kb
the calibration equation of each area in the whole screen is as follows:
XS1(X,Y)＝K0*Xn1+K1*Yn1+K2*Xn1-1*Y+K3*X*Yn1-1+K4*Xn1-1+K5*Yn1-1+K6*Xn1-2*Yn1-2…+Kc-3X*Y+Kc-2X+Kc-1Y+Kc
YS1(X,Y)＝K00*Xn1+K1*Yn1+K02*Xn1-1*Y+K03*X*Yn1-1+K04*Xn1-1+K05*Yn1-1+K06*Xn-2*Yn -2…Kb-3X*Y+Kb-2X+Kb-1Y+Kb
wherein n1 is not less than 3, n1 is not more than n, n1 is an integer, K0… Kc and K00… Kb are calibration parameters, (X, Y) are image coordinates, (Xs, Ys), (Xs)1，Ys1) Are screen coordinates.
Compared with the prior art, the embodiment of the invention provides a cursor position calibration method for screen interaction, a screen is divided into a plurality of areas, namely the screen is discretized, multi-point calibration is respectively carried out on each discrete area to form a plurality of groups of mapping relations, and the cursor position is calculated after the corresponding mapping relation is determined according to the area where the light point position is located.
In a third aspect, an embodiment of the present invention further provides a cursor position calibration apparatus for screen interaction, including:
the display module is used for displaying a calibration graph on a screen through a signal source, a plurality of calibration points are arranged on the calibration graph, and the coordinates of the calibration points are screen coordinates of signals displayed on the screen by the signal source;
the calculation module is used for acquiring an image of a screen where the calibration graph is located through the camera, and sending the image to the server for analysis and calculation to obtain the image coordinates of the calibration point;
the establishing module is used for establishing a first mapping relation between the image coordinates of each pixel point of the image acquired by the camera and the screen coordinates of each pixel point of the signal displayed on the screen by the signal source according to the screen coordinates and the image coordinates of the plurality of calibration points;
and the calibration module is used for calibrating the cursor of the signal source on the screen according to the first mapping relation.
Compared with the prior art, the embodiment of the invention provides a cursor position calibration device for screen interaction, which is characterized in that a calibration graph with a plurality of calibration points is displayed on a screen through a signal source, screen coordinates of the calibration points are obtained, an image of the screen where the calibration graph is located is obtained through a camera, and image coordinates of the calibration points displayed on the image are obtained; establishing a first mapping relation between the screen coordinates and the image coordinates of the calibration points; and calibrating the cursor of the signal source on the screen according to the first mapping relation, and greatly improving the accuracy of cursor position calculation by the technical scheme of the embodiment of the invention.
In a fourth aspect, an embodiment of the present invention further provides a cursor position calibration apparatus for screen interaction, including:
a dividing module for dividing the screen into a plurality of regions;
the establishing module is used for establishing a second mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the calibration graph in each area;
the device comprises a determining module, a light source module and a control module, wherein the determining module is used for determining the position of a light spot formed on a screen by transmitting light with certain wavelength through a transmitter and determining the area of the light spot according to the position of the light spot;
and the calibration module is used for calibrating the cursor of the signal source on the screen according to the second mapping relation.
Compared with the prior art, the embodiment of the invention provides a cursor position calibration device for screen interaction, a screen is divided into a plurality of areas, namely the screen is discretized, multi-point calibration is respectively carried out on each discrete area to form a plurality of groups of mapping relations, and the cursor position is calculated after the corresponding mapping relation is determined according to the area where the light point position is located.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. Some specific embodiments of the present application will be described in detail hereinafter by way of illustration and not limitation with reference to the accompanying drawings. The same reference numbers will be used throughout the drawings to refer to the same or like parts or portions, and it will be understood by those skilled in the art that the drawings are not necessarily drawn to scale, in which:
FIG. 1 is a schematic structural diagram of a system for calibrating a cursor position for screen interaction according to an embodiment of the present invention;
FIG. 2 is a schematic flowchart of a method for calibrating a cursor position for screen interaction according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a calibration graph according to an embodiment of the disclosure;
FIG. 4 is a flowchart illustrating another method for calibrating a cursor position for screen interaction according to an embodiment of the present disclosure;
FIG. 5 is a schematic structural diagram of a cursor position calibration apparatus for screen interaction according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of another cursor position calibration device for screen interaction according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of the embodiments of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or otherwise described herein.
The applicant finds that the concave-convex part in the screen can not be well simulated by the commonly used method at present, and if the curved screen or the special-shaped screen is subjected to the situation, the position deviation of the coordinates caused by the curved surface of the screen can not be well solved, so that the calculated coordinate position deviation is large.
In order to solve the technical problem, the embodiment of the invention forms a most appropriate calibration equation which can cover all curved surfaces of the calibration points by solving an overdetermined equation set by adopting multi-point calibration, establishes a mapping relation from the coordinates of the light spot image to the position of the cursor, and can greatly improve the accuracy of the calculated position of the cursor by the technical scheme of the embodiment of the invention.
As shown in fig. 1, the embodiment of the present invention discloses a method for calibrating a cursor position for screen interaction, which is required to be applied to a screen interaction system when being implemented specifically; the basic equipment of the system comprises: the system comprises a screen (a plane or curved surface or special-shaped screen), a signal source (a computer for providing signals for the screen), a camera (for collecting screen images in real time), a server (for receiving image information, calculating cursor positions and the like), and an emitter (for emitting light with certain wavelength to form light spots on the screen).
As shown in fig. 2, an embodiment of the present invention discloses a method for calibrating a cursor position for screen interaction, which specifically includes the following steps:
step S01, displaying a calibration graph on a screen through a signal source, wherein the calibration graph is provided with a plurality of calibration points, and the coordinates of the calibration points are the screen coordinates of the signal source for displaying the signal on the screen;
it should be noted that, as shown in fig. 3, in the embodiment of the present invention, by simulating the screen as a curved surface defined by 16 points and calibrating the position of the cursor on the screen according to the curved surface, first, a calibration graph (but not limited to this graph) needs to be displayed on the screen; the calibration graph is displayed on a screen through a signal source; the coordinates of each calibration point in the calibration graph are known, namely the coordinates of the pixels on the corresponding signal source display signals, namely the screen coordinates.
In the embodiment of the invention, 16 calibration points are adopted, or more or less than 16 calibration points can be adopted, and the more the calibration points are selected, the better the effect is.
Step S02, acquiring an image of a screen where the calibration graph is located through a camera, and sending the image to a server for analysis and calculation to obtain the image coordinates of the calibration point;
it should be noted that, a camera at a certain distance and height from the screen takes a screen image, the camera transmits the image to the server, and the server analyzes the image according to an algorithm to obtain pixel value coordinates of a plurality of calibration points displayed on the image, that is, image coordinates.
Step S03, according to the screen coordinates and the image coordinates of the calibration points, establishing a first mapping relation between the image coordinates of each pixel point of the image obtained by the camera and the screen coordinates of each pixel point of the signal displayed on the screen by the signal source;
the method specifically comprises the following steps:
acquiring a calibration equation, wherein the calibration equation is used for expressing the mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points, and the number of the calibration points is more than the number of calibration parameters of the calibration equation, so that an overdetermined equation can be obtained;
bringing the screen coordinates and the image coordinates of the calibration points into the overdetermined equations one by one to obtain a plurality of groups of coefficients of the overdetermined equations, and obtaining a plurality of calibration equations according to the plurality of groups of coefficients of the overdetermined equations;
the method comprises the steps of selecting an equation with the minimum calibration error from a plurality of calibration equations, and establishing a first mapping relation between image coordinates of each pixel point of an image acquired by a camera and screen coordinates of each pixel point of a signal displayed on a screen by a signal source.
It should be noted that, firstly, a calibration equation for expressing the mapping relationship between the screen coordinates and the image coordinates of a plurality of calibration points is obtained:
XS(X,Y)＝K0*Xn+K1*Yn+K2*Xn-1*Y+K3*X*Yn-1+K4*Xn-1+K5*Yn-1+K6*Xn-2*Yn-2…+Kc-3X*Y+Kc-2X+Kc-1Y+Kc；
YS(X,Y)＝K00*Xn+K1*Yn+K02*Xn-1*Y+K03*X*Yn-1+K04*Xn-1+K05*Yn-1+K06*Xn-2*Yn-2…Kb- 3X*Y+Kb-2X+Kb-1Y+Kb；
wherein n is not less than 3, n is an integer, K0… Kc and K00…KbFor calibration parameters, (X, Y) are image coordinates and (Xs, Ys) are screen coordinates.
The higher the order n of the calibration equation is, the more accurate the calibration is, but the larger the calculation amount is, the greater the number of calibration points is in the embodiment of the invention is, the number of calibration parameters of the calibration equation is determined, so that the over-determined equation can be obtained;
in the present embodiment, the calibration equation is determined by 16 calibration points, and the present embodiment is described in detail by the order n being 3.
If the order n is 3, the calibration equation is:
XS(X,Y)＝K0*X3+K1*Y3+K2*X2*Y+K3*X*Y2+K4*X2+K5*Y2+K6*X*Y+K7*X+K8*Y+K9
YS(X,Y)＝K00*X3+K01*Y3+K02*X2*Y+K03*X*Y2+K04*X2+K05*Y2+K06*X*Y+K07*X+K08*Y+K09
wherein, K0…KcAnd K00…KbFor calibration parameters, (X, Y) are image coordinates and (Xs, Ys) are screen coordinates.
Due to the above-mentioned calibration parameter K0…K9AndK00…K09is smaller than the number of calibration points, so that an over-determined equation can be obtained, so that the screen coordinates (X) of 16 calibration points on the screenS，YS) And the camera shoots image coordinates (X, Y) of 16 calibration points on a screen image, 16 sets of coordinate points which correspond to one another are substituted into the over-determined equation to obtain coefficients of a plurality of sets of over-determined equations, a plurality of calibration equations are obtained according to the coefficients of the plurality of sets of over-determined equations, an equation with the minimum calibration error is selected from the plurality of calibration equations, and a first mapping relation between the image coordinates of each pixel point of the image obtained by the camera and the screen coordinates of each pixel point of the display signal of the signal source on the screen is established, namely the calibration equation with the determined calibration parameters is determined.
And step S04, calibrating the cursor of the signal source on the screen according to the first mapping relation.
The method specifically comprises the following steps:
emitting certain wavelength light to a screen through an emitter, and forming a light spot on the screen;
acquiring an image of a screen where a light spot is located through a camera, and sending the image to a server for analysis and calculation to obtain an image coordinate for displaying the light spot on the image;
bringing the image coordinates into a first mapping relation, and calculating screen coordinates of a cursor of the signal source on a screen;
and sending the calculated screen coordinates to a signal source, and moving a cursor to the calculated screen coordinates by the signal source.
It should be noted that, the emitter emits light with a certain wavelength to the screen to generate a light spot, the light may be laser or infrared light, the camera shoots the screen image and transmits the image to the server, and the server calculates the image coordinate of the light spot on the screen image; then, the image coordinates are substituted into the calibration equation determined in step S03 with the determined calibration parameters, the position coordinates of the cursor on the screen of the signal source, i.e., the screen coordinates (X, Y), are calculated, and then the screen coordinates are transmitted to the signal source, and the signal source moves the cursor to this screen coordinate (X, Y) position.
Compared with the prior art, the embodiment of the invention provides a method for calibrating the position of a cursor on a screen, which comprises the steps of displaying a calibration graph with a plurality of calibration points on the screen through a signal source, acquiring screen coordinates of the calibration points, and acquiring an image of the screen where the calibration graph is located through a camera to acquire image coordinates of the calibration points displayed on the image; the method comprises the steps of establishing a first mapping relation between image coordinates of each pixel point of an image acquired by a camera and screen coordinates of each pixel point of a signal displayed on a screen by a signal source, and calibrating a cursor of the signal source on the screen according to the first mapping relation.
In a second aspect, as shown in fig. 4, an embodiment of the present invention discloses another method for calibrating a cursor position for screen interaction, which specifically includes the following steps:
step S21, dividing the screen into a plurality of regions;
step S22, establishing a second mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the calibration graph in each area;
step S23, determining the spot position formed on the screen by the light with certain wavelength emitted by the emitter, and determining the area of the spot according to the spot position;
and step S24, calibrating the cursor of the signal source on the screen according to the second mapping relation.
It should be noted that the screen may be divided into a plurality of areas, that is, the screen is discretized, multi-point calibration is performed on each discrete area to form a plurality of sets of mapping relationships, and the corresponding mapping relationship is determined according to the area where the light spot is located, and then the cursor coordinate position is calculated, in this case, it is also necessary to perform multi-point calibration on the entire screen, because the coordinate position where the light spot is located needs to be determined first, and the area where the light spot is located is determined according to the location where the light spot is located; if the discrete degree is higher and the discrete area is smaller, firstly carrying out high-order multi-point calibration on the whole screen to establish a primary mapping relation, and then respectively carrying out low-order few-point calibration on each discrete small area to establish a secondary mapping relation; and determining the discrete small area to which the light spot belongs through primary mapping, and determining the final position of the cursor according to corresponding secondary mapping.
Step S23, the determining the spot position formed on the screen by the emitter emitting light with a certain wavelength includes:
establishing a third mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the calibration graph in the whole screen;
acquiring an image of a screen where a light spot is located through a camera, and sending the image to a server for analysis and calculation to obtain an image coordinate for displaying the light spot on the image;
bringing the image coordinates into a third mapping relation, and calculating screen coordinates of a cursor of the signal source on a screen;
and determining the position of the light spot through the screen coordinates.
It should be noted that the calibration equation specifically obtaining the second mapping relationship and the third mapping relationship is determined in the following manner:
acquiring a calibration equation, wherein the calibration equation is used for expressing the mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the whole screen or each area in the whole screen, and the number of the calibration points is more than that of the calibration coefficients of the calibration equation, so that an overdetermined equation can be obtained;
screen coordinates and corresponding image coordinates of the calibration points in the whole screen or in each area of the whole screen are brought into the over-determined equations one by one to obtain a plurality of groups of coefficients of the over-determined equations of the whole screen or in each area of the whole screen, and a plurality of calibration equations of the whole screen or in each area of the whole screen are obtained according to the coefficients of the plurality of groups of the over-determined equations;
and respectively selecting an equation with the minimum calibration error from a plurality of calibration equations of the whole screen or each area in the whole screen as a calibration equation for expressing the second mapping relation and the third mapping relation.
The calibration equation for the entire screen is:
XS(X,Y)＝K0*Xn+K1*Yn+K2*Xn-1*Y+K3*X*Yn-1+K4*Xn-1+K5*Yn-1+K6*Xn-2*Yn-2…+Kc-3X*Y+Kc-2X+Kc-1Y+Kc
YS(X,Y)＝K00*Xn+K1*Yn+K02*Xn-1*Y+K03*X*Yn-1+K04*Xn-1+K05*Yn-1+K06*Xn-2*Yn-2…Kb- 3X*Y+Kb-2X+Kb-1Y+Kb
the calibration equation of each area in the whole screen is as follows:
XS1(X,Y)＝K0*Xn1+K1*Yn1+K2*Xn1-1*Y+K3*X*Yn1-1+K4*Xn1-1+K5*Yn1-1+K6*Xn1-2*Yn1-2…+Kc-3X*Y+Kc-2X+Kc-1Y+Kc
YS1(X,Y)＝K00*Xn1+K1*Yn1+K02*Xn1-1*Y+K03*X*Yn1-1+K04*Xn1-1+K05*Yn1-1+K06*Xn-2*Yn -2…Kb-3X*Y+Kb-2X+Kb-1Y+Kb
wherein n1 is not less than 3, n1 is not more than n, n1 is an integer, K0… Kc and K00… Kb are calibration parameters, (X, Y) are image coordinates, (Xs, Ys), (Xs)1，Ys1) Are screen coordinates.
The above-mentioned order n1 of the whole screen is greater than the order n of each area of the whole screen.
The specific steps of specifically determining the second mapping relationship and the third mapping relationship are consistent with the first aspect of the embodiment of the present invention, and are described in detail in this step.
After the third mapping relation is determined according to the steps, the image coordinates of the light spot on the image shot by the camera and calculated by the server are brought into the third mapping relation, namely, the image coordinates are brought into a determined calibration equation, and the screen coordinates of the signal displayed on the screen by the light spot on the signal source can be calculated; and determining the position of the light spot according to the coordinate position of the screen, and calibrating the cursor of the signal source on the screen according to the second mapping relation.
Compared with the prior art, the embodiment of the invention provides a method for calibrating the position of a cursor on a screen, which comprises the steps of dividing the screen into a plurality of areas, namely discretizing the screen, respectively carrying out multi-point calibration on each discrete area to form a plurality of groups of mapping relations, determining the corresponding mapping relations according to the areas where the positions of the light points are located, and then calculating the position of the cursor.
In a third aspect, as shown in fig. 5, an embodiment of the present invention further discloses a cursor position calibration apparatus for screen interaction, which is characterized by comprising:
the display module 31 is used for displaying a calibration graph on a screen through a signal source, wherein a plurality of calibration points are arranged on the calibration graph, and the coordinates of the calibration points are screen coordinates of signals displayed on the screen by the signal source;
the calculation module 32 is configured to obtain an image of a screen where the calibration graph is located through the camera, and send the image to the server for analysis and calculation to obtain an image coordinate of the calibration point;
the establishing module 33 is configured to establish a first mapping relationship between image coordinates of each pixel point of an image obtained by the camera and screen coordinates of each pixel point of a signal displayed on a screen by the signal source according to the screen coordinates and the image coordinates of the plurality of calibration points;
and the calibration module 34 is configured to calibrate the cursor of the signal source on the screen according to the first mapping relationship.
The specific implementation process of the cursor position calibration device for screen interaction disclosed in the third aspect of the embodiment of the present invention is the same as the implementation process of the first aspect of the embodiment of the present invention, and is not repeated here.
Compared with the prior art, the embodiment of the invention provides a method for calibrating the position of a cursor on a screen, which comprises the steps of displaying a calibration graph with a plurality of calibration points on the screen through a signal source, acquiring screen coordinates of the calibration points, and acquiring an image of the screen where the calibration graph is located through a camera to acquire image coordinates of the calibration points displayed on the image; the method comprises the steps of establishing a first mapping relation between image coordinates of each pixel point of an image acquired by a camera and screen coordinates of each pixel point of a signal displayed on a screen by a signal source, and calibrating a cursor of the signal source on the screen according to the first mapping relation.
In a fourth aspect, as shown in fig. 6, an embodiment of the present invention further discloses a cursor position calibration apparatus for screen interaction, which is characterized in that the apparatus includes:
a dividing module 41 for dividing the screen into a plurality of regions;
the establishing module 42 is configured to establish a second mapping relationship between the screen coordinates and the image coordinates of the plurality of calibration points in the calibration graph in each region;
a determining module 43, configured to determine a position of a light spot formed on the screen by the light of a certain wavelength emitted by the emitter, and determine an area where the light spot is located according to the position of the light spot;
and the calibration module 44 is configured to calibrate the cursor of the signal source on the screen according to the second mapping relationship.
The specific implementation process of the cursor position calibration device for screen interaction disclosed in the fourth aspect of the embodiment of the present invention is the same as the implementation process of the second aspect of the embodiment of the present invention, and is not repeated here.
Compared with the prior art, the embodiment of the invention provides a method for calibrating the position of a cursor on a screen, which comprises the steps of dividing the screen into a plurality of areas, namely discretizing the screen, respectively carrying out multi-point calibration on each discrete area to form a plurality of groups of mapping relations, determining the corresponding mapping relations according to the areas where the positions of the light points are located, and then calculating the position of the cursor.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the embodiments of the present invention, and are not limited thereto; although embodiments of the present invention have been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (12)
1. A cursor position calibration method for screen interaction, comprising:
displaying a calibration graph on a screen through a signal source, wherein a plurality of calibration points are arranged on the calibration graph, and the coordinates of the calibration points are screen coordinates of signals displayed on the screen by the signal source;
acquiring an image of a screen where a calibration graph is located through a camera, and sending the image to a server for analysis and calculation to obtain image coordinates of the calibration point;
establishing a first mapping relation between the image coordinates of each pixel point of the image acquired by the camera and the screen coordinates of each pixel point of the signal displayed on the screen by the signal source according to the screen coordinates and the image coordinates of the calibration points;
and calibrating the cursor of the signal source on the screen according to the first mapping relation.
2. The method as claimed in claim 1, wherein the step of establishing a first mapping relationship between the image coordinates of each pixel point of the image obtained by the camera and the screen coordinates of each pixel point of the signal displayed on the screen by the signal source according to the screen coordinates and the image coordinates of the calibration points comprises:
acquiring a calibration equation, wherein the calibration equation is used for expressing the mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points, and the number of the calibration points is more than the number of calibration parameters of the calibration equation, so that an overdetermined equation can be obtained;
bringing the screen coordinates and the image coordinates of the calibration points into the overdetermined equations one by one to obtain a plurality of groups of coefficients of the overdetermined equations, and obtaining a plurality of calibration equations according to the plurality of groups of coefficients of the overdetermined equations;
the method comprises the steps of selecting an equation with the minimum calibration error from a plurality of calibration equations, and establishing a first mapping relation between image coordinates of each pixel point of an image acquired by a camera and screen coordinates of each pixel point of a signal displayed on a screen by a signal source.
3. A cursor position calibration method for screen interaction according to claim 2, wherein said calibration equation is:
XS(X,Y)＝K0*Xn+K1*Yn+K2*Xn-1*Y+K3*X*Yn-1+K4*Xn-1+K5*Yn-1+K6*Xn-2*Yn-2…+Kc-3X*Y+Kc-2X+Kc-1Y+Kc；
YS(X,Y)＝K00*Xn+K1*Yn+K02*Xn-1*Y+K03*X*Yn-1+K04*Xn-1+K05*Yn-1+K06*Xn-2*Yn-2…Kb-3X*Y+Kb-2X+Kb-1Y+Kb；
wherein n is not less than 3, n is an integer, K0… Kc and K00…KbFor calibration parameters, (X, Y) are image coordinates and (Xs, Ys) are screen coordinates.
4. The method for calibrating cursor position for screen interaction according to claim 1, wherein said calibrating the cursor of said signal source on the screen according to said first mapping relation comprises:
emitting light with a certain wavelength to a screen and forming a light spot on the screen;
acquiring an image of a screen where a light spot is located, and sending the image to a server for analysis and calculation to obtain an image coordinate for displaying the light spot on the image;
bringing the image coordinates into a first mapping relation, and calculating screen coordinates of a cursor of the signal source on a screen;
and sending the calculated screen coordinates to a signal source, and moving a cursor to the calculated screen coordinates by the signal source.
5. A method as claimed in any one of claims 1 to 4, wherein the calibration pattern is a curved surface having 16 calibration points.
6. A method for cursor position calibration for screen interaction as claimed in any one of claims 1 to 4, characterized in that the screen coordinates and image coordinates are represented by pixel values.
7. A cursor position calibration method for screen interaction, comprising:
dividing the screen into a plurality of regions;
establishing a second mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the calibration graph in each area;
determining the position of a light spot formed on a screen by emitting certain wavelength light, and determining the area of the light spot according to the position of the light spot;
and calibrating the cursor of the signal source on the screen according to the second mapping relation.
8. The method of claim 7, wherein determining a location of a spot formed on the screen by the emission of a wavelength of light by the emitter comprises:
establishing a third mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the calibration graph in the whole screen;
acquiring an image of a screen where a light spot is located through a camera, and sending the image to a server for analysis and calculation to obtain an image coordinate for displaying the light spot on the image;
bringing the image coordinates into a third mapping relation, and calculating screen coordinates of a cursor of the signal source on a screen;
and determining the position of the light spot through the screen coordinates.
9. A cursor position calibration method for screen interaction according to claim 8, wherein said second and third mappings are determined by:
acquiring a calibration equation, wherein the calibration equation is used for expressing the mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the whole screen or each area in the whole screen, and the number of the calibration points is more than that of the calibration coefficients of the calibration equation, so that an overdetermined equation can be obtained;
screen coordinates and corresponding image coordinates of the calibration points in the whole screen or in each area of the whole screen are brought into the overdetermined equations one by one to obtain a plurality of groups of coefficients of the overdetermined equations of the whole screen or in each area of the whole screen, and a plurality of calibration equations of the whole screen or in each area of the whole screen are obtained according to the plurality of groups of coefficients of the overdetermined equations;
and respectively selecting an equation with the minimum calibration error from a plurality of calibration equations of the whole screen or each area in the whole screen as a calibration equation for expressing the second mapping relation and the third mapping relation.
10. The method of cursor position calibration for screen interaction of claim 9,
the calibration equation for the whole screen is:
XS(X,Y)＝K0*Xn+K1*Yn+K2*Xn-1*Y+K3*X*Yn-1+K4*Xn-1+K5*Yn-1+K6*Xn-2*Yn-2…+Kc-3X*Y+Kc-2X+Kc-1Y+Kc
YS(X,Y)＝K00*Xn+K1*Yn+K02*Xn-1*Y+K03*X*Yn-1+K04*Xn-1+K05*Yn-1+K06*Xn-2*Yn-2…Kb-3X*Y+Kb-2X+Kb-1Y+Kb
the calibration equation of each area in the whole screen is as follows:
XS1(X,Y)＝K0*Xn1+K1*Yn1+K2*Xn1-1*Y+K3*X*Yn1-1+K4*Xn1-1+K5*Yn1-1+K6*Xn1-2*Yn1-2…+Kc-3X*Y+Kc-2X+Kc-1Y+Kc
YS1(X,Y)＝K00*Xn1+K1*Yn1+K02*Xn1-1*Y+K03*X*Yn1-1+K04*Xn1-1+K05*Yn1-1+K06*Xn-2*Yn-2…Kb- 3X*Y+Kb-2X+Kb-1Y+Kb
wherein n1 is not less than 3, n1 is not more than n, n1 is an integer, K0… Kc and K00… Kb are calibration parameters, (X, Y) are image coordinates, (Xs, Ys), (Xs)1，Ys1) Are screen coordinates.
11. A cursor position calibration device for screen interaction, comprising:
the display module is used for displaying a calibration graph on a screen through a signal source, a plurality of calibration points are arranged on the calibration graph, and the coordinates of the calibration points are screen coordinates of signals displayed on the screen by the signal source;
the calculation module is used for acquiring an image of a screen where the calibration graph is located through the camera, and sending the image to the server for analysis and calculation to obtain the image coordinates of the calibration point;
the establishing module is used for establishing a first mapping relation between the image coordinates of each pixel point of the image acquired by the camera and the screen coordinates of each pixel point of the signal displayed on the screen by the signal source according to the screen coordinates and the image coordinates of the plurality of calibration points;
and the calibration module is used for calibrating the cursor of the signal source on the screen according to the first mapping relation.
12. A cursor position calibration device for screen interaction, comprising:
a dividing module for dividing the screen into a plurality of regions;
the establishing module is used for establishing a second mapping relation between the screen coordinates and the image coordinates of a plurality of calibration points in the calibration graph in each area;
the device comprises a determining module, a light source module and a control module, wherein the determining module is used for determining the position of a light spot formed on a screen by transmitting light with certain wavelength through a transmitter and determining the area of the light spot according to the position of the light spot;
and the calibration module is used for calibrating the cursor of the signal source on the screen according to the second mapping relation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010176903.0A CN111462247B (en) | 2020-03-13 | 2020-03-13 | Cursor position calibration method and device for screen interaction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010176903.0A CN111462247B (en) | 2020-03-13 | 2020-03-13 | Cursor position calibration method and device for screen interaction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111462247A true CN111462247A (en) | 2020-07-28 |
| CN111462247B CN111462247B (en) | 2024-04-02 |
Family
ID=71680744
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010176903.0A Active CN111462247B (en) | 2020-03-13 | 2020-03-13 | Cursor position calibration method and device for screen interaction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111462247B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113601514A (en) * | 2021-09-29 | 2021-11-05 | 季华实验室 | Robot hand-eye calibration precision verification system |
| CN114543669A (en) * | 2022-01-27 | 2022-05-27 | 珠海亿智电子科技有限公司 | Mechanical arm calibration method, device, equipment and storage medium |
| CN114943764A (en) * | 2022-05-19 | 2022-08-26 | 苏州华兴源创科技股份有限公司 | Curved surface screen pixel positioning method, device and equipment |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101102391A (en) * | 2006-07-03 | 2008-01-09 | 微光科技股份有限公司 | Correction and tracking method for image processing of pointer input system |
| CN102323868A (en) * | 2011-10-26 | 2012-01-18 | 中国人民解放军国防科学技术大学 | Man-machine multipoint interaction method and device based on infrared image |
| CN102591531A (en) * | 2011-12-26 | 2012-07-18 | 深圳市巨龙科教高技术股份有限公司 | Electronic whiteboard, coordinate mapping method for same, device |
| CN102789340A (en) * | 2012-06-27 | 2012-11-21 | 深圳市巨龙科教高技术股份有限公司 | Method and device for obtaining coordinates of whiteboard of electronic whiteboard and electronic whiteboard |
| CN103092432A (en) * | 2011-11-08 | 2013-05-08 | 深圳市中科睿成智能科技有限公司 | Trigger control method and system of man-machine interaction operating instruction and laser emission device |
| US20140145948A1 (en) * | 2012-11-26 | 2014-05-29 | Everest Display Inc. | Interactive projection system and method for calibrating position of light point thereof |
| CN105300523A (en) * | 2015-10-09 | 2016-02-03 | 北京航空航天大学 | Polarization calibration method of light field polarization imaging system |
| US20160342279A1 (en) * | 2015-05-19 | 2016-11-24 | Boe Technology Group Co., Ltd. | Touch implementation method and device and electronic device |
| CN106257388A (en) * | 2016-07-25 | 2016-12-28 | 北京仁光科技有限公司 | A kind of localization method of optical touch screen |
| KR101691880B1 (en) * | 2015-09-23 | 2017-01-03 | (주) 이즈커뮤니케이션즈 | Interactive System Having Auto Calibration Function And Auto Calibration Method of Interactive System |
| CN106293270A (en) * | 2016-08-05 | 2017-01-04 | 南京仁光电子科技有限公司 | A kind of scaling method of giant-screen touch-control system |
| CN106406570A (en) * | 2015-07-29 | 2017-02-15 | 中兴通讯股份有限公司 | Projection cursor control method and device and remote controller |
| CN109144375A (en) * | 2018-10-09 | 2019-01-04 | 中天智领（北京）科技有限公司 | A kind of screen control method and device |
-
2020
- 2020-03-13 CN CN202010176903.0A patent/CN111462247B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101102391A (en) * | 2006-07-03 | 2008-01-09 | 微光科技股份有限公司 | Correction and tracking method for image processing of pointer input system |
| CN102323868A (en) * | 2011-10-26 | 2012-01-18 | 中国人民解放军国防科学技术大学 | Man-machine multipoint interaction method and device based on infrared image |
| CN103092432A (en) * | 2011-11-08 | 2013-05-08 | 深圳市中科睿成智能科技有限公司 | Trigger control method and system of man-machine interaction operating instruction and laser emission device |
| CN102591531A (en) * | 2011-12-26 | 2012-07-18 | 深圳市巨龙科教高技术股份有限公司 | Electronic whiteboard, coordinate mapping method for same, device |
| CN102789340A (en) * | 2012-06-27 | 2012-11-21 | 深圳市巨龙科教高技术股份有限公司 | Method and device for obtaining coordinates of whiteboard of electronic whiteboard and electronic whiteboard |
| US20140145948A1 (en) * | 2012-11-26 | 2014-05-29 | Everest Display Inc. | Interactive projection system and method for calibrating position of light point thereof |
| US20160342279A1 (en) * | 2015-05-19 | 2016-11-24 | Boe Technology Group Co., Ltd. | Touch implementation method and device and electronic device |
| CN106406570A (en) * | 2015-07-29 | 2017-02-15 | 中兴通讯股份有限公司 | Projection cursor control method and device and remote controller |
| KR101691880B1 (en) * | 2015-09-23 | 2017-01-03 | (주) 이즈커뮤니케이션즈 | Interactive System Having Auto Calibration Function And Auto Calibration Method of Interactive System |
| CN105300523A (en) * | 2015-10-09 | 2016-02-03 | 北京航空航天大学 | Polarization calibration method of light field polarization imaging system |
| CN106257388A (en) * | 2016-07-25 | 2016-12-28 | 北京仁光科技有限公司 | A kind of localization method of optical touch screen |
| CN106293270A (en) * | 2016-08-05 | 2017-01-04 | 南京仁光电子科技有限公司 | A kind of scaling method of giant-screen touch-control system |
| CN109144375A (en) * | 2018-10-09 | 2019-01-04 | 中天智领（北京）科技有限公司 | A kind of screen control method and device |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113601514A (en) * | 2021-09-29 | 2021-11-05 | 季华实验室 | Robot hand-eye calibration precision verification system |
| CN113601514B (en) * | 2021-09-29 | 2022-01-18 | 季华实验室 | Robot hand-eye calibration precision verification system |
| CN114543669A (en) * | 2022-01-27 | 2022-05-27 | 珠海亿智电子科技有限公司 | Mechanical arm calibration method, device, equipment and storage medium |
| CN114543669B (en) * | 2022-01-27 | 2023-08-01 | 珠海亿智电子科技有限公司 | Mechanical arm calibration method, device, equipment and storage medium |
| CN114943764A (en) * | 2022-05-19 | 2022-08-26 | 苏州华兴源创科技股份有限公司 | Curved surface screen pixel positioning method, device and equipment |
| WO2023221290A1 (en) * | 2022-05-19 | 2023-11-23 | 苏州华兴源创科技股份有限公司 | Pixel positioning method, apparatus, and device for curved screen |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111462247B (en) | 2024-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111462247B (en) | Cursor position calibration method and device for screen interaction | |
| US11310480B2 (en) | Systems and methods for determining three dimensional measurements in telemedicine application | |
| US7190331B2 (en) | System and method for measuring the registration accuracy of an augmented reality system | |
| CN107505324B (en) | 3D scanning device and scanning method based on binocular collaborative laser | |
| CN107025663A (en) | It is used for clutter points-scoring system and method that 3D point cloud is matched in vision system | |
| KR101896339B1 (en) | Method and apparatus for calibration of light field display using multi-projectors | |
| CN105306922A (en) | Method and device for obtaining depth camera reference diagram | |
| CN111047650B (en) | Parameter calibration method for time-of-flight camera | |
| CN110703230A (en) | Position calibration method between laser radar and camera | |
| CN106686281B (en) | Fuse circuit board noise suppression ability test system | |
| CN111161358A (en) | Camera calibration method and device for structured light depth measurement | |
| WO2018168757A1 (en) | Image processing device, system, image processing method, article manufacturing method, and program | |
| CN111738971B (en) | Circuit board stereoscopic scanning detection method based on line laser binocular stereoscopic vision | |
| CN109215128B (en) | Object motion attitude image synthesis method and system | |
| Fujii et al. | A new flexible acquisition system of ray-space data for arbitrary objects | |
| KR101683763B1 (en) | Augmented Reality Robot Simulation System And Method Using 360 Degree Camera | |
| CN111462248A (en) | Calibration method and device for cursor position calibration parameters in screen interaction | |
| CN112308933B (en) | Method and device for calibrating camera internal reference and computer storage medium | |
| CN115407302A (en) | Laser radar pose estimation method and device and electronic equipment | |
| JP2019207127A (en) | Laser calibration device, calibration method therefor, and image input device including laser calibration device | |
| US9582887B2 (en) | Methods and apparatus for determining field of view dependent depth map correction values | |
| Smith et al. | Automatic Sub-pixel Projector Calibration Supporting Improved Calibration for Projected Environments | |
| CN114863036B (en) | Data processing method and device based on structured light, electronic equipment and storage medium | |
| JP2016192600A (en) | Information processor and projection apparatus | |
| US20190230460A1 (en) | Method and apparatus for creating a three-dimensional scenario |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |