CN110506252A - Based on the transformational relation positioning terminal screen for indicating graphical dots coordinate in pattern - Google Patents

Based on the transformational relation positioning terminal screen for indicating graphical dots coordinate in pattern Download PDF

Info

Publication number
CN110506252A
CN110506252A CN201780089459.5A CN201780089459A CN110506252A CN 110506252 A CN110506252 A CN 110506252A CN 201780089459 A CN201780089459 A CN 201780089459A CN 110506252 A CN110506252 A CN 110506252A
Authority
CN
China
Prior art keywords
coordinate
image
screen
pattern
logo
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201780089459.5A
Other languages
Chinese (zh)
Other versions
CN110506252B (en
Inventor
祝诗扬
罗巍巍
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of CN110506252A publication Critical patent/CN110506252A/en
Application granted granted Critical
Publication of CN110506252B publication Critical patent/CN110506252B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Telephone Function (AREA)

Abstract

A method of based on the transformational relation positioning terminal screen for indicating graphical dots coordinate in pattern, solving the problems, such as that terminal screen locating accuracy is lower.This method comprises: obtaining the first image, and determine the transformational relation between the first coordinate and the second coordinate, and according to transformational relation, determines the region of screen in the first image.Wherein, the first image is the image of the terminal of shooting, and terminal is provided with screen, and screen shows pattern, and pattern includes the contour pattern of regular distribution；First coordinate is for indicating the position of contour pattern on the screen, and the second coordinate is for indicating position of the contour pattern in the first image.

Description

Terminal screen is fixed a position to transform relation based on mark figure point coordinate in pattern Technical Field
The present application relates to the field of image processing technologies, and in particular, to a method and an apparatus for positioning a terminal screen, and a terminal.
Background
With the popularization of touch screen terminals such as mobile phones and tablet computers, users have higher requirements for terminal touch screen sensitivity and the like. At present, in order to provide better touch screen experience for a user, the user can be simulated to click, slide and other operations on a touch screen terminal through equipment such as a mechanical arm, so that the tests of touch screen accuracy, sensitivity and the like are completed through test equipment, and the touch screen terminal is improved according to a test result.
In the actual test process, the industrial camera can process the shot image of the terminal to realize the positioning of the boundary of the terminal screen, then find an intersection point according to the identified boundaries of the 4 sides to determine the corner point of the terminal screen, finally position the terminal screen from the collected image, and transmit the positioning result to the mechanical arm. That is, an operation such as clicking or sliding is performed on a certain point of the photographed image, and the robot arm actually clicks or slides at a position of the terminal screen corresponding to the point. In the image processing process, the terminal screen area and the non-terminal screen area in the image are mainly divided according to the threshold, and the different sizes of the threshold often affect the positioning result of the boundary, so the boundary of the terminal screen often cannot be accurately positioned in the implementation process, and the mechanical arm cannot accurately simulate the actual clicking, sliding and other operations of a user, taking the clicking operation as an example, namely, the position of the mechanical arm performing the clicking operation on the terminal screen does not correspond to the position of the user actually trying to click, so that the whole testing process is affected, and finally the accuracy of the testing result is low.
Disclosure of Invention
The embodiment of the invention provides a terminal screen positioning method and device and a terminal, and can solve the problem of low terminal screen positioning accuracy.
In a first aspect, an embodiment of the present invention provides a method for positioning a terminal screen. The method is applied to a terminal such as a test device. The method comprises the following steps: and acquiring the first image, then determining a conversion relation between the first coordinate and the second coordinate, and determining the area of the screen in the first image according to the conversion relation. The first image is a shot terminal image, the shot terminal is provided with a screen, and the screen displays a pattern which comprises regularly distributed sign graphs; the first coordinates are used to represent the position of the logo graphic on the screen and the second coordinates are used to represent the position of the logo graphic in the first image. Compared with the prior art that the terminal screen area and the non-terminal screen area in the image are divided according to the threshold value, the method and the device for determining the coordinate transformation relation between the terminal screen and the image can effectively determine the coordinate transformation relation between the terminal screen and the image, and then determine the area of the terminal screen according to the transformation relation.
In one possible design, determining the area of the screen in the first image according to the transformation relationship may be implemented as: and determining a fourth coordinate according to the conversion relation and the third coordinate, and determining an area formed by taking the positions of all corner points corresponding to the fourth coordinate as end points in the first image as an area of the screen in the first image. The third coordinate is used for representing the position of each corner point on the screen in the screen, and the fourth coordinate is used for representing the position of each corner point in the first image. The embodiment of the invention can accurately position the corner point position of the terminal screen in the image by adopting a coordinate conversion mode, and then accurately determine the screen edge according to the positions of 4 corner points, thereby accurately positioning the terminal screen in the image.
In one possible design, the pattern includes at least three logo images. In the process of shooting the image of the mobile phone by the industrial camera, the lens of the industrial camera is probably not completely parallel to the screen of the mobile phone, namely shooting is carried out at a certain angle, and the obtained image has certain distortion, namely the originally rectangular mobile phone screen is probably regular or irregular quadrangle. In general, two points can determine a straight line, and three points can determine a plane, whereas in the embodiment of the present invention, if there is distortion in the image and an attempt is made to determine the area of the mobile phone screen in the image, it can be regarded as a process of determining a plane, so that at least 3 points need to be determined to calculate the plane. And each point position can correspond to a mark figure, therefore, in the embodiment of the invention, the number of the mark figures in the pattern can be set to be more than or equal to 3.
In one possible design, the regular distribution includes the center points of the logo patterns being arranged in a matrix of rows and columns at equal intervals. Taking the symbol pattern as a circle as an example, the distance between every two adjacent circle centers in the transverse direction is the same, and the distance between every two adjacent circle centers in the longitudinal direction is the same. In addition, in the embodiment of the present invention, the mark patterns may be distributed according to other rules, not limited to the rule that the distances between the center points of two adjacent mark patterns in the same direction are equal, but also may be a case that the positions of the mark patterns are set according to a variable step length in the same direction, for example, in the same direction, the distance between the center points of a first mark pattern and a second mark pattern is X, the distance between the center points of a second mark pattern and a third mark pattern is 2X, and so on, every time a mark pattern is added, the distance between the added mark pattern and the adjacent previous mark pattern is increased by X; alternatively, in the same direction, the distance between the center points of the first and second logo patterns is X, the distance between the center points of the second and third logo patterns is X +2, and so on, every time a logo pattern is added, the distance between the added logo pattern and the adjacent previous logo pattern is increased by 2, and so on, which is not limited herein. Of course, the step length with variability may be a function in which an argument changes continuously, or may be in other implementation forms, and is not limited herein.
In one possible design, the logo patterns include a regular logo pattern and a special logo pattern, the regular logo pattern is identical, the special logo pattern is different from the regular logo pattern, and the special logo pattern is used for distinguishing the placement direction of the terminal when shooting. For example, the area of the special logo graphic may be significantly larger or smaller than the conventional logo graphic in the design.
In one possible design, the logo is black and the areas of the design other than the logo are white. Considering that an image obtained after the image is shot by the industrial camera is a gray image, if only two colors of black and white exist in the pattern, the imaging effect is better. Therefore, the color of the pattern is not limited, and the pattern can be configured in a more obvious way for other colors, for example, the logo is dark, and the rest is light.
In one possible design, the position of the center point of the logo pattern is unchanged after the logo pattern is deformed due to the change of the shooting angle. Note that, each of the mark patterns in the pattern may be a pattern such as a circle or an ellipse, and even if the pattern is deformed, the center point position is not affected, and the shape of the mark pattern is not limited herein.
In one possible design, the resolution of the pattern is the same as the resolution of the screen. Of course, the resolution of the pattern may also be in proportion to the resolution of the screen, that is, a pattern with the resolution equal to or in proportion to the resolution of the mobile phone screen may be generated on the mobile phone screen according to the resolution of the mobile phone screen. For example, the resolution of the mobile phone screen is 1280 × 720, and then the resolution of the pattern may be 1280 × 720 or a × B, where a is a product of 1280 and a first ratio, B is a product of 720 and a second ratio, and the first ratio and the second ratio may be the same or different.
In a second aspect, an embodiment of the present invention provides a method for positioning a terminal screen. The method comprises the following steps: the terminal displays the pattern through the screen, the industrial camera acquires the first image and determines the second coordinate, the testing equipment acquires the first coordinate and the second coordinate, determines the conversion relation between the first coordinate and the second coordinate, and determines the area of the screen in the first image according to the conversion relation. The pattern comprises mark graphs which are regularly distributed, and the first coordinates are used for representing the positions of the mark graphs in the screen; the second coordinates are used to represent the position of the logo in the first image, which is the image of the terminal being photographed.
In one possible design, after the test equipment determines the conversion relationship between the first coordinates and the second coordinates, the method further includes: the test equipment acquires the fifth coordinate, converts the fifth coordinate according to the conversion relation to obtain a sixth coordinate, and then sends the sixth coordinate to the mechanical arm; the robotic arm performs an operation at a location of the sixth coordinate in the first image in response to the message sent by the testing device. Wherein the fifth coordinate is used for representing the position to be tested on the screen, and the sixth coordinate is used for representing the position of the position to be tested in the first image.
In a third aspect, an embodiment of the present invention provides a terminal cleaning device. The device can realize the functions realized by the terminal in the embodiment of the method, and the functions can be realized by hardware or by executing corresponding software by hardware. The hardware or software comprises one or more modules corresponding to the functions.
In a fourth aspect, an embodiment of the present invention provides a terminal. The structure of the terminal comprises a display screen, a memory, one or more processors, a plurality of application programs and one or more programs; wherein the one or more programs are stored in the memory; the one or more processors, when executing the one or more programs, cause the terminal to perform the method of the first aspect and any of its various possible designs.
In a fifth aspect, an embodiment of the present invention provides a readable storage medium, including instructions. The instructions, when executed on the terminal, cause the terminal to perform the method of any one of the first aspect and its various possible designs above.
In a sixth aspect, embodiments of the present invention provide a computer program product comprising software code for performing the method of any one of the first aspect and its various possible designs.
Drawings
Fig. 1 is a schematic structural diagram of a first terminal according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a process of shooting a mobile phone by an industrial camera according to an embodiment of the present invention;
fig. 3 is an image of a mobile phone taken by a first industrial camera according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a pattern according to an embodiment of the present invention;
fig. 5 is an image of a mobile phone taken by a second industrial camera according to an embodiment of the present invention;
fig. 6 is a schematic diagram of a dotting position for testing the sensitivity of a mobile phone screen according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a straight line fitting for determining a screen area of a mobile phone according to an embodiment of the present invention;
FIG. 8 is a black-and-white image after a first process according to an embodiment of the present invention;
fig. 9 is a schematic diagram of a main interface of a mobile phone according to an embodiment of the present invention;
fig. 10 is a schematic diagram illustrating a two-dimensional coordinate system established on a mobile phone screen according to an embodiment of the present invention;
FIG. 11 is a black-and-white image after a second process according to an embodiment of the present invention;
fig. 12 is a schematic structural diagram of a positioning apparatus for a terminal screen according to an embodiment of the present invention;
fig. 13 is a schematic structural diagram of a second terminal according to an embodiment of the present invention.
Description of reference numerals:
201-lens of industrial camera;
202-mobile phone;
203-placement area of test equipment;
204-special logo graphics in the pattern;
205-interference pattern;
206-cell phone screen;
207-edge of bright screen area of the handset.
Detailed Description
The embodiment of the invention can be used for a terminal, and the terminal can comprise a notebook computer, a smart phone, an industrial camera, a test device, a Virtual Reality (VR) device, an Augmented Reality (AR) device, a vehicle-mounted device, an intelligent wearable device and other devices. The terminal is provided with at least a display screen, an input device, and a processor, and taking the terminal 100 as an example, as shown in fig. 1, the terminal 100 includes components such as a processor 101, a memory 102, a camera 103, an RF circuit 104, an audio circuit 105, a speaker 106, a microphone 107, an input device 108, another input device 109, a display screen 110, a touch panel 111, a display panel 112, an output device 113, and a power supply 114. The display screen 110 is composed of at least a touch panel 111 as an input device and a display panel 112 as an output device. It should be noted that the terminal structure shown in fig. 1 is not limited to the terminal, and may include more or less components than those shown in the drawings, or combine some components, or split some components, or arrange different components, and is not limited herein.
The various components of the terminal 100 will now be described in detail with reference to fig. 1:
a Radio Frequency (RF) circuit 104 may be configured to receive and transmit signals during information transmission and reception or during a call, for example, if the terminal 100 is a mobile phone, the terminal 100 may receive downlink information transmitted by a base station through the RF circuit 104 and then transmit the downlink information to the processor 101 for processing; in addition, data relating to uplink is transmitted to the base station. Typically, the RF circuitry includes, but is not limited to, an antenna, at least one Amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, the RF circuitry 104 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.
The memory 102 may be used to store software programs and modules, and the processor 101 executes various functional applications and data processing of the terminal 100 by operating the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function (e.g., a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (e.g., audio data, video data, etc.) created according to the use of the terminal 100, and the like. Further, the memory 102 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
Other input devices 109 may be used to receive input numeric or character information and generate key signal inputs relating to user settings and function control of terminal 100. In particular, other input devices 109 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, a light mouse (a light mouse is a touch-sensitive surface that does not display visual output, or is an extension of a touch-sensitive surface formed by a touch screen), and the like. Other input devices 109 may also include sensors built into terminal 100, such as gravity sensors, acceleration sensors, etc., and terminal 100 may also use parameters detected by the sensors as input data.
The display screen 110 may be used to display information input by or provided to the user and various menus of the terminal 100, and may also accept user input. In addition, the Display panel 112 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like to configure the Display panel 112; the touch panel 111, also referred to as a touch screen, a touch-sensitive screen, etc., may collect contact or non-contact operations (for example, operations performed by a user on or near the touch panel 111 using any suitable object or accessory such as a finger, a stylus, etc., and may also include body-sensing operations, where the operations include single-point control operations, multi-point control operations, etc., and drive the corresponding connection device according to a preset program. It should be noted that the touch panel 111 may further include two parts, namely, a touch detection device and a touch controller. The touch detection device detects the touch direction and gesture of a user, detects signals brought by touch operation and transmits the signals to the touch controller; the touch controller receives touch information from the touch detection device, converts the touch information into information that can be processed by the processor 101, and transmits the information to the processor 101, and also receives and executes commands sent by the processor 101. In addition, the touch panel 111 may be implemented by various types such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave, and the touch panel 111 may also be implemented by any technology developed in the future. In general, the touch panel 111 may cover the display panel 112, a user may operate on or near the touch panel 111 covered on the display panel 112 according to the content displayed on the display panel 112 (the display content includes, but is not limited to, a soft keyboard, a virtual mouse, virtual keys, icons, etc.), the touch panel 111 detects the operation on or near the touch panel 111, and transmits the operation to the processor 101 to determine a user input, and then the processor 101 provides a corresponding visual output on the display panel 112 according to the user input. Although in fig. 1, the touch panel 111 and the display panel 112 are two separate components to implement the input and output functions of the terminal 100, in some embodiments, the touch panel 111 and the display panel 112 may be integrated to implement the input and output functions of the terminal 100.
RF circuitry 104, speaker 106, and microphone 107 may provide an audio interface between a user and terminal 100. The audio circuit 105 may transmit the converted signal of the received audio data to the speaker 106, and the converted signal is converted into a sound signal by the speaker 106 and output; alternatively, the microphone 107 may convert the collected sound signals into signals, convert the signals into audio data after being received by the audio circuit 105, and output the audio data to the RF circuit 104 to be transmitted to a device such as another terminal, or output the audio data to the memory 102 for further processing by the processor 101 in conjunction with the content stored in the memory 102. In addition, the camera 103 may capture image frames in real time and transmit them to the processor 101 for processing, and store the processed results in the memory 102 and/or present the processed results to the user through the display panel 112.
The processor 101 is a control center of the terminal 100, connects various parts of the entire terminal 100 using various interfaces and lines, performs various functions of the terminal 100 and processes data by running or executing software programs and/or modules stored in the memory 102 and calling data stored in the memory 102, thereby monitoring the terminal 100 as a whole. It is noted that processor 101 may include one or more processing units; the processor 101 may also integrate an application processor, which mainly handles operating systems, User Interfaces (UIs), application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 101.
The terminal 100 may further include a power supply 114 (e.g., a battery) for supplying power to various components, and in an embodiment of the present invention, the power supply 114 may be logically connected to the processor 101 through a power management system, so as to implement functions of managing charging, discharging, and power consumption through the power management system.
In addition, there are also components not shown in fig. 1, for example, the terminal 100 may further include a bluetooth module and the like, which are not described herein again.
The following describes the technical solution of the embodiment of the present invention by taking the terminal 100 shown in fig. 1 as a mobile phone and an industrial camera as examples. The industrial camera can be regarded as a part of the test equipment, and for the test equipment, the industrial camera can transmit a shot image of the mobile phone to the test equipment, and the test equipment performs analysis processing.
In an embodiment of the invention, the industrial camera is used to take an image of the included cell phone, which is then processed by the industrial camera and/or the cell phone to locate the screen area of the cell phone. It should be noted that the method is also applicable to positioning the peripheral outline of the mobile phone or positioning a specific area presented on the screen of the mobile phone, and is not limited herein.
Fig. 2 is a schematic diagram illustrating a process of shooting a mobile phone by an industrial camera. The mobile phone 202 is placed in the placement area of the testing device, and the plane of the lens 201 of the industrial camera is parallel to the plane of the mobile phone 202. It should be noted that, in the embodiment of the present invention, the industrial camera may be regarded as a part of the testing device, that is, the industrial camera, the placement area, and other components together form the testing device, but of course, the industrial camera may also be separately provided with the testing device, and then the industrial camera may transmit the acquired image to the testing device or other devices such as a mobile phone for processing, or the industrial camera may transmit the processed image to the testing device or other devices such as a mobile phone for transmitting to the testing device, which is not limited herein.
As shown in fig. 3, an image of a cell phone taken for an industrial camera. The image obtained by shooting with the industrial camera is a grayscale image, and this image is only one possible shooting example, and is not a limitation on the image obtained by shooting.
In the actual shooting process, the placement angle of the mobile phone, the distance between the mobile phone and the industrial camera, the view angle of the industrial camera relative to the mobile phone, and the like all affect the image, that is, the image of the mobile phone shot by the industrial camera is different according to the actual shooting, which is not limited herein. That is, in the process of actually shooting the mobile phone image, it is likely that the industrial camera is not shot perpendicular to the plane of the mobile phone due to the fact that the placement area of the testing device is not smooth, or the plane of the placement area forms a certain angle with the plane of the lens of the industrial camera, and the like, so that a certain error exists in the shot image in the mobile phone screen area, for example, the mobile phone screen area is rectangular under the condition of normal shooting, and is irregular or regular quadrangle due to distortion at present. In addition, the image presented by the distorted mobile phone screen can be influenced by the actual shape of the mobile phone screen area.
In the embodiment of the present invention, as shown in fig. 4, a pattern with a resolution equal to or proportional to the resolution of the mobile phone screen may be generated on the mobile phone screen according to the resolution of the mobile phone screen, for example, the resolution of the mobile phone screen is 1280 × 720, and the resolution of the pattern may be 1280 × 720 or a × B, where a is a product of 1280 and a first ratio, B is a product of 720 and a second ratio, and the first ratio and the second ratio may be the same or different. In the pattern, a plurality of circular logo patterns are regularly distributed. The logo graphics include conventional logo graphics and special logo graphics. The conventional mark figures are completely the same, the special mark figures are used for distinguishing the placement directions of the mobile phone, and the area of the special mark figures is obviously larger than that of the conventional mark figures in the pattern.
In the embodiment of the invention, the circle centers of all the circular mark patterns in the pattern are distributed in a matrix form according to the row and the column at equal intervals, namely, the distance between every two adjacent circle centers in the transverse direction is the same, and the distance between every two adjacent circle centers in the longitudinal direction is the same. In addition, the circle in the pattern is black, and the portion other than the logo pattern in the pattern is white. Note that each of the mark patterns in the pattern may be an ellipse, for example, and even if the mark pattern is deformed, the shape of the mark pattern is not limited.
In addition, the regular distribution in the embodiment of the present invention is not limited to the rule that the distances between the center points of two adjacent mark patterns in the same direction are equal, and may also be a case where the positions of the mark patterns are set according to a variable step length in the same direction, for example, in the same direction, the distance between the center points of a first mark pattern and a second mark pattern is X, the distance between the center points of a second mark pattern and a third mark pattern is 2X, and so on, the distance between the added mark pattern and the adjacent previous mark pattern is increased by X for each increase of one mark pattern; alternatively, in the same direction, the distance between the center points of the first and second logo patterns is X, the distance between the center points of the second and third logo patterns is X +2, and so on, every time a logo pattern is added, the distance between the added logo pattern and the adjacent previous logo pattern is increased by 2, and so on, which is not limited herein.
Of course, the step length with variability may be a function in which an argument changes continuously, or may be in other implementation forms, and is not limited herein. In addition, for the distribution of colors in the pattern, in the embodiment of the present invention, the pattern is presented in a form of alternating black and white, that is, the logo is black, and the rest is white. Considering that an image obtained after the image is shot by the industrial camera is a gray image, if only two colors of black and white exist in the pattern, the imaging effect is better. Therefore, the color of the pattern is not limited, and the pattern can be configured in a more obvious way for other colors, for example, the logo is dark, and the rest is light.
In addition, one or more special mark patterns can be arranged in the pattern, and the aim of the special mark patterns is to distinguish the placing direction of the mobile phone. Specific differentiation will be provided later, and will not be described herein. In the embodiment of the present invention, the special logo pattern is distinguished from other logo patterns by adjusting the size of the logo patterns, which is not limited herein.
In addition, in the embodiment of the present invention, the process of generating the pattern may be executed by a mobile phone, or may be generated in an industrial camera, a testing device, or other devices according to the resolution of a screen of the mobile phone, and the generated pattern is transmitted to the mobile phone for full-screen display of the mobile phone, where the position, manner, and the like of generating the pattern are not limited.
Taking the regularly distributed pattern as an example, if the resolution of the mobile phone screen is 1280 × 720, the generated pattern can be regarded as a picture with the resolution of 1280 × 720. For example, the distribution of the logo patterns in the pattern may be 15 × 15, that is, the horizontal and vertical distributions are 15 logo patterns, then the distance between the center points of every two logo patterns in the 15 logo patterns in the horizontal direction (the direction of the edge corresponding to 1280 pixels in the mobile phone screen) is 80, and the vertical distance from the center point of the logo pattern close to the longitudinal edge of the mobile phone screen to the longitudinal edge is also 80, that is, 80 pixels, where the calculation mode of 80 pixels is 1280/(15+1) is 80, the distance between the center points of every two logo patterns in the 15 logo images in the vertical direction (the direction of the edge corresponding to 720 pixels in the mobile phone screen) is 45, and the vertical distance from the center point of the logo pattern close to the lateral edge of the mobile phone screen to the lateral edge is also 45, that is, 45 pixels. Taking the special logo as an example, the vertical distance from the special logo to the transverse edge of the screen is 45, and the vertical distance from the special logo to the longitudinal edge of the screen is 80, where the distance is from the center point to the edge of the logo. If the position of the special mark pattern is closer to the origin of coordinates of the mobile phone screen (an angular point of the mobile phone screen, namely an intersection point formed after two adjacent sides of the mobile phone screen are intersected), and the angular point is taken as the origin of coordinates, the direction far away from the origin on the transverse side of the mobile phone screen is the positive direction of an x axis in a two-dimensional coordinate system, the direction far away from the origin on the longitudinal side of the mobile phone screen is the positive direction of a y axis in the two-dimensional coordinate system, and when the coordinates of the angular point taken as the origin are (0,0), the coordinates of the center point of the special mark pattern are (80, 45). By analogy, the coordinates of the central point of each mark graph can be obtained when the pattern is displayed on the screen of the mobile phone.
It can be seen that each marker pattern present in the pattern can be regarded as a set of marker patterns, denoted as P ═ { P1, P2, …, Pn }, where n is an integer greater than or equal to 3, and the center point coordinates of each marker pattern can be regarded as a set of center points of the marker patterns on the screen pixel coordinates, denoted as Ps ═ Ps1, Ps2, …, Psn }. It should be noted that, in the process of taking an image of a mobile phone by an industrial camera, it is likely that a lens of the industrial camera is not completely parallel to a screen of the mobile phone, that is, the image is taken at a certain angle, and thus the obtained image has a certain distortion, that is, an originally rectangular mobile phone screen is likely to be a regular or irregular quadrangle. In general, two points can determine a straight line, and three points can determine a plane, whereas in the embodiment of the present invention, if there is distortion in the image and an attempt is made to determine the area of the mobile phone screen in the image, it can be regarded as a process of determining a plane, so that at least 3 points need to be determined to calculate the plane. And each point position can correspond to a mark figure, therefore, in the embodiment of the invention, the number of the mark figures in the pattern can be set to be more than or equal to 3. It should be noted that, in order to determine the coordinate transformation relationship of the center points of the logo patterns in the images captured by the mobile phone screen and the industrial camera as accurately as possible while ensuring that the placement direction of the mobile phone can be effectively distinguished, at least two coordinate pairs of the center points of the logo patterns (i.e., the coordinates of the same center point in the images captured by the mobile phone screen and the industrial camera) are required to be calculated as initial parameters. It should be noted that, the more the number of the marker figures in the pattern is, the more accurate the coordinate conversion relationship is obtained, and the less the number of the marker figures in the pattern is, the less resources are occupied for calculating the coordinate conversion relationship. Therefore, in the embodiment of the present invention, the current actual requirement may be combined to balance the resource occupation situation and the accuracy of the coordinate transformation relationship, so as to determine how many sign graphs are configured in the currently adopted pattern, which is not limited herein.
As shown in fig. 5, the screen of the mobile phone displays a regularly distributed pattern determined according to the resolution, the industrial camera is perpendicular to the mobile phone, and a mobile phone image obtained after shooting the mobile phone is displayed. The captured grayscale image may include not only the logo pattern but also an interference pattern. In the embodiment of the invention, the interference pattern can be a regular or irregular area, point and the like generated by external environment factors such as illumination and the like, or the lens of an industrial camera is not clean, or the display screen of a mobile phone is not clean and the like. It can be known from the captured image that all the logo patterns included in the pattern may not be fully captured by the industrial camera, such as 205 shown in fig. 5, which means that the image captured by the industrial camera is likely to include only a portion of the logo patterns. And because of this, the more logo patterns included in the pattern, the more accurate the processing result obtained after the acquired image is finally processed.
On the basis of the image shown in fig. 5, a two-dimensional coordinate system is established on the basis of the resolution of the mobile phone screen, and the coordinates of the center point of each mark figure in the image are determined. It should be noted that, because the image has the interference pattern, it is likely that some of the marker patterns cannot be accurately determined, and the coordinates of the center point of each marker pattern that cannot be accurately determined are difficult to be determined from the acquired image. In the embodiment of the present invention, the coordinates of the center point of the existing logo can be combined to calculate the coordinates of the center point of the missing or incomplete logo, and then the center point set of the logo on the image pixel coordinates is obtained and is marked as Pc ═ Pc1, Pc2, …, Pcn }.
The method for calculating the coordinates of the center point of the missing or incomplete logo pattern may be at least one of the following:
calculating the increasing and decreasing amplitude or increasing and decreasing rules of the central point coordinates of two adjacent sign graphs in the transverse direction and the longitudinal direction according to the determined central point coordinates of the sign graphs in the transverse direction, the longitudinal direction or the oblique direction, and then calculating the central point coordinates of the missing or incomplete sign graphs in the pattern;
if the number of the transverse marker patterns is odd, the coordinates of the central points of all the marker patterns in the longitudinal direction at the transverse center position are determined, the straight line where each determined central point coordinate is located is taken as the symmetry axis of the two marker patterns in the transverse direction, and the coordinates of the central points of the marker patterns which are missing or incomplete in the pattern are determined in a symmetry comparison mode.
If the number of the transverse sign patterns is odd, the coordinates of the central points of all the sign patterns in the longitudinal direction at the transverse center position are determined, namely the coordinates of the central points of two sign patterns in each transverse row are determined, then the coordinates of the central points of the two sign patterns belonging to the same transverse row are used as a group to determine the middle point of a line segment between the coordinates of the central points of the two sign patterns belonging to the same transverse row, then the straight lines where all the middle points are located are used as the symmetry axes of the two parts of sign patterns in the transverse direction, and the coordinates of the central points of the sign patterns which are missing or incomplete in the pattern are determined in a symmetrical comparison mode.
It should be noted that the above implementation manners are also applicable to the longitudinal and oblique terms, and the specific implementation manner may refer to the manner of determining the transverse direction, which is not described herein again. In addition, the method for calculating the coordinates of the center point of the missing or incomplete logo is not limited to the above examples, but may be other calculation methods based on the pattern distribution rule, which is not described herein again.
Considering that the pattern may have distortion, deficiency or incompleteness after being photographed, therefore, in order to ensure that the placement direction of the mobile phone can be accurately determined, the number of the special mark patterns may be set to be multiple, or the coordinate origin of the mobile phone screen is directly used as a starting point, and parameters such as the area or radius of the mark patterns are increased or decreased in a diffusion manner, so that although there is no unique or multiple special mark patterns in the pattern, each pattern in the pattern exhibits a change rule that can be visually seen, thereby facilitating distinguishing the position of the coordinate origin of the mobile phone screen in the image, which is not limited herein.
It should be noted that the operation of determining the center point set of the logo on the image pixel coordinates may be performed on a mobile phone, an industrial camera, a testing device, or other devices, which is not limited herein.
Then, any one of the mobile phone, the industrial camera, the test equipment and other equipment corresponds the coordinates of each central point in the image to the coordinates of each central point in the pattern one by one according to the central point set of the mark pattern on the pixel coordinates of the image, the central point set of the mark pattern on the pixel coordinates of the screen and the positions of the origin of coordinates in two coordinate systems, so as to determine the point coordinate conversion relationship, namely the functional relationship, of each mark pattern in the pattern in different coordinate systems, and obtain the position coordinates of the screen corner point of the mobile phone in the image.
Taking the resolution of the mobile phone screen as 1280 × 720 as an example, the coordinates of the 4 corner points of the mobile phone screen may be (0,0), (1280,0), (0,720), and (1280,720), respectively. And then, according to the calculated functional relationship, determining the position coordinates of the 4 corner points in the image, so as to position the positions of the 4 corner points of the mobile phone screen in the image, and then, establishing a connecting line between two adjacent corner points to determine the position of the mobile phone screen in the image. Similarly, the point coordinate conversion relationship obtained through the above calculation can convert the position of any point in the mobile phone screen into the position corresponding to the any point in the image, thereby implementing subsequent detection on the mobile phone screen sensitivity, click operation and the like, which is not limited herein.
The solution to be explained in the embodiments of the present invention is further described below with reference to an example.
In the process of detecting a screen of a terminal such as a mobile phone, the content to be identified is often a specific icon or a specific character, and thus the requirement for the identification accuracy is not high. In addition, for the use process of the user, the user also has difficulty in ensuring that the click operation can accurately fall on the center point of the application icon. However, for precision testing of a mobile phone touch screen, higher precision is required, and compared with an application icon recognition process, the precision testing does not always have a specific icon template for matching, so how to recognize a specific position of a mobile phone screen from an image shot by an apparatus with a shooting function, such as an industrial camera, is very important.
As shown in fig. 6, the sensitivity of the touch screen is tested, that is, the touch test on the screen of the mobile phone is performed, that is, whether the pressing position of the finger is different from the actual sensed point location of the mobile phone is tested, so as to determine whether the touch screen is good or bad. In the embodiment of the present invention, the dotting positions 1 to 9 shown in fig. 6 are distances from border lines of the mobile phone screen 4. It follows that the accuracy of the determination of the screen boundaries is highly demanding.
In addition, the placement position of the mobile phone on the test equipment at each time may be the same or different for the test equipment, and the screen size of the mobile phone placed at each time may be the same or different for the same model of test equipment. For example, if the screen of the mobile phone is non-directional, for the mobile phone with pixels 1080 × 1920, if the user tries to click (500 ) this point and the mobile phone is placed in the forward direction, the point (500 ) can be accurately obtained, and if the mobile phone is placed in the reverse direction, the position that the user should actually click is (1080-500,1920-500), so that a large error occurs between the actual click position and the position that the user tries to click. Therefore, in order to solve the problem of mobile phone screen positioning, the technical scheme provided by the embodiment of the invention can also reduce the process of manually identifying the mobile phone screen through vision. The specific way of distinguishing the placing direction of the mobile phone screen can be realized by referring to the difference of the mark patterns in the drawings, which is not described herein again.
After the mobile phone is placed in the placement area of the test equipment, if the mobile phone is in a bright screen state, the mobile phone is shot by an industrial camera, and an image shown in fig. 3 is obtained. The boundary of the mobile phone screen is the edge 207 of the mobile phone bright screen area in the mobile phone screen 206. As can be seen from fig. 3, the mobile phone screen 206 includes a mobile phone bright screen area and a black edge 208 surrounded by edges 207 of the mobile phone bright screen area, wherein a boundary between the mobile phone bright screen area and the black edge 208 attached to the mobile phone screen 206 is very obvious, 4 boundaries of the mobile phone screen can be fitted in a visual algorithm fitting manner, and an intersection point obtained by intersecting the 4 boundaries is 4 corner points of the mobile phone screen.
In the process of fitting the visual algorithm, the image shown in fig. 3 needs to be subjected to pixel point amplification, wherein, taking the corner point of the mobile phone screen at the lower right of fig. 3 as an example, a gray image shown in fig. 7 can be obtained, and then a black-and-white distinct image is obtained in a straight line fitting manner, so that the boundary contour of the mobile phone screen is determined, and further the position of the corner point is determined.
It should be noted that, the straight line fitting method is mainly to set a threshold, then generate a black-and-white image by referring to the gray-scale image with the set threshold, and further fit the black-and-white image into a straight line by combining irregular edges in the black-and-white image. In addition, different threshold value settings also affect different boundaries of the mobile phone screen to be finally determined. As can be known from fig. 7, although the positions of the corner points can be determined by adopting a line fitting method, it is difficult to fit an accurate line, for example, for a mobile phone screen, it is likely that a rectangular screen region is fitted into an irregular quadrilateral region, thereby affecting the positioning of the corner points.
In the embodiment of the invention, the pattern is generated by combining the pixels of the mobile phone screen in the aforementioned manner, and then the pattern is displayed on the full screen of the mobile phone when the mobile phone is shot. The more the number of the mark patterns in the pattern is, the stronger the robustness of the algorithm for determining the coordinate conversion relation of other arbitrary points, which is obtained by calculating the conversion relation by using the least square method.
In addition, in the process of generating the pattern, the device generating the pattern may be connected to the mobile phone by using a Universal Serial Bus (USB) to establish a communication connection between the mobile phone and the device generating the pattern, so that after the device generating the pattern determines a screen pixel of the mobile phone and generates the pattern, the pattern is pushed to the mobile phone, the mobile phone may store the pattern in a local storage area, and then the mobile phone may automatically display the pattern in a full screen manner to prepare for a test.
In the embodiment of the present invention, when the industrial camera acquires an image, the image sensor of the industrial camera may be parallel to the screen of the mobile phone, and the object distance of the industrial camera is not limited herein, based on the size of the field of view adapting to the size of the mobile phone. Wherein, the image sensor includes but not limited to one of the following items: a Charge Coupled Device (CCD) image sensor, a Complementary Metal-Oxide-Semiconductor (CMOS).
It should be noted that, an image captured by the industrial camera is a gray-scale image, and in order to reduce an error brought to the recognition process by a subsequent gray-scale image, in the embodiment of the present invention, the captured gray-scale image may be processed in advance, for example, the maximum inter-class variance method (OSTU) is adopted, and the image is segmented by combining with a preset or currently automatically generated threshold, so as to obtain a clearly black-and-white image. That is, the gray-scale image shown in fig. 5 captured by the industrial camera is processed to obtain a black-and-white image shown in fig. 8.
Then, a device for image processing, such as a mobile phone, an industrial camera, a testing device, or other devices, may obtain the screen area of the mobile phone by means of contour extraction. Considering that the edge of the mobile phone screen area is likely to have unevenness such as burrs in the image obtained by the OSTU automatic threshold segmentation, in order to ensure that the obtained mobile phone screen area completely includes the mobile phone screen, the maximum contour obtained by extracting the previous contour is processed to find the minimum rectangle circumscribed with the maximum contour, thereby preliminarily determining the range of the mobile phone screen area.
In order to determine the point coordinates of each marker pattern in the image conveniently, in the embodiment of the present invention, the range of the preliminarily determined mobile phone screen area may be used as the range of each marker pattern in the search pattern, so as to extract contour information, that is, contour information of each marker pattern in the image, from the preliminarily determined mobile phone screen area in the image acquired by the industrial camera. Wherein the contour information includes but is not limited to at least one of the following: the area, radius, etc. of the logo. And then, filtering invalid contours of each mark figure which can be displayed in the image according to the contour information, thereby filtering invalid mark figures generated due to illumination and the like.
The missing or incomplete logo pattern may then be optionally recovered to determine the coordinates of the center point of the missing or incomplete logo pattern. It should be noted that the way of recovering the incomplete mark pattern may be based on the definition of the mark pattern itself, for example, if the current mark pattern is a circle, the circle may be triggered from the definition of the circle, and the pattern with completely equal distance from the center point to any point on the periphery is the mark pattern in the embodiment of the present invention, so that the circular outline is completely supplemented according to this principle. The method for recovering the missing sign graphic may determine the center point coordinates of the missing sign graphic according to the relationship between the center point coordinates of the existing sign graphic, and then determine the contour characteristics of the missing sign graphic, such as the radius of a circle, in combination with the contour information of other sign graphics, so as to recover the missing sign graphic. In the embodiment of the present invention, the coordinates of the center points of the missing or incomplete logo patterns can be determined directly according to the relative relationship between the coordinates of the center points of the existing logo patterns, without complementing the incomplete logo patterns or restoring the missing logo patterns.
After the image is processed, each of the mark patterns may be further processed selectively, for example, a Random Sample Consensus (RANSAC) algorithm is performed on the circular contour one by one to perform circle extraction, that is, positions of each point on the circular contour from the center of the circle are detected, points with large deviation on the circular contour are filtered out by comparison, and then least square fitting is performed according to the unfiltered points, so as to obtain mark pattern parameters after fine processing such as radius.
In the embodiment of the present invention, in order to effectively position the positive direction of the mobile phone screen, if the pattern is the image shown in fig. 4, the relative position between the maximum circle and each of the other circles may be determined by combining the parameters of the marker pattern, so as to determine the positive direction of the mobile phone screen, and then the coordinates of each point on the two coordinate systems are in one-to-one correspondence according to the positive direction and the coordinates of the central point of each marker pattern in the mobile phone screen and the image acquired by the industrial camera, and then the conversion relationship between the two sets of point coordinates may be calculated by the least square method, that is, the rotation-translation scaling conversion matrix is calculated.
The above mentioned algorithm such as the least square method is the prior art, and the implementation conditions, the calculation method, and the like in the prior art may be referred to, which is not described herein again.
After the conversion relation is accurately determined, under the condition that coordinates of the screen corner points of the mobile phone are known, the positions of the screen corner points of the mobile phone in the image can be calculated by combining the conversion relation, and the positive direction of the screen of the mobile phone can be determined. And then, fitting 4 boundary lines of the mobile phone screen in the image through the positions of the 4 corner points so as to determine the area of the mobile phone screen. In addition, according to the conversion relation, the position of any point on the mobile phone screen in the image after conversion can be determined.
In order to further ensure the reliability of the conversion relationship, in the embodiment of the present invention, a pattern may be generated again according to the resolution of the mobile phone screen, and the above implementation manner may be repeatedly executed, so as to obtain another conversion relationship, and then the two conversion relationships are compared and adjusted, so as to obtain a more accurate conversion manner, that is, the conversion relationship is finally used to determine the corresponding relationship between the two coordinate systems. It should be noted that, as the number of times of re-determining the conversion relationship increases, the finally obtained conversion relationship is more accurate. Of course, it is also possible to detect one or more positions on the screen of the mobile phone by using the current transformation relationship, so as to determine whether the current obtained transformation relationship is accurate, and to perform fine adjustment and the like on the transformation relationship in combination with the actual situation, which is not described herein again.
Therefore, the embodiment of the invention can be effectively applied to automatic positioning of the mobile phone screen, so that the dotting and drawing of the appointed position in the mobile phone screen, accurate acquisition of the icon position and the like can be more accurately positioned. In addition, the pattern is generated based on the pixels of the mobile phone screen, so that the implementation mode adopted by the embodiment of the invention can be effectively suitable for detection of electronic equipment with a display screen, such as mobile phones and tablet computers of various types. In addition, in the implementation manner provided by the embodiment of the present invention, the placing direction of the device to be tested, such as a mobile phone, on the testing device is not limited, which means that the probability that the detection process is affected by the environment is low, thereby improving the implementation accuracy of the whole detection process.
Taking a mobile phone as an example, under a normal condition, an angular point of a mobile phone screen is probably not exactly the origin of a coordinate system where the mobile phone screen is located due to a bonding process error of the mobile phone screen, but an error caused by the above problem can be effectively avoided by adopting the implementation mode of point coordinate conversion provided by the embodiment of the invention. In addition, because the problems of light leakage of the mobile phone screen, quality of the collected image and the like can affect the test process more or less, the implementation scheme provided by the embodiment of the invention eliminates the influence of the mobile phone screen boundary processing technology, and the more the number of the mark patterns in the pattern is, the more the information for determining the conversion relation is, the more the determined conversion relation is accurate, so that the precision of fitting the point coordinate of the position corresponding to the coordinate system where the mobile phone screen is located in the image is improved.
For example, after obtaining the conversion relationship, the mobile phone may switch the interface presented on the screen of the mobile phone, for example, close the current full-screen displayed pattern, and present the main interface of the mobile phone, as shown in fig. 9.
If the operation of clicking the calendar icon by the user is tried to be simulated at present, so that the sensitivity of the area where the calendar icon is located in the mobile phone screen is detected, the test equipment can acquire the position which is actually clicked when the user clicks the calendar icon in advance, namely the display position of the calendar icon on the mobile phone screen. In the embodiment of the present invention, referring to the implementation manner adopted for positioning the mobile phone screen, the two-dimensional coordinate system based on the mobile phone screen is established by using the corner point on the upper left of the mobile phone screen shown in fig. 9 as the origin of the two-dimensional coordinate system, where the established coordinate system is the same as the two-dimensional coordinate system established on the mobile phone screen when the mobile phone screen is positioned, and the resolution of the mobile phone screen is still 1280 × 720. And obtaining coordinates of the center point of the calendar icon, namely the coordinates of the point shown in the figure 10 according to the display position of the center point of the calendar icon in the mobile phone screen and the established two-dimensional coordinate system (288,213). That is, in the actual operation, if the user tries to click on the calendar icon, it is necessary to generate a click operation at a point corresponding to (288,213) in the two-dimensional coordinate system. That is, the point coordinates (288,213) are used as input to the test equipment.
As shown in fig. 11, a black-and-white image obtained by processing a grayscale image of a cellular phone captured by an industrial camera is calculated (288,213) based on the conversion relationship obtained previously, and the point coordinate corresponding to the point coordinate in the two-dimensional coordinate system shown in fig. 11 is calculated (1620,532). That is, the testing device can determine the position of the mechanical arm to perform the clicking operation in the image acquired by the industrial camera according to the input point coordinates (288,213) and the obtained conversion relation. For the robot arm, after the test equipment completes the calculation of the conversion relationship, the robot arm is controlled to perform a click operation at a point with the coordinate (1620,532).
It should be noted that, because the conversion relationship has considered that the mobile phone placing direction changes when determining, the conversion of the mobile phone placing direction has been completed in the obtained conversion relationship, that is, no matter what direction the mobile phone takes to place, once the conversion relationship is determined, the mobile phone placing position does not change, and the relative position between the industrial camera and the mobile phone does not change when the industrial camera collects an image, the existing conversion relationship can be directly applied in the process of finding one point in the image according to one point on the mobile phone screen without performing other operations. In addition, in order to conveniently execute the click operation on each position on the screen of the mobile phone and reduce data interaction between the mobile phone and the testing equipment, in the embodiment of the invention, the mobile phone can send the coordinates of each icon on the screen of the mobile phone, which are displayed in an interface to be tested, to the testing equipment in advance, when the equipment to be tested needs the coordinates of a certain icon on the screen of the mobile phone, the coordinates of the certain icon, which are sent by the mobile phone and are received before, can be directly used as input, the position of the certain icon in the image is obtained through a coordinate conversion mode, and the coordinates for expressing the position of the certain icon in the image are transmitted to the mechanical arm, so that the mechanical arm can realize the click operation on the corresponding coordinates in the image.
Therefore, the conversion relation of the point coordinates between the mobile phone screen and the image of the mobile phone shot by the industrial camera can be effectively determined by utilizing each mark graph in the pattern, the position of the corner point of the mobile phone screen can be positioned in the image by virtue of the conversion relation, the area of the mobile phone screen can be further positioned, and any one point or multiple points in the mobile phone screen can be positioned in the image according to the conversion relation.
The terminal may be provided with a positioning device of the terminal screen, and the positioning device of the terminal screen includes a hardware structure and/or a software module corresponding to each function for implementing the functions. Those of skill in the art will readily appreciate that the present invention can be implemented in hardware or a combination of hardware and computer software, with the exemplary elements and algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
According to the embodiment of the present invention, the functional modules of the positioning device of the terminal screen may be divided according to the above method, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, the division of the modules in the embodiment of the present invention is schematic, and is only a logic function division, and there may be another division manner in actual implementation.
Fig. 12 is a schematic diagram of a possible structure of the positioning device of the terminal screen according to the above embodiment. The positioning device 30 of the terminal screen comprises: an acquisition module 31 and a determination module 32. The acquisition module 31 is configured to support the positioning device 30 of the terminal screen to complete acquisition of the first image; the determining module 32 is configured to support the positioning device 30 of the terminal screen to determine a transformation relationship between the first coordinate and the second coordinate, and to implement positioning of the area of the terminal screen in the first image according to the transformation relationship. In the embodiment of the present invention, the positioning device 30 of the terminal screen may further include a storage module 33 for storing program codes and data of the terminal; a communication module 34, configured to support data interaction between the positioning apparatus 30 of the terminal screen and a terminal where the positioning apparatus of the terminal screen is located, for example, various modules in a testing device, and/or support communication between the testing device and other devices such as a mobile phone and an industrial camera; and/or other processes for the techniques described herein.
The determination module 32 may be implemented as a Processor or a controller, and may be, for example, a Central Processing Unit (CPU), a general-purpose Processor, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor may also be a combination of computing functions, e.g., comprising one or more microprocessors, DSPs, and microprocessors, among others. The communication module 34 may be implemented as a transceiver, transceiving circuitry, or a communication interface, etc. The storage module 33 may be implemented as a memory.
If the determining module 32 is implemented as a processor, the communication module 34 is implemented as a transceiver, and the storage module 33 is implemented as a memory, as shown in fig. 13, the terminal 40 includes: a processor 41, a transceiver 42, a memory 43, and a bus 44. Wherein the processor 41, the transceiver 42 and the memory 43 are connected to each other by a bus 44; the bus 44 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 13, but this is not intended to represent only one bus or type of bus.
The steps of a method or algorithm described in connection with the disclosure herein may be embodied in hardware or in software instructions executed by a processor. The software instructions may be comprised of corresponding software modules that may be stored in Random Access Memory (RAM), flash Memory, Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), registers, a hard disk, a removable disk, a Compact Disc Read-Only Memory (CD-ROM), or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in the same apparatus or may be separate components in different apparatuses.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the embodiments of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention.

Claims (25)

  1. A method for positioning a terminal screen is characterized by comprising the following steps:
    acquiring a first image, wherein the first image is a shot image of a terminal, the terminal is provided with a screen, and the screen displays patterns which comprise regularly distributed sign graphs;
    determining a conversion relationship between a first coordinate and a second coordinate, wherein the first coordinate is used for representing the position of the sign graph on the screen, and the second coordinate is used for representing the position of the sign graph in the first image;
    and determining the area of the screen in the first image according to the conversion relation.
  2. The method of claim 1, wherein said determining the region of the screen in the first image according to the transformation relationship comprises:
    determining a fourth coordinate according to the conversion relation and a third coordinate, wherein the third coordinate is used for representing the position of each corner point on the screen, and the fourth coordinate is used for representing the position of each corner point in the first image;
    and in the first image, determining a region formed by taking the position of each corner point corresponding to the fourth coordinate as an end point as a region of the screen in the first image.
  3. A method according to claim 1 or 2, wherein the pattern comprises at least three logo images.
  4. The method according to any one of claims 1 to 3, wherein the regular distribution comprises a matrix distribution of the center points of the logo patterns in rows and columns at equal intervals.
  5. The method according to any one of claims 1 to 4, wherein the logo patterns comprise a regular logo pattern and a special logo pattern, the regular logo pattern is identical, the special logo pattern is different from the regular logo pattern, and the special logo pattern is used for distinguishing the placement direction of the terminal when shooting.
  6. A method as claimed in any one of claims 1 to 5, wherein the logo graphic is black and the area of the design other than the logo graphic is white.
  7. The method according to any one of claims 1 to 6, wherein the position of the center point of the logo pattern is not changed after the logo pattern is deformed due to the change of the photographing angle.
  8. The method of claim 7, wherein the logo graphic is circular or oval.
  9. The method according to any one of claims 1 to 8, wherein the resolution of the pattern is the same as the resolution of the screen.
  10. A method for positioning a terminal screen is characterized by comprising the following steps:
    the terminal displays a pattern through a screen, wherein the pattern comprises mark graphs which are regularly distributed, and the first coordinate is used for representing the positions of the mark graphs on the screen;
    the industrial camera acquires a first image and determines a second coordinate, the second coordinate is used for representing the position of the mark graph in the first image, and the first image is a shot image of the terminal;
    the test equipment acquires the first coordinate and the second coordinate and determines a conversion relation between the first coordinate and the second coordinate;
    and the test equipment determines the area of the screen in the first image according to the conversion relation.
  11. The method of claim 10, wherein after the test device determines the translation relationship between the first coordinate and the second coordinate, the method further comprises:
    the test equipment acquires a fifth coordinate, and the fifth coordinate is used for representing a position to be tested on the screen;
    the testing equipment converts the fifth coordinate according to the conversion relation to obtain a sixth coordinate, and sends the sixth coordinate to the mechanical arm, wherein the sixth coordinate is used for representing the position of the position to be tested in the first image;
    the robotic arm performs an operation at the location of the sixth coordinate in the first image in response to a message sent by the testing device.
  12. A positioning device for a terminal screen, the device comprising:
    the terminal comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a first image, the first image is a shot terminal image, the terminal is provided with a screen, and the screen displays patterns which comprise regularly distributed sign patterns;
    a determining module, configured to determine a conversion relationship between a first coordinate and a second coordinate, where the first coordinate is used to represent a position of the logo image displayed by the display module on the screen, and the second coordinate is used to represent a position of the logo image in the first image acquired by the acquiring module;
    the determining module is further configured to determine an area of the screen in the first image according to the conversion relationship.
  13. The apparatus of claim 12, wherein the determining module is further configured to:
    determining a fourth coordinate according to the conversion relation and a third coordinate, wherein the third coordinate is used for representing the position of each corner point on the screen, and the fourth coordinate is used for representing the position of each corner point in the first image;
    and in the first image, determining a region formed by taking the position of each corner point corresponding to the fourth coordinate as an end point as a region of the screen in the first image.
  14. A device according to claim 12 or 13, wherein the pattern comprises at least three logo images.
  15. The apparatus according to any one of claims 12 to 14, wherein the regular distribution comprises a matrix distribution of the center points of the logo patterns in rows and columns at equal intervals.
  16. The apparatus according to any one of claims 12 to 15, wherein the logo patterns include a regular logo pattern and a special logo pattern, the regular logo pattern is identical, the special logo pattern is different from the regular logo pattern, and the special logo pattern is used for distinguishing a placement direction of the terminal when photographing.
  17. A device according to any one of claims 12 to 16, wherein the logo is black and the area of the design other than the logo is white.
  18. The apparatus according to any one of claims 12 to 17, wherein the position of the center point of the marker pattern is not changed after the marker pattern is deformed due to the change of the photographing angle.
  19. The apparatus of claim 18, wherein the logo graphic is circular or oval.
  20. The apparatus of any of claims 12 to 19, wherein the resolution of the pattern is the same as the resolution of the screen.
  21. A terminal comprising a display, a memory, one or more processors, a plurality of applications, and one or more programs; wherein the one or more programs are stored in the memory; one or more processors, when executing the one or more programs, cause the terminal to implement the method of any of claims 1-9.
  22. A readable storage medium, having stored therein instructions, which, when run on a terminal, cause the terminal to perform the method of any of the preceding claims 1 to 9.
  23. A computer program product, characterized in that it comprises a software code for performing the method of any one of the preceding claims 1 to 9.
  24. A system for positioning a terminal screen, the system comprising:
    the terminal is used for displaying a pattern through a screen and determining a first coordinate, wherein the pattern comprises mark graphs which are regularly distributed, and the first coordinate is used for representing the positions of the mark graphs on the screen;
    the industrial camera is used for acquiring a first image and determining a second coordinate, the second coordinate is used for representing the position of the mark graph in the first image, and the first image is a shot image of the terminal;
    the test equipment is used for acquiring the first coordinate and the second coordinate and determining a conversion relation between the first coordinate and the second coordinate;
    the test equipment is further used for determining the area of the screen in the first image according to the conversion relation.
  25. The system of claim 24, wherein after the testing device determines the transformation relationship between the first coordinate and the second coordinate, the testing device is further configured to obtain a fifth coordinate, the fifth coordinate being used to represent a position to be tested on the screen;
    the testing equipment is further used for converting the fifth coordinate according to the conversion relation to obtain a sixth coordinate, and sending the sixth coordinate to the mechanical arm, wherein the sixth coordinate is used for representing the position of the position to be tested in the first image;
    the mechanical arm is used for responding to the message sent by the test equipment and executing operation at the position of the sixth coordinate in the first image.
CN201780089459.5A 2017-11-27 2017-11-27 Terminal screen is fixed a position to transform relation based on mark figure point coordinate in pattern Active CN110506252B (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/113216 WO2019100407A1 (en) 2017-11-27 2017-11-27 Positioning of terminal screen based on transformation relation of coordinates of marking graphic points in pattern

Publications (2)

Publication Number Publication Date
CN110506252A true CN110506252A (en) 2019-11-26
CN110506252B CN110506252B (en) 2021-01-29

Family

ID=66631781

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201780089459.5A Active CN110506252B (en) 2017-11-27 2017-11-27 Terminal screen is fixed a position to transform relation based on mark figure point coordinate in pattern

Country Status (2)

Country Link
CN (1) CN110506252B (en)
WO (1) WO2019100407A1 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111290660A (en) * 2020-01-21 2020-06-16 上海悦易网络信息技术有限公司 Method and equipment for converting screen coordinates into touch pen coordinates
CN111311687A (en) * 2020-01-21 2020-06-19 上海悦易网络信息技术有限公司 Method and equipment for detecting spatial position of mobile phone screen pixel point
CN111325717A (en) * 2020-01-21 2020-06-23 上海悦易网络信息技术有限公司 Mobile phone defect position identification method and equipment
CN113537218A (en) * 2021-07-27 2021-10-22 京东数科海益信息科技有限公司 Image recognition method and device
CN114393576A (en) * 2021-12-27 2022-04-26 江苏明月智能科技有限公司 Four-axis mechanical arm clicking and position calibrating method and system based on artificial intelligence

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1799252A (en) * 2003-06-02 2006-07-05 卡西欧计算机株式会社 Captured image projection apparatus and captured image correction method
JP2007053784A (en) * 2006-09-22 2007-03-01 Seiko Epson Corp Projector type image display apparatus and flat body to be projected
WO2009046468A2 (en) * 2007-10-12 2009-04-16 At & S Austria Technologie & Systemtechnik Aktiengesellschaft Method and device for positioning and aligning an object relatively to or with a substrate and use thereof
CN102231093A (en) * 2011-06-14 2011-11-02 伍斌 Screen locating control method and device
CN102298467A (en) * 2010-06-24 2011-12-28 北京威亚视讯科技有限公司 Automatic calibration method and system for display screen
CN102968794A (en) * 2012-11-22 2013-03-13 上海交通大学 In-situ calibration method for structure parameters of large-visual-field binocular stereo vision system by small two-dimensional targets
CN104766315A (en) * 2015-03-30 2015-07-08 联想(北京)有限公司 Method for calibrating relative position relation between image collection device and display screen and equipment
CN105427284A (en) * 2015-11-06 2016-03-23 西北工业大学 Fixed target marking method based on airborne android platform
CN106293270A (en) * 2016-08-05 2017-01-04 南京仁光电子科技有限公司 A kind of scaling method of giant-screen touch-control system

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104090689B (en) * 2014-06-27 2019-11-05 努比亚技术有限公司 A kind of method and system of mobile terminal and its interactive projection
CN106257388A (en) * 2016-07-25 2016-12-28 北京仁光科技有限公司 A kind of localization method of optical touch screen

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1799252A (en) * 2003-06-02 2006-07-05 卡西欧计算机株式会社 Captured image projection apparatus and captured image correction method
JP2007053784A (en) * 2006-09-22 2007-03-01 Seiko Epson Corp Projector type image display apparatus and flat body to be projected
WO2009046468A2 (en) * 2007-10-12 2009-04-16 At & S Austria Technologie & Systemtechnik Aktiengesellschaft Method and device for positioning and aligning an object relatively to or with a substrate and use thereof
CN102298467A (en) * 2010-06-24 2011-12-28 北京威亚视讯科技有限公司 Automatic calibration method and system for display screen
CN102231093A (en) * 2011-06-14 2011-11-02 伍斌 Screen locating control method and device
CN102968794A (en) * 2012-11-22 2013-03-13 上海交通大学 In-situ calibration method for structure parameters of large-visual-field binocular stereo vision system by small two-dimensional targets
CN104766315A (en) * 2015-03-30 2015-07-08 联想(北京)有限公司 Method for calibrating relative position relation between image collection device and display screen and equipment
CN105427284A (en) * 2015-11-06 2016-03-23 西北工业大学 Fixed target marking method based on airborne android platform
CN106293270A (en) * 2016-08-05 2017-01-04 南京仁光电子科技有限公司 A kind of scaling method of giant-screen touch-control system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李冬 等: "多投影机非平面自动几何校正", 《计算机工程与应用》 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111290660A (en) * 2020-01-21 2020-06-16 上海悦易网络信息技术有限公司 Method and equipment for converting screen coordinates into touch pen coordinates
CN111311687A (en) * 2020-01-21 2020-06-19 上海悦易网络信息技术有限公司 Method and equipment for detecting spatial position of mobile phone screen pixel point
CN111325717A (en) * 2020-01-21 2020-06-23 上海悦易网络信息技术有限公司 Mobile phone defect position identification method and equipment
CN111311687B (en) * 2020-01-21 2022-12-02 上海万物新生环保科技集团有限公司 Method and equipment for detecting spatial position of mobile phone screen pixel point
CN111325717B (en) * 2020-01-21 2023-08-29 上海万物新生环保科技集团有限公司 Mobile phone defect position identification method and equipment
CN113537218A (en) * 2021-07-27 2021-10-22 京东数科海益信息科技有限公司 Image recognition method and device
CN113537218B (en) * 2021-07-27 2024-02-06 京东科技信息技术有限公司 Image recognition method and device
CN114393576A (en) * 2021-12-27 2022-04-26 江苏明月智能科技有限公司 Four-axis mechanical arm clicking and position calibrating method and system based on artificial intelligence

Also Published As

Publication number Publication date
WO2019100407A1 (en) 2019-05-31
CN110506252B (en) 2021-01-29

Similar Documents

Publication Publication Date Title
CN110506252B (en) Terminal screen is fixed a position to transform relation based on mark figure point coordinate in pattern
US10755063B2 (en) Method and apparatus for detecting two-dimensional barcode
US10796133B2 (en) Image processing method and apparatus, and electronic device
CN109684980B (en) Automatic scoring method and device
US10915750B2 (en) Method and device for searching stripe set
CN107613202B (en) Shooting method and mobile terminal
CN110100251A (en) For handling the equipment, method and graphic user interface of document
CN107977659A (en) A kind of character recognition method, device and electronic equipment
CN112040223A (en) Image processing method, terminal device and storage medium
CN110795019B (en) Key recognition method and device for soft keyboard and storage medium
CN109558000B (en) Man-machine interaction method and electronic equipment
JP2011221604A (en) Handwriting data management system, handwriting data management program, and handwriting data management method
CN110442521B (en) Control unit detection method and device
CN113052919A (en) Calibration method and device of visual sensor, electronic equipment and storage medium
CN110463177A (en) The bearing calibration of file and picture and device
CN110738185B (en) Form object identification method, form object identification device and storage medium
CN104766354A (en) Method for augmented reality drawing and mobile terminal
CN106569716B (en) Single-hand control method and control system
WO2020078344A1 (en) Method for identifying special-shaped touch, terminal and storage medium
JP2011118466A (en) Difference noise replacement device, difference noise replacement method, difference noise replacement program, computer readable recording medium, and electronic equipment with difference noise replacement device
CN110796096B (en) Training method, device, equipment and medium for gesture recognition model
CN107622235B (en) Fingerprint unlocking method and related product
CN108848265B (en) Screen brightness adjusting method and device for local area, terminal and storage medium
CN111062258A (en) Text region identification method and device, terminal equipment and readable storage medium
CN108021313B (en) Picture browsing method and terminal

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant