WO2019080229A1

WO2019080229A1 - Chess piece positioning method and system based on machine vision, storage medium, and robot

Info

Publication number: WO2019080229A1
Application number: PCT/CN2017/112210
Authority: WO
Inventors: 张帆
Original assignee: 南京阿凡达机器人科技有限公司
Priority date: 2017-10-25
Filing date: 2017-11-21
Publication date: 2019-05-02
Also published as: CN107766855A; CN107766855B

Abstract

A chess piece positioning method based on machine vision, used for positioning chess pieces on a chess board provided with flat markers, the method comprising: acquiring a video stream by means of a camera and collecting video image frames from the video stream (S100); performing image processing on the video image frames to identify flat markers (S200); and, on the basis of the flat markers and a pre-acquired internal reference matrix of the camera, calculating the position of a chess piece on the chess board relative to the markers, and thereby locating the position of the chess piece (S300). Also provided are a chess piece positioning system based on machine vision, a storage medium, and a robot. The system using the chess piece positioning method can be implemented without the need for a special circuit in the chess board, and there is also no need for data communication between the chessboard and the robot, so that arrangement is more convenient. The arrangement of the flat markers enables the robot to accurately identify the flat markers in a complex scene without suffering interference from the complex scene.

Description

Chess positioning method, system, storage medium and robot based on machine vision

The present application claims priority to Chinese Patent Application No. 201711006222.4, entitled "Machine Vision-Based Chess Positioning Method, System, Storage Medium, and Robot", the entire contents of which are incorporated herein by reference. .

Technical field

The invention relates to the field of robots, in particular to a machine vision based chess piece positioning method, system, storage medium and robot.

Background technique

Today, with the increasing popularity of robotics, people want to interact with robots like real people and feel the power of artificial intelligence. Playing chess now becomes an important interaction with robots. Although it is possible to play chess on the interactive screen of some robots, this method is not attractive, and the user does not feel that he is playing chess with the robot. Therefore, in order to create a better human-computer interaction experience, it is necessary to let the robot understand the real chess board and play chess in the same way as real people. This can enable the user to get a better sense of bringing in, increase the anthropomorphic characteristics of the robot, and enable the user to have a better interactive experience. The robot plays chess like a human and needs to recognize the exact position of the pieces on the board.

The invention patents of the application Nos. 201310554765.5 and 201621069461.5, 201310159264.7 all disclose a physical chessboard man-machine game system. This type of invention uses a special physical chessboard to sense the position of the chess piece relative to the board through the corresponding sensor, and then sends the piece position information to the robot device. The invention is capable of accurately obtaining the exact position of the piece on a particular board.

However, the scheme requires a specific chessboard hardware circuit to be implemented, and the chessboard and the robot need to perform data communication, which is complicated to implement, and the site layout and debugging installation are relatively cumbersome.

Application No. 201610967421.0 discloses an image-based chess piece positioning and recognition method capable of locating the position of a chess piece relative to a chessboard.

However, the invention is to binarize the image by the threshold method and then detect the boundary of the board. Position the pieces. When the image has a complex background, this boundary extraction method is very unstable, and it is difficult to correctly segment the true checkerboard boundary. Some objects in the image with rectangular boundaries, such as the bench surface, are easily mistaken for the board.

Summary of the invention

In view of the above deficiencies of the prior art, the present invention provides a machine vision based chess piece positioning method and system, a storage medium and a robot for accurately locating a specific position of a chess piece on a chessboard.

In a first aspect, the invention discloses a machine vision-based chess piece positioning method, which is applied to positioning a chess piece on a chessboard provided with a planar identifier, the chess piece positioning method comprising:

S100 acquires a video stream by using a camera, and collects a video image frame from the video stream;

S200 performs image processing on the video image frame to identify the plane identifier;

S300 calculates a position of the piece on the board relative to the identifier according to the plane identifier and the internal parameter matrix of the camera acquired in advance, thereby positioning the position of the piece.

The solution is to further position the piece by setting a plane marker on the board and then visually recognizing the plane marker by machine vision. This solution does not require the placement of circuitry at the bottom of the board, so no complicated arrangements are required. And because of the establishment of the planar marker, the position of the pawn is accurately identified even in a complicated background.

Preferably, before the capturing of the video stream by the camera in the step S100, and before acquiring the video image frame from the video stream, the method further includes:

S010 calibrates the camera to obtain an internal reference matrix of the camera.

Before you can position the pieces, one of the preparations you need to do is to calibrate the camera. The camera calibration calculates the internal parameters of the camera, that is, determines the geometric model of the camera imaging, so that the images captured by the ray are used later to calculate the three-dimensional position.

Preferably, the step S200 performs image processing on the video image frame, and the identifying the planar identifier is specifically:

S210 performs binary segmentation on the video image frame, and extracts a contour from the segmented binarized image;

S220: selecting, according to a common feature of the pre-stored contour identifiers of the plane identifiers, an outline contour having the common features as an alternative contour from the extracted contour contours;

S230 acquires a front view of the candidate contour;

S240: When the front view of the candidate contour is consistent with the pre-stored planar identifier template, the image corresponding to the candidate contour is identified as an image of the planar identifier.

In this solution, the video image frame is processed, the video image frame is binary-divided, and the contour is extracted from the segmented image. The binarization processing method is adopted to facilitate the extraction of information in the image, and the binary image can increase the recognition efficiency when performing computer recognition. In addition, binarization can easily and quickly process image features, simplify post processing, and increase processing speed. The image is subdivided on the binarized image, and the different regions of the image have special meanings. These regions are mutually disjoint, and each region satisfies the consistency of a particular region. Then, the contour image is extracted and recognized, which can reduce the difficulty of recognition and speed up the recognition efficiency.

Preferably, the planar identifier is a square, and the common feature of the outline corresponding to the planar identifier pre-stored in the above step S220 is a quadrilateral.

If the set markers are different, then the common features of the corresponding outlines will be different. When selecting a flat marker, we should try to select a marker that is easy to identify and has obvious feature points. For example, a square with four vertices and four sides are equal in length, which is convenient for later calculations. When the square identifier is selected as the planar identifier, the corresponding contour common feature is a quadrilateral. After the square plane identifier is selected, the step S230 obtains a front view of the candidate contour, which is specifically:

S231 reading pixel coordinates of four vertices of the candidate contour;

S232 defines a pixel coordinate of the four vertices after the candidate profile is subjected to projective transformation into a front view as pixel coordinates of four pre-existing front views of the planar marker;

S233, the pixel coordinates of the vertices of the four sets of the candidate contours obtained by the above-mentioned acquired candidate contours and the pixel coordinates of the vertices after the definition of the candidate contour are converted into the following equations, and the projection transformation matrix H is obtained:

Where X is the homogeneous pixel coordinate of the candidate silhouette vertex, and the non-homogeneous pixel coordinates are

X' is the homogeneous coordinate of the vertex after the projective transformation, and its non-homogeneous pixel coordinates are

S234 performs projective transformation on all pixels of the binarized image corresponding to the candidate contour according to the projective transformation matrix H, and obtains a front view corresponding to the region included in the candidate contour.

The solution is based on the selection of a square planar identifier. The contour of the square planar identifier must be a quadrilateral, and the quadrilateral has four vertices. Therefore, the pixel coordinates of the four vertices and the defined quadrilateral front view can be utilized. The pixel coordinates are calculated to obtain the projective transformation matrix H, so that all pixels in the quadrilateral contour are subjected to projective transformation to obtain a corresponding front view for subsequent comparison.

Preferably, the step S300 calculates the position of the piece on the board relative to the identifier according to the plane identifier and the internal parameter matrix of the camera acquired in advance, so that the position of the piece is specifically:

S310: calculating, according to the plane identifier and the pre-acquired internal reference matrix of the camera, a homography matrix of a current field of view of the camera;

S320: according to the known coordinates of the preset reference point on the checkerboard in the plane identifier coordinate system and the rank position on the checkerboard, combined with the grid length feature of the checkerboard, acquiring the four corner points of the checkerboard in the plane The coordinates in the marker coordinate system;

S330 identifies a piece in the video image frame, reads pixel coordinates of the piece, and acquires coordinates of the piece in the plane identifier coordinate system according to the homography matrix;

S340 obtains the position of the chess piece on the chessboard according to the coordinates of the chess piece and the four corner points of the chessboard in the plane identifier coordinate system, combined with the grid length feature of the chessboard.

After the planar marker is identified, the chess piece can be positioned according to the planar identifier and the internal reference matrix of the camera. The most important thing is to obtain the current field of view of the camera according to the planar identifier and the internal reference matrix of the camera. Sexual matrix. After obtaining the homography matrix, the position of the chessboard and the chess piece in the plane coordinate coordinate system is determined. Finally, the position of the chess piece on the chessboard can be calculated by combining the length characteristics of the checkerboard mesh. Wherein, the chess piece in the video image frame is identified in step S330 in the solution, the chess piece The recognition can be performed according to the feature of the pre-extracted chess piece, which is similar to the previous planar marker recognition, and the image processing frame is subjected to image processing and recognition, which will not be described here. The scheme is simple in calculation and strong in implementation.

Preferably, the step S310 calculates the homography matrix of the current field of view of the camera according to the plane identifier and the internal parameter matrix of the camera obtained in advance:

S311 selects the plane identifier coordinate system as a world coordinate system;

S312 reading pixel coordinates of four vertices of the planar identifier;

S313 substituting the coordinates of the four vertices of the known planar identifier in the coordinates of the world coordinate system, the pixel coordinates of the four vertices of the planar identifier, and the internal parameters of the camera obtained in advance into the following formula:

Calculate the homography matrix H=M[r1,r2,r3,t] of the current field of view;

Where (x, y, 1) represents the homogeneous coordinate of the pixel coordinates of any vertex in the plane image in the image coordinate system of the camera, and (X, Y, Z, 1) indicates that the vertex is in the world coordinate system The homogeneous coordinates in the middle are reduced to (X, Y, 0, 1), s is the introduced arbitrary scale proportional parameter, M is the camera internal reference matrix, and r1, r2, r3 respectively represent the planar marker coordinate system relative to the camera. Three column vectors in the rotation matrix of the coordinate system, t is the translation vector.

This scheme specifies the homography matrix of the current field of view of the camera. This homography matrix is the rotation matrix R and the translation vector t of the planar marker coordinate system with respect to the camera coordinate system. The following formula exists because the pixel coordinates of the vertex of the planar identifier on the image and the coordinates of the vertex in the corresponding world coordinate system:

The above passes the identifier of a vertex pixel coordinate and its corresponding coordinate in the coordinate system of the marker. Obtain an equation, the identifier corresponds to four vertices, then you can write four corresponding equations correspondingly, and then solve the r1, r2, r3, t in the homography matrix through the equations.

Preferably, after the step S330 and before the step S340, the method further includes:

S335 determines, according to the coordinates of the pieces and the four corner points of the board in the plane identifier coordinate system, that the pieces are in a quadrilateral area composed of four corner points of the board.

In this solution, after the previous step S320, the coordinates of the four corner points of the checkerboard in the plane identifier coordinate system can be determined, and the quadrilateral area composed of the four corner points is the checkerboard area, which can be determined through step S330. The coordinates of the identified pieces in the plane coordinate system are determined. Therefore, step S335 is to determine that the identified pieces are on the board. Because only the pieces on the board need to be further positioned on the board, if they are not on the board, they can choose not to go to the tube, which can reduce the workload of the positioning of the pieces, and also facilitate the calculation of the position of the subsequent pieces on the board.

Preferably, after the step S320 and before the step S330, the method further includes:

S325: obtaining pixel coordinates of four corner points of the checkboard according to coordinates of the four corner points of the checkerboard in the plane identifier coordinate system and a homography matrix of the current field of view of the camera;

S326: selecting, according to pixel coordinates of the four corner points of the checkerboard, a quadrilateral region composed of four corner points of the checkerboard as a target area in the video image frame;

The step S330 identifies the pieces in the video image frame, and reads the pixel coordinates of the piece, and acquires the coordinates of the piece in the plane identifier coordinate system according to the homography matrix:

S331 identifies a piece in a target area in the video image frame, and reads pixel coordinates of the piece;

S332: Acquire coordinates of the chess piece in the plane identifier coordinate system according to the pixel coordinates of the chess piece and the homography matrix.

In this solution, by obtaining the pixel coordinates of the four corner points of the board, the position area of the board in the video image frame is determined, and the position area composed of the four corner points is used as the target area, and in the video image frame. Among them, only the pieces in the target area are the pieces on the board that need to be positioned, and the images in the target area, we can not identify them, that is, only identify the pieces in the target area, which greatly reduces the number of pieces. The amount of calculation speeds up the positioning of the pieces.

In a second aspect, the invention also discloses a machine vision-based chess piece positioning system, which is applied to positioning a chess piece on a chessboard provided with a plane identifier, the machine vision-based chess piece positioning system comprising: a camera acquisition module, For acquiring a video stream, and collecting a video image frame from the video stream; and an image recognition module, configured to perform image processing on the video image frame collected by the camera acquisition module, thereby identifying the image in the video image frame a plane identifier; a positioning module, configured to calculate a position of the piece on the board relative to the identifier according to the plane identifier and an internal parameter matrix of the camera acquired in advance, thereby locating the position of the piece.

Preferably, the machine vision-based piece positioning system further comprises: a calibration module, configured to calibrate the camera to obtain an internal parameter matrix of the camera.

Preferably, the image recognition module includes: a binary segmentation sub-module for performing binary segmentation on the video image frame; and a shape extraction sub-module for extracting a contour from the segmented binarized image; storing a sub-module, configured to store a common feature of the outer shape corresponding to the planar marker, the planar identifier template, and a determining processing sub-module, configured to correspond to the planar identifier stored in the storage sub-module a common feature of the outline of the profile, selecting a profile having the common feature as an alternative profile from the profile extracted by the profile extraction sub-module; a front view acquisition sub-module for obtaining a front view of the candidate profile An identification sub-module, configured to identify an image corresponding to the candidate contour as an image of the planar identifier when a front view of the candidate contour is consistent with the planar identifier template prestored by the storage sub-module .

Preferably, the planar identifier is a square, and the common feature of the outline corresponding to the planar identifier stored by the storage sub-module is a quadrilateral; the front view obtaining sub-module includes: a reading unit for reading Taking pixel coordinates of four vertices of the candidate contour; defining a unit for defining pixel coordinates of the four vertices after the candidate contour is subjected to projective transformation into a front view as pre-stored a pixel coordinate of four vertices of a front view of the face marker; a calculation unit for calculating four vertex pixel coordinates read by the reading unit and four vertex pixel coordinates defined by the defining unit, by using the following equation , to obtain the projective transformation matrix H,

a transforming unit, configured to perform a projective transformation on all pixels of the binarized image corresponding to the candidate contour according to the projective transformation matrix H obtained by the calculating unit, to obtain a front view corresponding to the region included in the candidate contour Figure.

Preferably, the image recognition module is further configured to perform image processing on the video image frame to identify a pawn in the video image frame; the positioning module includes: a homography matrix acquisition submodule, configured to: Calculating a homography matrix of a current field of view of the camera according to the plane identifier and an internal parameter matrix of the camera obtained in advance; a corner point coordinate obtaining sub-module for using a plane reference object according to a preset reference point on the chessboard The known coordinates in the coordinate system and the position of the row and the row on the board, combined with the grid length feature of the board, acquire coordinates of the four corner points of the board in the plane identifier coordinate system; the chess piece coordinate acquisition a module, configured to read a pixel of the piece according to the piece recognized by the image recognition module, and acquire coordinates of the piece in the coordinate system of the plane identifier according to the homography matrix; a positioning sub-module for combining the coordinates of the four corner points of the chess piece and the checkerboard in the plane identifier coordinate system with the grid length feature of the checkerboard, Taking the position of the pieces on the board.

Preferably, the homography matrix acquisition sub-module includes: a coordinate system determining unit, configured to select the plane identifier coordinate system as a world coordinate system; and a vertex coordinate reading unit, configured to read the plane label a pixel coordinate of the four vertices of the object; an operation unit for using the coordinates of the four vertices of the known planar identifier in the coordinates of the world coordinate system, the pixel coordinates of the four vertices of the planar identifier, and The internal parameters of the camera obtained in advance are substituted into the following formulas:

Calculate the homography matrix H=M[r1,r2,r3,t] of the current field of view;

Where (x, y, 1) represents the homogeneous coordinate of the pixel coordinates of any corner point in the visual positioning identifier in the image coordinate system of the camera, and (X, Y, Z, 1) indicates that the vertex is in the world coordinate system. The homogeneous coordinates are reduced to (X, Y, 0, 1), s is the introduced arbitrary scale proportional parameter, M is the camera internal reference matrix, and r1, r2, and r3 respectively represent the rotation of the visual positioning identification coordinate system relative to the camera coordinate system. Three column vectors in the matrix, t is the translation vector.

Preferably, the positioning module further includes: a determining submodule, configured to determine, according to the coordinates of the pieces and the four corner points of the board in the plane identifier coordinate system, that the piece is on the board Inside the quadrilateral area consisting of four corner points.

Preferably, the positioning module further includes: a target area determining sub-module, configured to: coordinate according to the four corner points of the checkerboard in the plane identifier coordinate system, and homography of the current field of view of the camera a matrix, obtaining pixel coordinates of the four corner points of the checkerboard; and selecting a quadrilateral region composed of four corner points of the checkerboard as a target area in the video image frame; the chess piece coordinate acquisition sub-module includes: a chess piece pixel coordinate a reading unit, configured to: after the image recognition module recognizes a piece in the target area in the video image frame, read pixel coordinates of the piece; the piece coordinate calculation unit is configured to perform pixel coordinates according to the piece And the homography matrix, acquiring coordinates of the chess piece in the plane identifier coordinate system.

In a third aspect, the present invention also discloses a storage medium storing a plurality of instructions executed by one or more processors to implement the machine vision-based pieces of the present invention. The steps of the positioning method.

The storage medium in the solution can be used to store software programs and modules, such as the machine vision based chess piece positioning method of the present invention and the program instructions/modules corresponding to the system. These program instructions/modules can be executed by the processor to implement the machine vision based piece positioning method/device described above.

In a fourth aspect, the present invention further discloses a robot, comprising: a processor, configured to implement each instruction; a storage medium, configured to store a plurality of instructions; wherein: the processor is configured to execute an instruction stored by the storage medium, The steps of implementing the machine vision based piece positioning method of the present invention.

In the present solution, the processor in the robot executes the instructions stored in the storage medium of the present invention provided in the robot to implement the machine vision-based piece positioning method of the present invention. The robot of the invention can position the pieces by machine vision, facilitates subsequent simulation of real people playing chess, and enhances the user's interactive experience.

The invention only needs to place a plane marker on the plane where the chessboard is located, by which the chess piece can be visually and accurately positioned. The invention does not require a special circuit implementation of the board, and no data communication is required between the board and the robot, which is more convenient to arrange. In addition, due to the identification of the planar marker, even if the planar marker can be accurately identified in a complicated scene, it will not be interfered by complicated scenes.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention, Those skilled in the art can also obtain other drawings based on these drawings without paying for inventive labor.

1 is a flow chart of an embodiment of a machine vision based chess piece positioning method according to the present invention;

2 is a schematic diagram of the principle of Zhang Zhengyou's plane calibration method;

3 is a flow chart of another embodiment of a machine vision based chess piece positioning method according to the present invention;

4 is a flow chart of another embodiment of a machine vision based chess piece positioning method according to the present invention;

5 is a flow chart of another embodiment of a machine vision based chess piece positioning method according to the present invention;

6 is a schematic diagram of a system composition of another embodiment of a chess piece positioning method based on machine vision according to the present invention;

7 is a flow chart of another embodiment of a machine vision based chess piece positioning method according to the present invention;

8 is a flow chart of another embodiment of a machine vision based chess piece positioning method according to the present invention;

Figure 9a is a schematic diagram of a calibration template of Zhang Zhengyou's plane calibration method;

Figure 9b is a schematic diagram of the detection feature points in the Zhang Zhengyou plane calibration method;

10 is a front view of a quadrilateral contour after rotation in another embodiment of a machine vision based chess piece positioning method according to the present invention;

11 is a block diagram of another embodiment of a machine vision based chess piece positioning system according to the present invention;

12 is a block diagram of another embodiment of a machine vision based chess piece positioning system according to the present invention;

13 is a block diagram of another embodiment of a machine vision based chess piece positioning system according to the present invention;

14 is a block diagram of another embodiment of a machine vision based chess piece positioning system according to the present invention;

15 is a block diagram of another embodiment of a machine vision based chess piece positioning system of the present invention;

Figure 16 is a block diagram showing the structure of an embodiment of the robot of the present invention.

Reference mark:

1001--memory; 1002--processor; 1003--peripheral interface; 1004--camera module; 1005--audio module; 1006--touch screen; 1007--bus.

Detailed ways

The present invention will be further described in detail with reference to the accompanying drawings, in which . All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

Machine vision is a branch of artificial intelligence that is rapidly evolving. Simply put, machine vision is to use the machine instead of the human eye to make measurements and judgments. Machine vision system is through machine vision products (ie images The ingesting device converts the ingested object into an image signal and transmits it to an image processing system to obtain a shape information of the object to be captured, and converts it into a digitized signal according to information such as pixel distribution, brightness, color, and the like; The image system performs various operations on these signals to extract features of the target, and then controls the action of the device on the spot according to the result of the discrimination. The invention provides a chess piece positioning method based on machine vision, which is applied to position a piece on a board provided with a plane identifier. As shown in FIG. 1 , the piece positioning method comprises:

In this embodiment, only a plane identifier is placed on the plane where the board is located, and the board can be visually and accurately positioned by the identifier, and the present invention does not require a special circuit implementation of the board. The planar markers here enable accurate identification in complex scenes without interference from complex scenes.

Before the method is used to locate the chess piece, the camera for collecting the video stream needs to be calibrated. Specifically, by calibrating the camera, the internal reference matrix of the camera is obtained, that is, the geometric model of the imaging of the camera is determined.

Use the image captured by the camera to restore objects in space. Here, it is assumed that there is a simple linear relationship between the image captured by the camera and the object in the three-dimensional space: [image]=M[object], where the matrix M can be regarded as the geometric model of the camera imaging. . The parameters in M are camera parameters. Usually, these parameters are obtained through experiments and calculations. This process of solving parameters is called camera calibration. The more common method is Zhang Zhengyou's plane calibration method. For the sake of simplicity, in the image measurement process and machine vision applications, it is often the case that the image captured by the camera is used to restore objects in space. Here, let's assume that there is a simple linear relationship between the image captured by the camera and the object in three-dimensional space: [like]=M[object].

Zhang Zhengyou calibration method, with the help of the machine vision calibration plate, the feature circle on the calibration plate is distributed in display, with a spacing of 30mm; the board size is 300X300mm. The four large circles mark the circle, and the big ring mark determines the direction. Calibration is performed using the center coordinates of the feature circle. The method of extracting the center coordinates of the feature circle: obtaining four mark circle coordinates, using the affine transformation to adjust the coordinates of the feature circle, and then sorting them to determine the coordinates of the corresponding feature circle image.

It is assumed here that the template plane is on the plane of the world coordinate system Z=0. The schematic diagram is shown in Figure 2. The basic principle is as follows:

Where K is the internal reference matrix of the camera, [X Y 1] ^T is the homogeneous coordinate of the point on the template plane, [u v 1] ^T is the homogeneous coordinate of the point on the template plane projected onto the corresponding point on the image plane, [r1 r2 R3] and t are the rotation matrix and translation vector of the camera coordinate system relative to the world coordinate system, respectively.

According to the nature of the rotation matrix, ie r1 ^T r2=0 and ||r1||=||r2||=1, the following two basic constraints on the internal reference matrix can be obtained for each image.

Since the camera has 5 unknown internal parameters, when the number of captured images is greater than or equal to 3, K can be solved linearly and uniquely, that is, the internal reference matrix of the camera can be solved.

Zhang Zhengyou's plane calibration algorithm can be described as:

1. Print a template and attach it to a flat surface;

2. Shooting several template images from different angles;

3. detecting feature points in the image;

4. Find the internal parameters, external parameters, and distortion coefficients of the camera.

In the above embodiment, preferably, step S100 acquires a video stream by using a camera, and collects a video image frame collected in the video image frame from the video stream as a key frame, and the key frame can reflect a change of the chess piece on the chessboard. .

Another embodiment of the method of the present invention, as shown in FIG. 3, includes:

S230 acquires a front view of the candidate contour;

S240, when the front view of the candidate contour is consistent with the pre-stored planar identifier template, identifying an image corresponding to the candidate contour as an image of the planar identifier;

In the above embodiment, S210 performs binary separation on the video image frame, and binarization can be regarded as setting the gray value of the pixel on the image to 0 or 255, that is, presenting the entire image with obvious black and white vision. effect. Image segmentation is to separate different regions of a particular meaning in an image. These regions are mutually disjoint, and each region satisfies the consistency of a particular region. Image segmentation is an important issue in image processing and a classic problem in computer vision research. Image understanding in computer vision, including object detection, feature extraction, and target recognition, depends on the quality of the segmentation.

Of course, we can use the adaptive threshold method to binary segment the video image frames obtained by the camera. The threshold method is a simple and effective segmentation method. Its biggest feature is that the calculation is simple and thus widely used.

After the image is segmented, you need to identify the screen marker, because the camera may have taken pictures from different angles. The video stream, therefore, the planar identifier on the captured video frame does not have to be the front view. Then, it is necessary to extract the planar marker photographed from various angles, and perform edge detection on the planar marker to obtain the common features of the contour of the planar marker, and then store the common feature in advance to facilitate the video capture. The planar identifier is identified after the frame. Specifically, after the video frame image is binary-divided, the contours of the divided images are extracted, and the candidate contours are searched according to the common features of the contours of the planar identifiers. For example, if the screen identifier we selected is a square, then after the edge detection, the common feature of the outline can be quadrilateral, then we can exclude those non-convex polygons from the extracted outline, and Not a quadrilateral outline, etc.

Preferably, we can also set some restrictions to further exclude the outline of the plane marker, or the square of the plane marker, for example, the extracted outline has a side smaller than the other side ( The shape is relatively long and thin, the contour perimeter or the area is too small, etc., and the remaining eligible contours are the alternative contours that the planar markers may correspond to.

After the candidate contour is obtained, the next step is to obtain a front view of each candidate contour, that is, to change the candidate contour image area corresponding to the plane identifier into a front view, thereby further determining whether the candidate contour region is a plane identifier. The image corresponding to the object. Specifically, by comparing the front view of each candidate contour with the pre-stored planar identifier template, when the alignment is consistent, it can be determined that the candidate contour is the shape of the planar identifier. The contour, the image corresponding to this alternative contour is the image of the planar marker, and thus the planar marker is identified.

Preferably, on the basis of the above embodiment, the planar identifier disposed on the chessboard is a square, and step S220 of the above embodiment is based on the pre-existing pre-existing common features of the contour profile corresponding to the pre-stored planar identifier. A common feature of the contour corresponding to the planar identifier is that the contour of the planar identifier corresponds to a quadrilateral. If the set markers are different, then the common features of the corresponding outlines will be different. When selecting a flat marker, we should try to select a marker that is easy to identify and has obvious feature points. Like a square, with four vertices, of course, in order to match the chessboard Separate, other shapes such as patterns can be set in the square to facilitate subsequent recognition. The four sides of the square are consistent in length, and the four vertices are more favorable for subsequent positioning operations, which makes the calculation more simple.

In another embodiment of the present invention, as shown in FIG. 4, the chess piece positioning method of the present invention is applied to positioning a piece on a board provided with a square plane identifier, and specifically includes:

S221 is a quadrilateral according to a common feature of the pre-stored contour identifier corresponding to the plane identifier, and selecting a quadrilateral contour contour from the extracted contour contour as an alternative contour;

S231 reading pixel coordinates of four vertices of the candidate contour;

S233: Calculate the photographic transformation matrix H according to the pixel coordinates of the vertices of the four sets of the candidate contours obtained by the above-mentioned acquired and the defined candidate contours, and obtain the photographic transformation matrix H; The pixel coordinates of the vertices of the four sets of the candidate contours and the pixel coordinates of the vertices after the defined candidate contours are converted into the front view are respectively substituted into the following equations to obtain the projective transformation matrix H:

S234 performing, according to the projective transformation matrix H, performing projective transformation on all pixels of the binarized image corresponding to the candidate contour, and obtaining a front view corresponding to the region included in the candidate contour;

In the above embodiment, a square planar marker is taken as an example, and steps S231-S234 specifically explain how to obtain a front view of the extracted candidate contour. According to the square characteristics of the square plane identifier, for example, we can define the pixel coordinates of the four vertices after the candidate contour is transformed into the front view by the projective transformation. For example, the four vertex pixel coordinates of the square plane identifier front view are respectively (0) , 0), (0,200), (200,200), (200,0), then the four vertex coordinates can be used as the pixel coordinates of the four vertices of the quadrilateral candidate contour, in addition, the quadrilateral candidate contour The pixel coordinates of the original image can be read from the image, and the projective transformation matrix H can be obtained by using the above-described projective transformation equation. By using the projective transformation matrix H, all the pixels of the quadrilateral candidate contour region can be subjected to projective transformation, and a front view corresponding to the region of the quadrilateral candidate contour can be obtained, and then the front view and the pre-stored planar identifier template are obtained. For comparison, see if it is consistent. If it is consistent, the quadrilateral contour is the outline of the planar marker, and the planar marker is recognized. Here, the front view is selected for comparison because the image of the marker photographed by the camera from various angles is different. If the camera is directly above the marker, the image of the marker is also a rectangle. If the camera is tilted, the rectangle is rectangular. The image of the marker is an irregular quadrilateral. The purpose of obtaining the front view for comparison is that the front view of the planar marker is an image viewed from directly above, regardless of the angle from which the camera is viewed. Identification of objects.

According to another embodiment of the method of the present invention, on the basis of any of the foregoing embodiments, after the planar identifier is identified, the chess piece on the chessboard can be located according to the planar identifier combined with the pre-acquired camera internal reference matrix. As shown in Figure 5, including:

The homography matrix in the above embodiment, that is, the rotation matrix R and the translation vector t of the plane identifier coordinate system with respect to the camera coordinate system. The camera coordinate system here refers to the coordinate system used for the pixel coordinates and is the image coordinate system of the camera. In the foregoing embodiment, step S310 calculates a homography matrix of the current field of view of the camera according to the plane identifier and the internal parameter matrix of the camera that is acquired in advance:

S312 reading pixel coordinates of four vertices of the planar identifier;

Calculate the homography matrix H=M[r1,r2,r3,t] of the current field of view;

Where (x, y, 1) represents the homogeneous coordinate of the pixel coordinates of any vertex in the plane image in the image coordinate system of the camera, and (X, Y, Z, 1) indicates that the vertex is in the world coordinate system The homogeneous coordinates in the middle are reduced to (X, Y, 0, 1), s is the scale parameter of any scale introduced, and M is the camera. The internal reference matrix, r1, r2, and r3 respectively represent three column vectors in the rotation matrix of the plane identifier coordinate system with respect to the camera coordinate system, and t is a translation vector.

The above can write an equation by the coordinates of a vertex pixel of the marker and the coordinates of the corresponding marker coordinate system; the four vertices of the identifier can write four equations, then four unknowns r1, r2 can be obtained. R3 and t; specifically, through the direct forward transform (DLT) algorithm of the equations, r1, r2, r3 and t can be obtained, that is, the homography matrix of the camera in the current field of view.

After obtaining the homography matrix, the next step is to determine the chessboard. In the above step S320, a reference point is set in advance on the chessboard according to the coordinates of the reference point in the plane coordinate system (ie, the world coordinate system). By using its position on the board and the grid length relationship of the board, the coordinates of the four vertices of the board in the plane coordinate system can be expressed by the value of the reference point and the length of the grid. Therefore, the board can be obtained. The coordinates of the four corner points in the plane marker coordinate system, that is, the position area of the checkerboard in the plane marker coordinate system. The preset reference point on the board is known because the coordinates are known, and the length of the grid on the board is known. Then, according to the position of the reference point on the board, combined with the length of the checkerboard grid, the four corners of the board are obtained in the plane marker. The coordinates in the coordinate system. Preferably, when the reference point is set on the board, one of the four corner points is directly selected as the preset reference point, or the center point of the board, for example, the center point of the board is selected, and the center point is in the world coordinate system (equivalent The coordinates in the plane marker coordinate system are (X1, Y1). According to the grid length feature of the board, for example, the side length of the board is L, then the coordinates of the four vertices of the board in the world coordinate system are (X1- L/2, Y1+L/2), (X1-L/2, Y1-L/2), (X1+L/2, Y1+L/2), (X1+L/2, Y1-L/ 2), the coordinates of the four corner points in the plane marker coordinate system (world coordinate system) are obtained.

After determining the coordinate position of the board in the plane identifier coordinate system, it is also necessary to determine the coordinates of the piece in the plane identifier coordinate system. In the above step S330, the pieces in the video image frame are identified, and we can use the previous plane identifier. A similar method is to binarize the video image frame and then identify the pieces from it. Specifically, the contour recognition can be performed according to the characteristics of the chess piece, for example, a black circular chess piece is selected, and the thickness of the chess piece can be neglected, then the video image frame after binarization is selected. In the middle, the image of the chess piece is a circular or elliptical contour, and then the contour in the binary image is judged as a piece according to the circumference, the area, the second-order center distance, and the like of the contour, and the center of the contour is the pixel coordinate of the piece. After the pieces are identified, the pixel coordinates of the pieces can be read directly from the video image frame. After the pixel coordinates of the piece are obtained, the coordinates of the piece in the plane coordinate coordinate system (that is, the world coordinate system) can be obtained by using the homography matrix of the current field of view of the camera obtained before.

Finally, in step S340, we can obtain the specific position of the piece on the board based on the previously determined pieces of the chess piece, the coordinates of the board in the plane coordinate coordinate system, and the grid length feature of the board. If the coordinates of the four corner points of the board in the plane marker have been determined, then we can select any one of the corner points as the comparison point for the calculation of the position of the board. For example, the corner point in the upper left corner is selected as the comparison point. Since the coordinates of the upper left corner vertex A as the comparison point in the plane identifier coordinate system are known, the position of the piece in the checkerboard grid can be found (ie, The first few grids). The method of obtaining is shown in Figure 6:

1. It is known that the coordinates of point A in the world coordinate system O1 are (6, 6, 0), and the horizontal and vertical spacing of the checkerboard grid are both 5.

2. If the calculated piece is in the world coordinate system O1, the coordinates are (11, 11, 0). Then the position of the pawn B in the checkerboard grid is calculated as: lateral position = (11-6)/5=1; vertical position = (11-6)/5=1; that is, the position of the pawn B in the checkerboard grid For (1,1), the position of the checkerboard reference point A in the checkerboard grid is (0, 0).

Preferably, on the basis of the foregoing embodiment, after step S330, before step S340, the method further includes: S335, according to the coordinates of the pieces and the four corner points of the board in the plane identifier coordinate system, It is determined that the piece is in a quadrilateral region composed of four corner points of the board.

Only the pieces on the board need to be positioned, and the pieces that are not on the board can be left without the need for positioning. Obtaining the coordinates of the four corner points of the board in the plane identifier coordinate system is equivalent to determining the area of the board in the plane identifier coordinate system (the quadrilateral area composed of four corner points), then the piece is obtained. After the coordinates in the plane marker coordinate system, you can judge that the piece is No on the board, only the pieces that are determined to be on the board need further positioning before they enter the calculation of the position of the upper row of the board. It should be noted that the quadrilateral region composed of the four corner points of the chess piece includes the checkerboard boundary line itself, and the chess piece is on the checkerboard boundary line and belongs to the chessboard. That is to say, if the piece is on either side of the quadrilateral composed of four corner points, it is also considered to be in the quadrilateral area (on the board). Of course, a piece outside the quadrilateral area consisting of corner points indicates that the piece is not on the board, so there is no need to further locate it.

Another embodiment of the method of the present invention is applied to positioning a chess piece on a board provided with a planar identifier. The chess piece positioning method is as shown in FIG. 7 and includes:

S332: acquiring, according to the pixel coordinates of the piece and the homography matrix, coordinates of the piece in the coordinate system of the plane identifier;

On the basis of the above embodiment, the present embodiment determines the identified target area before identifying the piece, that is, first determines the area where the board is located in the video image frame. In this way, it is only necessary to identify the pieces in the target area, and the parts outside the target area need not be identified, so that the range of the piece recognition can be reduced, and the pieces identified in the target area are definitely on the board. In this way, the workload of image processing can be reduced, the recognition range can be narrowed, and the recognition speed can be accelerated. In addition, the image outside the board can be ignored when detecting the pieces and calculating the position of the pieces, thereby increasing system robustness.

Another embodiment of the method of the present invention is shown in Figure 6 as a schematic diagram of the system composition:

a. As shown in the figure, a specially designed checkerboard has a plane marker in the upper left corner of the board. The center O1 of the plane marker is used as the origin of the entire checkerboard coordinate system.

b. Point A is the vertex of the checkerboard grid closest to the plane identifier, and the two sides of the checkerboard grid are respectively parallel to the X and Y axes of the checkerboard coordinate system. The coordinates of vertex A in the checkerboard coordinate system are known as (Ax, Ay, 0), and B is the piece placed at the intersection of the checkerboard grid.

c.O2 is the origin of the coordinate system of the robot camera.

Specifically, the chess piece needs to be calibrated before positioning, and the subsequent chess piece recognition is performed after the camera is calibrated, as shown in FIG. 8 , including: planar marker recognition (acquisition video stream → image binarization → quadrilateral detection → acquisition) Front view → template matching → matching success), chess piece positioning (calculation R and t → chess piece positioning). The detailed process is as follows:

Step 1. Camera calibration

The camera calibration calculates the internal parameters of the camera, determines the geometric model of the camera image, and then uses the image captured by the camera to calculate the position of the three-dimensional space. The invention uses Zhang Zhengyou's plane calibration method to calculate the internal parameter matrix M of the camera. The process of the plane calibration method is as follows:

1 print a plane-calibrated template and affix it to a plane, as shown in Figure 9a;

2 taking several template images from different angles;

3 detecting feature points in the image; a schematic diagram is shown in Figure 9b;

4 according to the coordinate position of the feature point in the world coordinate system and its corresponding pixel coordinates in the image, The simultaneous equations are used to find the internal parameters, external parameters and distortion coefficients of the camera.

Step 2. Flat marker identification

This step is used to identify a specific planar marker in the image and determine the four corner points of the planar marker. The specific process is as follows:

1 quadrilateral contour detection

The process detects a quadrilateral contour that the planar marker in the image may correspond to. The image obtained by the camera is binarized by the adaptive threshold method, and then the contour is extracted from the binary image obtained above. Polygon fitting the contours, discarding those non-convex polygons, and those that are not quadrilateral. Some additional constraints are used to eliminate quadrilateral contours that are not likely to be planar marker images, such as the quadrilateral side being significantly smaller than the remaining edges (the shape is relatively elongated), the contour perimeter or area being too small. The remaining eligible contours are the alternative contours that the planar markers may correspond to.

2Get a front view of the quadrilateral outline area

The process is used to change the quadrilateral image area corresponding to the planar marker into a front view, thereby further determining whether the quadrilateral region is an image corresponding to the planar marker. The four vertex pixel coordinates after the quadrilateral contour is converted into a front view by projective transformation are (0, 0), (0, 100), (100, 100), (100, 0). The four vertex pixel coordinates of the quadrilateral outline are known in the original image. For each set of points corresponding to the following plane projective transformation equation is established:

Where X is the homogeneous coordinate of the quadrilateral vertices in the original image, and the non-homogeneous pixel coordinates are

Through the four sets of vertex correspondences, the simultaneous equations can be solved to solve the projective transformation matrix H, the original image All of the pixels perform a projective transformation H, and a front view corresponding to the quadrilateral contour area is obtained, and the front view is a square planar identification image.

Because the image of the marker taken by the camera from various angles is different, if the camera is directly above the marker, the image of the marker is also a rectangle; if the camera is tilted, the image of the rectangular marker is not Regular quadrilateral. The purpose of obtaining the front view in this step is that no matter which angle the camera looks from, the obtained front view is an image viewed from directly above. It facilitates the subsequent identification of the marker ID, that is, the normalization of the image.

3 Determine whether the quadrilateral corresponds to the plane rectangle identifier

As shown in the figure below, the front view image of the acquired plane mark is rotated by 90°, 180°, and 270° in sequence to finally obtain four images, as shown in FIG. 10 .

The four images obtained above are respectively matched with the image templates of the planar markers. When the matching with any of the sub-images is successful, the quadrilateral contour corresponds to the planar identifier.

Step 3. Chess position calculation

This step calculates the position of the pawn on the board relative to the marker by the above identified planar marker and its four corner points. The position of the piece in the checkerboard grid is finally calculated by the coordinates of the known checkerboard vertex A. The specific process is as follows:

1 Calculate the rotation matrix R and translation vector t of the plane marker coordinate system relative to the camera coordinate system

The coordinate system of the plane marker is the world coordinate system, and its center O1 is the origin of the world coordinate system, and the point Z coordinate of the point on the plane where the marker is located is 0. The plane marker has a side length of 80 mm, and the coordinates of its four vertices in the world coordinate system are (-40, -40, 0), (40, -40, 0), (40, 40, 0), (-40,40,0). The pixel coordinates of the vertex of the plane identifier on the image and the coordinates of the vertex in the corresponding world coordinate system have the following equation:

Where (x, y, 1) indicates that any corner point in the visual positioning identifier is in the image coordinate system of the camera The homogeneous coordinates of the pixel coordinates, (X, Y, Z, 1) represent the homogeneous coordinates of the corner points in the world coordinate system, that is, simplified to (X, Y, 0, 1), and s is an arbitrary introduction. The scale ratio parameter, M is the internal parameter matrix of the camera, and r1, r2, and r3 respectively represent three column vectors in the rotation matrix of the visual positioning identification coordinate system with respect to the camera coordinate system, and t is a translation vector.

The above equation can be written by a vertex pixel coordinate of the marker and its corresponding coordinate in the marker coordinate system. The four vertices of the marker can be written to correspond to the four equations, and the direct linear transformation (DLT) algorithm of the equations is used to obtain r1, r2, r3 and t.

2 Get the four corner pixel coordinates of the board

It is known that the coordinates of the checkerboard origin A in the plane identifier coordinate system O1 are known as (Ax, Ay, 0), and the side length of the checkerboard is L, and the coordinates of the other three vertices of the checkerboard in the plane identifier coordinate system are (Ax+L, Ay, 0), (Ax+L, Ay+L, 0), (Ax, Ay+L, 0). According to the above step e, the rotation matrix matrix and the translation vector of the current plane identifier coordinate system with respect to the camera coordinate system have been obtained. Substituting the coordinates of the four vertices of the board in the world coordinate system into the formula 1.1, the pixel coordinate values of the four vertices of the board can be obtained.

3 Get the pixel coordinates of the piece in the original image

This program uses black round pieces, the thickness of the pieces is relatively thin and can be ignored. In the binary image of the original image, the image of the chess piece is a circular or elliptical outline. According to the perimeter, the area and the second-order center distance of the contour, it is judged whether the contour in the binary image is a chess piece, and the center of the contour is the pixel coordinate of the chess piece. Verify that the piece is in the quadrilateral area of the four vertices of the board, and if so, the piece is on the board.

4 Get the position of the piece on the board

Substituting the pixel coordinates of the chess piece into the above formula 1.1, solving the equation can obtain the coordinate B (Bx, By, 0) of the chess piece in the world coordinate system. It is known that the side length of each cell of the board is a, and the position of the row on the board where the piece is placed can be calculated by row=(By-Ay)/a, col=(Bx-Ax)/a.

The invention only needs to place a plane marker on the plane where the chessboard is located, and the robot passes the standard The object can be visually and accurately positioned on the pieces. The invention does not require a special circuit implementation of the board, and no data communication is required between the board and the robot, which is more convenient to arrange. The template of the planar marker is specifically designed, and the robot can accurately recognize the planar marker in a complicated scene without being disturbed by a complicated scene.

Based on the same technical concept, the present invention also discloses a machine vision-based chess piece positioning system, which is applied to position a chess piece on a chessboard provided with a planar identifier. Specifically, as shown in FIG. 11, the machine-based machine The visual chess piece positioning system includes: a camera acquisition module 100, configured to acquire a video stream, and collect a video image frame from the video stream; and an image recognition module 200, configured to perform video image frames collected by the camera acquisition module 100 An image processing to identify the planar identifier in the video image frame; the positioning module 300 is configured to calculate a chess piece on the chessboard relative to the plane according to the planar identifier and the pre-acquired internal reference matrix of the camera Describe the location of the marker to position the piece.

The system embodiment of the present invention can be integrated into the robot so that the robot can accurately position the pieces in a complex background, so that the subsequent chess strategy can be laid out according to the position of the pieces on the board. Only let the robot understand the real chess board, increase the anthropomorphic characteristics of the robot, and let the user get a better sense of substitution, so that the user has a better interactive experience.

On the basis of the foregoing system embodiment, as shown in FIG. 12, the machine vision-based chess piece positioning system further includes: a calibration module 400, configured to calibrate the camera, and acquire an internal parameter matrix of the camera.

There are many methods for camera calibration. The more common one is Zhang Zhengyou's plane calibration method. For details, please refer to the description of Zhang Zhengyou's calibration method in the previous method embodiment.

According to another embodiment of the system of the present invention, on the basis of any of the foregoing system embodiments, as shown in FIG. 13, the image recognition module 200 includes: a binary segmentation sub-module 210, configured to perform the video image frame. Binary segmentation; a shape extraction sub-module 220, configured to extract a contour from the segmented binarized image; a storage sub-module 230, configured to store a common feature of the contour corresponding to the planar marker, the plane a identifier template; a determination processing sub-module 240, configured to store according to the storage sub-module 230 The common feature of the contour corresponding to the planar identifier, selects the contour having the common feature from the contour extracted by the shape extraction sub-module 220 as an alternative contour; the front view acquisition sub-module 250, Obtaining a front view of the candidate profile; the identification sub-module 260 is configured to identify the candidate when the front view of the candidate profile is consistent with the planar identifier template prestored by the storage sub-module 230 The image corresponding to the contour is an image of the planar marker.

In the above embodiment, the image recognition module is described in detail. After the video image frame is acquired, the image recognition module performs image processing and recognition on the video image frame, and the binary segmentation sub-module is used to perform binarization segmentation. Processing, and then extracting the outline of the segmented image through the shape extraction sub-module, and then identifying by the image recognition module to see if it is the outline of the planar marker. Here we need to obtain the common features of the outline of the plane identifier in advance, and store it in the storage sub-module in the image recognition module in advance, so that the judgment processing sub-module in the image recognition module can be judged when selecting the candidate contour. standard. Preferably, in the selected candidate contours, the culling is further performed by a preset constraint condition, and the outline of the planar marker is further excluded, and the subsequent comparison range is reduced. After the candidate contours are selected, the front view acquisition sub-modules are required to obtain the front view of the candidate contours, and the comparison sub-module performs the front view of the candidate contours one by one with the pre-stored template of the planar identifiers. In comparison, the template of the planar marker here can directly select the front view image of the planar marker. Only if the comparison is consistent, the candidate contour can be judged as the outline of the planar marker. In addition, during the alignment process, it may be necessary to planarly rotate the front view of the candidate contours from different rotation angles so that the alignment is correct.

Preferably, the planar identifier is a square, and the common feature of the contour corresponding to the planar identifier stored by the storage sub-module is that the contour corresponding to the planar identifier is a quadrilateral.

The difference in the selection of the planar markers, the difficulty of the calculations will be more or less different, but the basic ideas are the same, all need to identify the markers first, and then according to the identified markers and the previous calibration The internal reference matrix of the camera is used to position the pieces. Only the selection of the planar marker directly determines the difficulty of identifying the planar marker. We generally use easy to identify, with outstanding The shape of the feature point is used as a marker. For example, selecting a square as a planar marker facilitates subsequent calculation and recognition.

According to another embodiment of the system of the present invention, the front view obtaining submodule includes: a reading unit, configured to read pixel coordinates of four vertices of the candidate contour; and a defining unit, a pixel coordinate for defining four vertices of the candidate contour after being projectively transformed into a front view is a pixel coordinate of four pre-existing front views of the plane marker; a calculating unit for reading according to the Taking the four vertex pixel coordinates read by the unit and the four vertex pixel coordinates defined by the defining unit, the projective transformation matrix H is obtained by the following equation,

By reading the pixel coordinates of the four vertices of the candidate contour and the defined transformation into the pixel coordinates of the four vertices after the front view, four sets of equations can be obtained by the formula equation of the projective transformation matrix, thereby obtaining the projective transformation matrix H. Then, a front view of all pixels of the inner region of the candidate contour is obtained according to the projective transformation matrix, so as to be easily compared with the planar identifier template.

According to another embodiment of the present invention, the image recognition module is further configured to perform image processing on the video image frame to identify a pawn in the video image frame. As shown in FIG. 13 , the positioning module 300 includes: a homography matrix acquisition sub-module 310, configured to calculate a current field of view of the camera according to the plane identifier and an internal parameter matrix of the camera acquired in advance. Responsive matrix; corner coordinate acquisition sub-module 320 for pre-setting reference based on the board Obtaining the coordinates of the four corner points of the checkerboard in the plane identifier coordinate system by combining the known coordinates in the plane identifier coordinate system and the row and column positions on the checkerboard in combination with the grid length feature of the checkerboard a piece coordinate acquisition sub-module 330, configured to read a pixel of the piece according to the piece recognized by the image recognition module 200, and acquire the piece of the piece in the plane identifier according to the homography matrix a coordinate in the coordinate system; a chess piece locating sub-module 340, configured to acquire, according to the coordinates of the four corner points of the chess piece and the checkerboard in the plane identifier coordinate system, combined with the grid length feature of the checkerboard The position of the piece on the board.

In the foregoing device embodiment, the homography matrix acquisition sub-module, on the basis of the foregoing embodiment, the homography matrix acquisition sub-module includes: a coordinate system determining unit, configured to select the plane identifier coordinate system as the world coordinate a vertex coordinate reading unit for reading pixel coordinates of four vertices of the plane identifier, and an operation unit for using the coordinates of the four vertices of the known planar identifier in the coordinates of the world coordinate system, The pixel coordinates of the four vertices of the planar identifier and the internal parameters of the pre-obtained camera are respectively substituted into the following formula:

Calculate the homography matrix H=M[r1,r2,r3,t] of the current field of view;

The corner point coordinate obtaining sub-module acquires the chessboard according to the known coordinates of the preset reference point on the checkerboard in the plane identifier coordinate system and the position of the row on the board, combined with the grid length feature of the checkerboard The coordinates of the four corner points in the plane identifier coordinate system; the chess piece coordinate acquisition sub-module, first identifying the pieces in the board by the image recognition module, so that the piece coordinates acquisition sub-module Obtaining the pixel coordinates of the piece; then, obtaining the coordinates of the piece in the plane marker coordinate system (world coordinate system) by the homography matrix of the current field of view acquired by the homography matrix acquisition sub-module. Finally, the chess piece positioning sub-module can be obtained according to the position coordinates of the chessboard in the plane marker coordinate system (world coordinate system), the position coordinates of the chess piece in the plane marker coordinate system, and the mesh length feature of the checkerboard. The position of the pieces in the checkerboard grid (ie, the grid of the horizontal and vertical).

Preferably, as shown in FIG. 14, the positioning module 300 further includes: a determining sub-module 350, configured to coordinate the four corner points of the chess piece and the checkerboard in the plane identifier coordinate system, It is determined that the piece is in a quadrilateral region composed of four corner points of the board.

The judging sub-module obtains the coordinates of the four corner points of the board acquired by the sub-module in the plane identifier coordinate system and the pieces obtained by the sub-module acquisition sub-module in the plane identifier according to the corner point coordinate before the piece positioning sub-module performs positioning. The coordinates in the coordinate system determine whether the piece is on the board. If it is determined that the piece is on the board, the piece positioning sub-module performs subsequent calculations to obtain the position of the piece on the board. If it is not on the board, no subsequent positioning calculations are required.

According to another embodiment of the apparatus of the present invention, as shown in FIG. 15, on the basis of any of the above embodiments, the positioning module 300 further includes: a target area determining sub-module 360 for using four corners of the board Obtaining a coordinate in the plane identifier coordinate system and a homography matrix of the current field of view of the camera, obtaining pixel coordinates of the four corner points of the checkerboard; and selecting a quadrilateral composed of four corner points of the checkerboard a region as a target region in the video image frame; the pawn coordinate acquisition sub-module 330 includes: a pawn pixel coordinate reading unit 331 for the image recognition module 200 to identify within a target region in the video image frame After the piece is played, the pixel coordinates of the piece are read; the piece coordinate calculation unit 332 is configured to acquire the piece of the piece in the coordinate system of the plane according to the pixel coordinate of the piece and the homography matrix. coordinate.

In the embodiment of the device, after the corner coordinate acquisition sub-module acquires the coordinates of the four corner points of the checkerboard in the plane identifier coordinate system, the target area determining sub-module is according to the four corner points of the checkboard. The coordinates in the plane identifier coordinate system and the homography matrix of the current field of view of the camera Depicting pixel coordinates of four corner points of the board; and selecting a quadrilateral area composed of four corner points of the board as a target area in the video image frame; the image recognition module can narrow the image recognition range to the target area, The pieces recognized by the image recognition module in the target area are definitely on the board, which is the target piece that we need to accurately locate. When detecting the pieces and calculating the position of the pieces, the images outside the board can be ignored, thereby increasing the robustness of the system and improving the positioning efficiency.

The system embodiment of the present invention corresponds to the method embodiment of the present invention, and the technical details of the method embodiment of the present invention are also applicable to the system embodiment of the present invention. Therefore, the system embodiment of the present invention can also refer to the method of the present invention. For the embodiments, the two can be referred to each other.

The invention also discloses a storage medium storing a plurality of instructions, the plurality of instructions being executed by one or more processors to implement the following steps:

Preferably, on the basis of the above, another embodiment of the storage medium of the present invention, the storage medium stores a plurality of instructions, and the plurality of instructions are executed by one or more processors to implement any of the present inventions. The steps of the embodiment of the machine vision based chess piece positioning method.

The steps of the machine vision-based chess piece positioning method of the present invention can be referred to the previous method embodiment part, and the repetition is not repeated here.

Finally, the present invention also discloses a robot, comprising: a processor for implementing each instruction; a storage medium for storing a plurality of instructions; wherein: the processor is configured to execute an instruction stored by the storage medium to implement The steps of the embodiment of the machine vision based chess piece positioning method of the present invention.

As shown in FIG. 16, another embodiment of the robot of the present invention includes: a memory 1001, one or more (only one shown in the figure) processor 1002, a peripheral interface 1003, a camera module 1004, and a sound. The frequency module 1005 and the touch screen 1006. These components communicate with one another via one or more communication buses 1007/signal lines.

It will be understood that the structure shown in FIG. 16 is merely illustrative and does not limit the structure of the robot. The robot may also include more or less components than those shown in FIG. 16, or have different devices than those shown in FIG. The components shown in FIG. 16 can be implemented in hardware, software, or a combination thereof.

The memory can be used to store software programs and modules, such as the machine vision-based chess piece positioning method and the program instructions/modules corresponding to the system embodiments in the embodiment of the present invention, and the processor executes by allowing software programs/modules stored in the memory. Various functional applications and data processing, that is, the above-described machine vision-based chess piece positioning method/system is implemented.

The memory can include high speed random access memory and can also include non-volatile memory such as one or more magnetic storage devices, flash memory or other non-volatile solid state memory. The storage medium may be a magnetic disk, an optical disk, a read only memory (ROM), a random access memory (RAM), or the like.

The peripheral interface couples various input/output devices to the CPU level one memory. The processor runs various software in the memory, commands various functions of the robot, and performs data processing.

The camera module is used to capture video, which is equivalent to the eyes of a robot, such as a CCD camera.

An audio module for receiving or transmitting a sound signal, which may include one or more microphones, one or more speakers, and an audio circuit. It is convenient for the robot to capture sound signals from the environment and to communicate with people through a microphone or speaker.

The touch screen provides an output and input interface between the robot and the person. Specifically, the touch screen displays video output to a person, and the content of the video output may include text, graphics, video, and any combination thereof. Some input results are corresponding to some users decrypting the object. The touch screen also receives human gestures such as clicks, slides, and the like. Specific embodiments of touch screens include, but are not limited to, liquid crystal displays or light emitting polymer displays.

In the embodiment of the present invention, the machine vision-based chess piece positioning method and system, the storage medium and the robot belong to the same concept, and on the robot, the instructions stored in the storage medium are executed by the processor, and the corresponding office can be operated. The method provided in the embodiment of the machine-based chess piece positioning method is described in detail in the previous embodiment of the machine vision-based chess piece positioning method, and details are not described herein again.

It should be noted that a general tester in the art can understand that all or part of the process of implementing the machine vision-based piece positioning method according to the embodiment of the present invention can be completed by controlling a related hardware by a computer program. Stored in a computer readable storage medium, such as in a memory of the robot, and executed by at least one processor within the robot, and may include, as described, a machine vision based chess piece positioning method embodiment Process.

While the preferred embodiment of the invention has been described, it will be understood that Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and the modifications and

It is apparent that those skilled in the art can make various modifications and variations to the invention without departing from the spirit and scope of the invention. Thus, it is intended that the present invention cover the modifications and modifications of the invention

Claims

A machine vision-based chess piece positioning method is characterized in that it is applied to positioning a chess piece on a chessboard provided with a planar identifier, the chess piece positioning method comprising:

S100 acquires a video stream by using a camera, and collects a video image frame from the video stream;

S200 performs image processing on the video image frame to identify the plane identifier;

S300 calculates a position of the piece on the board relative to the identifier according to the plane identifier and the internal parameter matrix of the camera acquired in advance, thereby positioning the position of the piece.
The machine vision-based piece positioning method according to claim 1, wherein the capturing of the video stream by the camera in the step S100 and the capturing of the video image frame from the video stream further comprises:

S010 calibrates the camera to obtain an internal reference matrix of the camera.
The machine vision-based chess piece locating method according to claim 1, wherein the step S200 performs image processing on the video image frame, and the identifying the planar identifier is specifically:

S210 performs binary segmentation on the video image frame, and extracts a contour from the segmented binarized image;

S220: selecting, according to a common feature of the pre-stored contour identifiers of the plane identifiers, an outline contour having the common features as an alternative contour from the extracted contour contours;

S230 acquires a front view of the candidate contour;

S240: When the front view of the candidate contour is consistent with the pre-stored planar identifier template, the image corresponding to the candidate contour is identified as an image of the planar identifier.
The machine vision-based piece positioning method according to claim 3, wherein the plane identifier is a square, and the plane identifier pre-stored in the step S220 is The common feature of the shape profile is quadrilateral;

The step S230 obtains a front view of the candidate contour, which is specifically:

S231 reading pixel coordinates of four vertices of the candidate contour;

S232 defines a pixel coordinate of the four vertices after the candidate profile is subjected to projective transformation into a front view as pixel coordinates of four pre-existing front views of the planar marker;

S233, the pixel coordinates of the vertices of the four sets of the candidate contours obtained by the above-mentioned acquired candidate contours and the pixel coordinates of the vertices after the definition of the candidate contour are converted into the following equations, and the projection transformation matrix H is obtained:

Where X is the homogeneous pixel coordinate of the candidate silhouette vertex, and the non-homogeneous pixel coordinates are

X' is the homogeneous coordinate of the vertex after the projective transformation, and its non-homogeneous pixel coordinates are

S234 performs projective transformation on all pixels of the binarized image corresponding to the candidate contour according to the projective transformation matrix H, and obtains a front view corresponding to the region included in the candidate contour.
The machine vision-based chess piece positioning method according to any one of claims 1 to 4, wherein the step S300 calculates the chessboard based on the plane identifier and the pre-acquired internal parameter matrix of the camera. The position of the piece relative to the marker, thereby locating the position of the piece is specifically:

S310: calculating, according to the plane identifier and the pre-acquired internal reference matrix of the camera, a homography matrix of a current field of view of the camera;

S320: according to the known coordinates of the preset reference point on the checkerboard in the plane identifier coordinate system and the rank position on the checkerboard, combined with the grid length feature of the checkerboard, acquiring the four corner points of the checkerboard in the plane The coordinates in the marker coordinate system;

S330 identifies a piece in the video image frame, reads pixel coordinates of the piece, and acquires coordinates of the piece in the plane identifier coordinate system according to the homography matrix;

S340 obtains the position of the chess piece on the chessboard according to the coordinates of the chess piece and the four corner points of the chessboard in the plane identifier coordinate system, combined with the grid length feature of the chessboard.
The machine vision-based chess piece positioning method according to claim 5, wherein the step S310 calculates the current field of view of the camera according to the plane identifier and the internal parameter matrix of the camera acquired in advance. The homography matrix is specifically:

S311 selects the plane identifier coordinate system as a world coordinate system;

S312 reading pixel coordinates of four vertices of the planar identifier;

S313 substituting the coordinates of the four vertices of the known planar identifier in the coordinates of the world coordinate system, the pixel coordinates of the four vertices of the planar identifier, and the internal parameters of the camera obtained in advance into the following formula:

Calculate the homography matrix H=M[r1,r2,r3,t] of the current field of view;

Where (x, y, 1) represents the homogeneous coordinate of the pixel coordinates of any vertex in the plane image in the image coordinate system of the camera, and (X, Y, Z, 1) indicates that the vertex is in the world coordinate system The homogeneous coordinates in the middle are reduced to (X, Y, 0, 1), s is the introduced arbitrary scale proportional parameter, M is the camera internal reference matrix, and r1, r2, r3 respectively represent the planar marker coordinate system relative to the camera. Three column vectors in the rotation matrix of the coordinate system, t is the translation vector.
The machine vision-based chess piece locating method according to claim 5, wherein after the step S330 and before the step S340, the method further comprises:

S335 determines, according to the coordinates of the pieces and the four corner points of the board in the plane identifier coordinate system, that the pieces are in a quadrilateral area composed of four corner points of the board.
The machine vision-based chess piece locating method according to claim 5, wherein after the step S320 and before the step S330, the method further comprises:

S325: obtaining pixel coordinates of four corner points of the checkboard according to coordinates of the four corner points of the checkerboard in the plane identifier coordinate system and a homography matrix of the current field of view of the camera;

S326: selecting, according to pixel coordinates of the four corner points of the checkerboard, a quadrilateral region composed of four corner points of the checkerboard as a target area in the video image frame;

The step S330 identifies the pieces in the video image frame, and reads the pixel coordinates of the piece, and acquires the coordinates of the piece in the plane identifier coordinate system according to the homography matrix:

S331 identifies a piece in a target area in the video image frame, and reads pixel coordinates of the piece;

S332: Acquire coordinates of the chess piece in the plane identifier coordinate system according to the pixel coordinates of the chess piece and the homography matrix.
A machine vision-based chess piece positioning system is characterized in that it is applied to positioning a chess piece on a chessboard provided with a plane identifier, the machine vision-based chess piece positioning system comprising:

a camera acquisition module, configured to acquire a video stream, and collect a video image frame from the video stream;

An image recognition module, configured to perform image processing on a video image frame acquired by the camera acquisition module, to identify the planar identifier in the video image frame;

And a positioning module, configured to calculate a position of the piece on the board relative to the identifier according to the plane identifier and an internal parameter matrix of the camera acquired in advance, thereby locating the position of the piece.
A machine vision-based chess piece positioning system according to claim 9, further comprising:

The calibration module is configured to calibrate the camera to obtain an internal parameter matrix of the camera.
A machine vision based chess piece positioning system according to claim 9, wherein

The image recognition module includes:

a binary segmentation submodule, configured to perform binary segmentation on the video image frame;

a shape extraction sub-module for extracting a contour from the segmented binarized image;

a storage submodule, configured to store a common feature of the outline corresponding to the planar marker, and the planar identifier template;

a judging processing module, configured to select, according to a common feature of the contour contour corresponding to the plane identifier stored in the storage submodule, an outer shape extracted from the outer shape extracting submodule The contour is used as an alternative contour;

a front view obtaining submodule for obtaining a front view of the candidate contour;

And an identifying sub-module, configured to identify, when the front view of the candidate contour is consistent with the planar identifier template pre-stored by the storage sub-module, an image corresponding to the candidate contour as an image of the planar identifier.
The machine vision-based chess piece positioning system according to claim 11, wherein the planar identifier is a square, and the common feature of the outline corresponding to the planar identifier stored by the storage sub-module is quadrilateral;

The front view obtaining submodule includes:

a reading unit, configured to read pixel coordinates of four vertices of the candidate contour;

Defining a unit for defining four vertices after the candidate contour is projectively transformed into a front view The pixel coordinates are the pixel coordinates of the four vertices of the front view of the pre-stored planar marker;

a calculating unit, configured to obtain a projective transformation matrix H according to four vertex pixel coordinates read by the reading unit and four vertex pixel coordinates defined by the defining unit, by using an equation

Where X is the homogeneous pixel coordinate of the candidate silhouette vertex, and the non-homogeneous pixel coordinates are

X' is the homogeneous coordinate of the vertex after the projective transformation, and its non-homogeneous pixel coordinates are

a transforming unit, configured to perform a projective transformation on all pixels of the binarized image corresponding to the candidate contour according to the projective transformation matrix H obtained by the calculating unit, to obtain a front view corresponding to the region included in the candidate contour Figure.
A machine vision-based piece positioning system according to any one of claims 9 to 12, wherein the image recognition module is further configured to perform image processing on the video image frame, thereby The pieces are identified in the image frame;

The positioning module includes:

a homography matrix acquisition sub-module, configured to calculate a homography matrix of a current field of view of the camera according to the plane identifier and an internal parameter matrix of the camera acquired in advance;

a corner coordinate acquisition sub-module, configured to acquire the checkerboard according to a known coordinate of a preset reference point on the checkerboard in a plane identifier coordinate system and a rank position on the checkerboard, combined with a grid length feature of the checkerboard The coordinates of the four corner points in the plane identifier coordinate system;

a piece coordinate acquisition sub-module, configured to read a pixel coordinate of the piece according to the piece recognized by the image recognition module, and obtain the piece of the piece in the plane mark according to the homography matrix The coordinates in the object coordinate system;

a piece positioning sub-module, configured to acquire the piece of the piece on the chessboard according to the coordinates of the piece and the four corner points of the board in the plane identifier coordinate system, combined with the grid length feature of the board position.
The machine vision-based chess piece positioning system according to claim 13, wherein the homography matrix acquisition sub-module comprises:

a coordinate system determining unit, configured to select the plane identifier coordinate system as a world coordinate system;

a vertex coordinate reading unit, configured to read pixel coordinates of four vertices of the plane identifier;

An operation unit, configured to use the coordinates of the four vertices of the known planar identifier in the coordinates of the world coordinate system, the pixel coordinates of the four vertices of the planar identifier, and the pre-acquired internal parameters of the camera Substituting the following formulas:

Calculate the homography matrix H=M[r1,r2,r3,t] of the current field of view;

Where (x, y, 1) represents the homogeneous coordinate of the pixel coordinates of any corner point in the visual positioning identifier in the image coordinate system of the camera, and (X, Y, Z, 1) indicates that the vertex is in the world coordinate system. The homogeneous coordinates are reduced to (X, Y, 0, 1), s is the introduced arbitrary scale proportional parameter, M is the camera internal reference matrix, and r1, r2, and r3 respectively represent the rotation of the visual positioning identification coordinate system relative to the camera coordinate system. Three column vectors in the matrix, t is the translation vector.
The machine vision-based chess piece positioning system according to claim 13, wherein the positioning module further comprises:

a judging sub-module, configured to determine, according to the coordinates of the pieces and the four corner points of the board in the plane identifier coordinate system, that the pieces are in a quadrilateral area composed of four corner points of the board Within the domain.
The machine vision-based chess piece positioning system according to claim 13, wherein the positioning module further comprises:

a target area determining submodule, configured to obtain four corner points of the checkboard according to coordinates of the four corner points of the checkerboard in the plane identifier coordinate system and a homography matrix of a current field of view of the camera Pixel coordinates; and selecting a quadrilateral region composed of four corner points of the checkerboard as a target region in the video image frame;

The chess piece coordinate acquisition sub-module includes:

a chess piece pixel coordinate reading unit, configured to: after the image recognition module recognizes a piece in the target area in the video image frame, read pixel coordinates of the piece;

The piece coordinate calculation unit is configured to acquire coordinates of the piece in the plane identifier coordinate system according to the pixel coordinates of the piece and the homography matrix.
A storage medium, characterized in that the storage medium stores a plurality of instructions, the plurality of instructions being executed by one or more processors to implement the machine vision according to any one of claims 1-8 The steps of the chess piece positioning method.
A robot characterized by comprising:

a processor for implementing each instruction;

a storage medium for storing a plurality of instructions;

Wherein: the processor is configured to execute the instructions stored in the storage medium to implement the steps of the machine vision based piece positioning method according to any one of claims 1-8.