KR102618285B1 - Method and system for determining camera pose - Google Patents

Abstract

This disclosure relates to a method for determining a camera pose. The camera pose determination method includes extracting a plurality of first corner points on a visual marker from an image including a visual marker, calculating homography based on coordinates on a virtual plane of the plurality of first corner points and coordinates in the image. , determining coordinates in the image corresponding to coordinates on the virtual plane of the plurality of second corner points using homography, and determining a pose of the camera based on the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image. Includes decision-making steps.

Description

Camera pose determination method and system {METHOD AND SYSTEM FOR DETERMINING CAMERA POSE}

The present disclosure relates to a method and system for determining a camera posture, and specifically to a method and system for determining the posture of a camera using a visual marker.

Recently, visual markers have been used for camera calibration or posture measurement in various fields such as augmented reality and robotics. Here, the visual marker may refer to an artificial landmark made in a certain format including square patterns. For example, a mechanical or electronic device including a camera captures a visual marker using a camera, and determines the transformation relationship between the visual marker on the reference coordinate system and the visual marker on the image coordinate system and the parameters of the camera based on the captured image. It can be calculated. Using the transformation relationship between coordinate systems and camera parameters calculated in this way, the device can identify a specific location that is the target of operation, or determine its own posture including direction, angle, tilt, etc.

However, it may be difficult for a mechanical or electronic device to recognize a visual marker and accurately identify a specific location to be operated depending on changes in the operating environment. For example, if the distance between the device and the visual marker is long or the location of a specific point on the visual marker is inaccurate, it may be difficult for the device to accurately identify a specific location required for operation. In this case, there is a problem that the posture of the device or the camera installed in the device cannot be accurately determined, so the results of the operation performed based on the posture may also be inaccurate.

The present disclosure provides a method for determining a camera posture, a computer program stored in a recording medium, and a system (device) to solve the above problems.

The present disclosure may be implemented in various ways, including a method, a system (device), or a computer program stored in a readable recording (storage) medium.

According to an embodiment of the present disclosure, a method for determining a camera pose executed by at least one processor includes extracting a plurality of first corner points on a visual marker from an image including a visual marker, the plurality of first corner points calculating homography based on the coordinates on the virtual plane and the coordinates in the image, determining coordinates in the image corresponding to the coordinates on the virtual plane of the plurality of second corner points using the homography, and a plurality of second corner points. It includes determining the pose of the camera based on the coordinates of the corner point on the virtual plane and the coordinates in the image.

A computer program stored in a computer-readable recording medium is provided to execute the camera pose determination method according to an embodiment of the present disclosure on a computer.

A computing device according to an embodiment of the present disclosure includes a communication module, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory. At least one program extracts a plurality of first corner points on a visual marker from an image including a visual marker, and calculates homography based on coordinates on a virtual plane of the plurality of first corner points and coordinates in the image, Using homography, determine coordinates in the image corresponding to coordinates on the virtual plane of the plurality of second corner points, and determine the pose of the camera based on the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image. Includes commands for

In various embodiments of the present disclosure, the computing device may improve the precision of camera pose determination by determining the pose of the camera using not only the plurality of first corner points but also all of the plurality of extracted second corner points.

In various embodiments of the present disclosure, the processor may remove inaccurate corner points among a plurality of extracted corner points and/or adjust coordinates to reduce errors in corner point recognition and improve the precision of camera pose determination.

In various embodiments of the present disclosure, even when the processor does not know the relative distance within the image of each of the plurality of second corner points extracted from the visual marker, the processor calculates the coordinates within the image of the plurality of second corner points using homography. It can be used to determine the camera pose.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as “a person skilled in the art”) in the technical field to which the present disclosure pertains from the description of the claims. It will be understandable.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals indicate like elements, but are not limited thereto.
FIG. 1 is a diagram illustrating an example of a computing device determining the posture of a camera according to an embodiment of the present disclosure.
Figure 2 is a block diagram showing the internal configuration of a computing device according to an embodiment of the present disclosure.
Figure 3 is a functional block diagram showing the internal configuration of a processor according to an embodiment of the present disclosure.
FIG. 4 is a diagram illustrating an example in which a processor extracts a plurality of corner points from a visual marker included in an image according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating an example in which a processor calculates coordinates in an image of a plurality of second corner points according to an embodiment of the present disclosure.
FIG. 6 is an exemplary diagram showing the location of a corner point displayed on a visual marker in an image according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating an example in which coordinates in an image of a corner point of a visual marker are adjusted according to an embodiment of the present disclosure.
FIG. 8 is a diagram illustrating an example in which a processor determines the posture of a camera according to an embodiment of the present disclosure.
FIG. 9 is a diagram illustrating an example in which visual markers are aligned on a grid map according to an embodiment of the present disclosure.
Figure 10 is a diagram illustrating an example of a visual marker being utilized according to an embodiment of the present disclosure.
11 is a flowchart illustrating an example of a method for determining a camera posture according to an embodiment of the present disclosure.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the present disclosure is complete and that the present disclosure does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the function in the present disclosure, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term 'module' or 'unit' used in the specification refers to a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. Components and 'modules' or 'parts' may be combined into smaller components and 'modules' or 'parts' or further components and 'modules' or 'parts'. Could be further separated.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

In the present disclosure, a ‘visual (visual) marker’ or a ‘visual fiducial marker’ refers to a specific function or function in a three-dimensional space, such as augmented reality (AR), robotics, autonomous vehicles, etc. It may refer to an indicator used to identify the location of the target of an action or the posture of an object. For example, a visual marker may be configured to include a specific pattern that can be captured or identified by a camera installed on a computing device, robot, autonomous vehicle, etc.

In the present disclosure, 'camera pose' refers to rotation (yaw) and tilt (pitch, roll) indicating the orientation of the camera, or the pose and position of the camera (e.g., three-dimensional coordinates in space). For example, the pose of a camera has 6 degrees of freedom, including coordinates (x, y, z) in three-dimensional space, pitch, roll, and yaw. It can be expressed as, or at least part of 6 degrees of freedom. Additionally, camera intrinsic parameters used to determine the camera posture may be determined in advance or extracted through camera calibration, etc.

FIG. 1 is a diagram illustrating an example of how the computing device 110 determines the posture of a camera according to an embodiment of the present disclosure. According to one embodiment, the computing device 110 may determine the posture of the camera 120 using a visual marker included in an image captured by the camera 120. Here, the computing device 110 may refer to any mechanical device including a camera (image sensor) 120, a motion sensor, etc. For example, the computing device 110 is a robot or vehicle capable of autonomous driving based on the image captured by the camera 120, a portable computer or wearable computer that provides an augmented reality function, etc., based on the image captured by the camera. It may include, but is not limited to, a device configured to determine the posture of the camera 120 and perform autonomous driving or a specific operation according to the posture of the camera 120.

According to one embodiment, the computing device 110 may move in an arbitrary area and capture images of the surrounding environment or objects using the camera 120 installed on the computing device 110. For example, the computing device 110 may capture an image 130 that includes a visual marker attached or printed on one side of a specific object. Additionally, the computing device 110 may pre-store or receive from the outside information about the visual marker 140 on the virtual plane corresponding to the visual marker included in the image 130. Here, the visual marker 140 may include a predetermined pattern for guiding or instructing the computing device 110 to an object or object located in a specific location or specific area. For example, the visual marker 140 may refer to a visual reference marker created based on libraries and/or applications for augmented reality, robotics, and camera calibration, such as ARToolKit, ARTag, AprilTag, ArUco, etc. The visual marker 140 may include a plurality of grid patterns arranged on a grid map composed of grids at regular intervals. For example, the visual marker 140 may include grid patterns of a plurality of first colors (eg, white) and grid patterns of a plurality of second colors (eg, black). In other words, the visual marker 140 may be composed of a combination of grids of a plurality of first colors and grids of a plurality of second colors. The computing device 110 may determine the posture of the camera 120 using the image 130 including the visual marker configured as described above.

In the first operation 112, the computing device 110 may extract a plurality of first corner points on the visual marker from the image 130 including the visual marker. For example, a corner point may refer to a point where vertices of grids of different colors meet among the vertices of grids in a visual marker. In addition, the plurality of first corner points may refer to four points corresponding to the vertices of a square (i.e., a square or rectangle determined by a predetermined format/standard of the visual marker) included in the visual marker, but are limited to this. It doesn't work. Here, the square included in the visual marker may be composed of a combination of grids of one or more colors.

In the second operation 114, the computing device 110 may calculate homography using a plurality of first corner points. Specifically, the computing device 110 may calculate homography based on the coordinates of the plurality of first corner points on the virtual plane and the coordinates in the image. Here, the virtual plane may represent a plane of a grid map in which grid patterns including a plurality of first corner points are aligned. In this case, the coordinates in the image of the plurality of first corner points and/or the distance between the plurality of first corner points may be determined based on the format of the predetermined visual marker. Additionally, homography may include or refer to a matrix for converting coordinates on a virtual plane of a plurality of first corner points into coordinates in an image of a plurality of first corner points. That is, the homography may include a matrix (H) calculated by Equation 1 below.

Then, in a third operation 116, computing device 110 uses the coordinates on the virtual plane of the plurality of second corner points and the calculated homography to generate a plurality of second corner points corresponding to the coordinates on the virtual plane. The coordinates within the image can be determined. Here, the plurality of second corner points may include a plurality of first corner points, but is not limited thereto. For example, the plurality of second corner points may include five or more points where vertices of grids of different colors meet among the vertices of grids in the visual marker. Additionally, when homography is applied to the coordinates on the virtual plane of the plurality of second corner points calculated by the computing device 110, the coordinates of the plurality of second corner points in the image may be determined. Through the above-described process, the computing device 110 may determine the coordinates of the plurality of second corner points extracted from the visual marker 140 in the image.

According to one embodiment, in the fourth operation 118, the computing device 110 may determine the posture of the camera 120 based on the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image. In this case, the computing device 110 may measure or determine the posture of the camera 120 based on the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image using the Perspective n-Point (PnP) algorithm. there is. For example, the PnP algorithm can be expressed as Equation 2 below.

here, is the scale constant, is the coordinate in the image of the plurality of second corner points, May be the coordinates on the virtual plane of the plurality of second corner points. also, is a matrix representing the intrinsic matrix or internal parameters of the camera 120 and may be predetermined by camera calibration, may be a rotation and movement transformation matrix representing a transformation relationship between coordinates on a virtual plane of a plurality of second corner points and coordinates in an image. That is, the posture of the camera 120 determined by the PnP algorithm is a rotation and movement transformation matrix representing the transformation relationship between the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image and/or the interior of the camera 120. Can contain parameters.

According to one embodiment, some of the coordinates in the image of the plurality of third corner points (i.e., the plurality of third points including the plurality of second corner points) extracted from the visual marker correspond to the corners on the visual marker in the image. It may not work. In this case, the computing device 110 determines the coordinates in the image corresponding to the coordinates on the virtual plane of the plurality of third corner points using homography, and the coordinates in the image of the plurality of third corner points are the visual markers in the image. You can determine whether it corresponds to the coordinates of the corner of the image. Then, the computing device 110 may determine the coordinates of the plurality of second corner points in the image by removing corner points that do not correspond to the coordinates of the corners on the visual marker in the image among the plurality of third corner points.

According to one embodiment, the coordinates in the image of the plurality of second corner points extracted from the visual marker may not match the coordinates of the corners on the visual marker in the image. For example, each of the coordinates in the image of the plurality of second corner points may indicate a vicinity within a preset distance of the corner on the corresponding visual marker. In this case, the computing device 110 may adjust the coordinates of the plurality of second corner points in the image so that the coordinates of the plurality of second corner points match the coordinates of the corners on the visual marker in the image. For example, the computing device 110 may perform refinement on coordinates in an image of a plurality of second corner points. With this configuration, the computing device 110 determines the posture of the camera 120 using not only the plurality of first corner points but also all of the extracted second corner points, thereby determining the posture of the camera 120. Precision can be improved.

FIG. 2 is a block diagram showing the internal configuration of the computing device 110 according to an embodiment of the present disclosure. The computing device 110 may include a memory 210, a processor 220, a communication module 230, and an input/output interface 240. As shown in FIG. 2 , computing device 110 may be configured to communicate information and/or data over a network using a communication module 230 .

Memory 210 may include any non-transitory computer-readable recording medium. According to one embodiment, the memory 210 is a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. mass storage device). As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the computing device 110 as a separate persistent storage device that is distinct from memory. In addition, the memory 210 includes an operating system and at least one program code (e.g., corner point extraction from a visual marker image installed and driven in the computing device 110, homography calculation, corner point coordinate calculation, camera posture code for measurement/decision, etc.) can be stored.

These software components may be loaded from a computer-readable recording medium separate from the memory 210. Recording media readable by such a separate computer may include recording media directly connectable to the computing device 110, for example, floppy drives, disks, tapes, DVD/CD-ROM drives, memory cards, etc. It may include a computer-readable recording medium. As another example, software components may be loaded into the memory 210 through the communication module 230 rather than a computer-readable recording medium. For example, at least one program is a computer program (e.g., corner point extraction, It may be loaded into the memory 210 based on a program for calculating graphics, calculating coordinates of a corner point, measuring/determining the pose of a camera, etc.).

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to a user terminal (not shown) or another external system by the memory 210 or the communication module 230. For example, the processor 220 may receive an image including a visual marker and extract a plurality of first corner points and/or a plurality of second corner points from the visual marker. In this case, the processor 220 calculates homography based on the coordinates on the virtual plane of the extracted plurality of first corner points and the coordinates in the image, and uses the homography to calculate the coordinates on the virtual plane of the plurality of second corner points. The coordinates in the image corresponding to may be determined, and the posture of the camera may be determined based on the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image.

The communication module 230 may provide a configuration or function for a user terminal (not shown) and the computing device 110 to communicate with each other through a network, and the computing device 110 may be connected to an external system (for example, a separate cloud system). etc.) may provide a configuration or function for communication. For example, control signals, commands, data, etc. provided under the control of the processor 220 of the computing device 110 pass through the communication module 230 and the network to the user terminal and/or the communication module of the external system. and/or transmitted to an external system. For example, the user terminal and/or the external system may receive the visual marker image received from the computing device 110, the determined camera posture, etc.

Additionally, the input/output interface 240 of the computing device 110 may be connected to the computing device 110 or may be a means for interfacing with a device (not shown) for input or output that the computing device 110 may include. . In FIG. 2 , the input/output interface 240 is shown as an element configured separately from the processor 220, but the present invention is not limited thereto, and the input/output interface 240 may be included in the processor 220. Computing device 110 may include more components than those of FIG. 2 . However, there is no need to clearly show most prior art components.

The processor 220 of the computing device 110 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals and/or a plurality of external systems. According to one embodiment, the processor 220 may receive an image including a visual marker from a user terminal and/or an external system. In this case, the processor 220 extracts a plurality of first corner points on the visual marker based on the received image, calculates homography, determines the coordinates of the plurality of second corner points in the image, and determines the pose of the camera. can be measured or determined.

FIG. 3 is a functional block diagram showing the internal configuration of the processor 220 according to an embodiment of the present disclosure. As shown, the processor 220 may include a photographing unit 310, a homography calculation unit 320, a coordinate adjustment unit 330, a posture determination unit 340, etc. According to one embodiment, the photographing unit 310 may continuously or intermittently capture images of an external area by controlling a camera installed in the computing device. For example, an image captured by a camera under the control of the capturing unit 310 may include a visual marker attached or displayed on a portion of the external area.

According to one embodiment, the processor 220 may extract a plurality of first corner points on the visual marker from the image captured and acquired by the photographing unit 310. For example, the plurality of first corner points may refer to four corner points corresponding to the vertices of a square included in the visual marker. Then, the processor 220 may determine coordinates on the virtual plane and within the image of the plurality of first corner points.

The homography calculation unit 320 may calculate homography using the coordinates of the plurality of first corner points on the virtual plane and the coordinates in the image. For example, homography may represent a matrix for converting coordinates on a virtual plane of a plurality of first corner points into coordinates in an image of the plurality of first corner points. This homography can be used to convert coordinates on a virtual plane of all corner points on a visual marker including a plurality of first corner points to coordinates in an image. That is, the processor 220 uses the homography calculated by the homography calculation unit 320 to calculate the coordinates in the image of each of the plurality of second corner points from the coordinates on the virtual plane of each of the plurality of second corner points. can do.

According to one embodiment, the processor 220 may determine coordinates in an image corresponding to coordinates on a virtual plane of a plurality of third corner points using homography. For example, the plurality of third corner points are points extracted from the visual marker and may include a plurality of first corner points and a plurality of second corner points. The processor 220 may determine whether the coordinates of the plurality of third corner points in the image correspond to the coordinates of the corners on the visual marker in the image. Then, the processor 220 may determine the coordinates of the plurality of second corner points in the image by removing corner points that do not correspond to the coordinates of the corners on the visual marker in the image among the plurality of third corner points. In this case, the processor 220 extracts the coordinates of the corner on the visual marker in the image using an appropriate corner detector algorithm, and the coordinates in the image of the plurality of third corner points are the corners of the corner on the visual marker in the image. You can determine whether it corresponds to the coordinates. For example, the corner detector algorithm may include a known algorithm for extracting corner coordinates from an image including a visual marker, such as a Harris corner detector.

The coordinate adjustment unit 330 may adjust the coordinates of the plurality of second corner points in the image so that the coordinates of the plurality of second corner points match the coordinates of the corners on the visual marker in the image. In this case, adjusting the coordinates may include correcting the coordinates of each of the plurality of second corner points so that the coordinates in the image match the coordinates of the corners of the visual marker. Here, the coincidence between coordinates is not limited to both coordinates being the same, and may include both coordinates becoming closer or similar than before adjusting at least one of the coordinates. In this case, the processor 220 extracts the coordinates of the corners on the visual marker in the image using a corner detector, and adjusts the coordinates in the image of the plurality of second corner points to match the coordinates of the corners on the visual marker in the image. You can. For example, the corner detector may include the Harris corner detector described above.

According to one embodiment, the posture determination unit 340 may determine the posture of the camera based on the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image. For example, the posture determination unit 340 may use a PnP algorithm to determine the posture of the camera based on the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image. In this case, as described above, the posture determination unit 340 removes corner points that do not correspond to the coordinates of the corners on the visual marker in the image, and then creates a virtual plane for each of the remaining corner points and/or the corner points whose coordinates have been adjusted. The pose of the camera can be determined by inputting the coordinates of the image and the coordinates within the image into the PnP algorithm. In this case, the PnP algorithm may include, but is not limited to, the EPnP (Efficient Perspective-n-Point) algorithm, and an appropriate algorithm for determining the pose of the camera may be used.

In FIG. 3, the configuration of the processor 220 is described separately for each function, but this does not necessarily mean that they are physically separated. For example, the homography calculation unit 320 and the coordinate adjustment unit 330 are described separately, but this is only to aid understanding of the invention, and a single computing device installed in the processor 220 may perform more than one function. It may be possible. With this configuration, the processor 220 removes inaccurate corner points among a plurality of corner points extracted from an image including a visual marker and/or adjusts the coordinates of the corner points to reduce errors in corner point recognition. This can reduce and improve the precision of camera pose determination.

FIG. 4 is a diagram illustrating an example in which the processor 220 extracts a plurality of corner points from the visual marker 410 included in an image according to an embodiment of the present disclosure. According to one embodiment, when receiving an image including a visual marker 410, the processor 220 may extract a plurality of first corner points 420 from the visual marker 410. In this case, the plurality of first corner points 420 are corner points (420_1, 420_1, 420_2, 420_3, 420_4).

According to one embodiment, the processor 220 creates a plurality of first corner points 420 using a predetermined algorithm that can extract the plurality of first corner points 420 from the visual marker 410 in the image. It can be extracted. Additionally or alternatively, the processor 220 may determine a plurality of first corner points 420 from the extracted plurality of corner points (e.g., a plurality of second corner points, a plurality of third corner points, etc.) A plurality of first corner points 420 can be extracted using a predetermined algorithm. Additionally or alternatively, the processor 220 may extract a plurality of first corner points 420 from information on the visual marker on the virtual plane that is the same as the visual marker 410 included in the image and is pre-stored in the computing device. there is. Here, the distance on the virtual plane between the plurality of first corner points 420 may be predetermined by the format of the visual marker. That is, the processor 220 identifies the positions of the plurality of first corner points 420 in the image, and the positions of the plurality of first corner points 420 in the identified image and the plurality of predetermined first corner points 420 ) can be used to determine the coordinates of the plurality of first corner points 420 in the image.

According to one embodiment, the processor 220 may generate homography using the coordinates of the plurality of first corner points 420 on a virtual plane and the coordinates in the image. Here, the coordinates on the virtual plane of the plurality of first corner points 420 may represent arbitrary coordinates determined by projecting the visual marker in the image onto the virtual plane. That is, the coordinates of the plurality of first corner points 420 on the virtual plane may be arbitrarily determined to be proportional to the positions of the plurality of first corner points 420 in the image and the distances between the plurality of first corner points 420, etc. , but is not limited to this.

FIG. 5 is a diagram illustrating an example in which the processor 220 calculates coordinates 530 in an image of a plurality of second corner points according to an embodiment of the present disclosure. As shown, the processor 220 can calculate the coordinates 530 in the image of the plurality of second corner points using the homography 510 and the coordinates 520 on the virtual plane of the plurality of second corner points. there is. For example, the homography 510 may be calculated based on a plurality of first corner points of the same visual marker as the visual marker from which the plurality of second corner points are extracted.

According to one embodiment, the processor 220 may extract a plurality of second corner points from the visual marker. In this case, the processor 220 may determine the coordinates 520 of each of the plurality of second corner points on the virtual plane. Similar to what was described above, the coordinates 520 on the virtual plane of the plurality of second corner points may represent arbitrary coordinates determined by projecting visual markers in the image onto the virtual plane.

According to one embodiment, the coordinates 520 on the virtual plane of the plurality of second corner points may represent three-dimensional coordinate values, and the coordinates 530 in the image of the plurality of second corner points may represent two-dimensional pixel coordinate values. It can be expressed. In other words, the homography 510 converts the coordinates 520 on the virtual plane of the 3D corner point into the coordinates 530 in the image of the 2D corner point, or converts the coordinates 530 in the image of the 2D corner point into 3 It may include a matrix for converting dimensional corner points into coordinates 520 on a virtual plane. With this configuration, even when the processor 220 does not know the relative distance within the image of each of the plurality of second corner points extracted from the visual marker, the coordinates ( 530) can be calculated and used to determine the camera pose.

FIG. 6 is an exemplary diagram showing the location of a corner point displayed on a visual marker in an image according to an embodiment of the present disclosure. As described above, the processor 220 uses homography and coordinates on a virtual plane of corner points (e.g., a plurality of first corner points, second corner points, third corner points, etc.) to determine the number of corner points. Coordinates within the image can be calculated. As shown, based on the coordinates of the corner point in the image, each corner point may correspond to a specific point of the visual marker in the image.

According to one embodiment, the processor 220 may extract a plurality of corner points including the first point 610 and the second point 620 from the visual marker. Then, the processor 220 uses homography to extract a third point 630 in the image corresponding to the first point 610, and a fourth point in the image corresponding to the second point 620 ( 640) can be extracted. For example, the third point 630 and the fourth point 640 may not correspond to the corner of the visual marker in the image. In this case, the third point 630 and fourth point 640 are not used to determine the pose of the camera and can be removed.

According to one embodiment, the processor 220 may extract the corner and/or the coordinates of the corner on the visual marker in the image using a corner detector. For example, the processor 220 may determine a point where the grid of the first color and the grid of the second color intersect in the visual marker as a corner on the visual marker, and extract the coordinates of the corner. Then, the processor 220 may determine whether at least one of the coordinates of the corner on the visual marker in the image corresponds to the coordinates of the third point 630 and/or the fourth point 640. If the coordinates do not correspond, the third point 630 and/or the fourth point 640 are not used to determine the pose of the camera and may be removed. With this configuration, the processor 220 can improve the accuracy of camera pose determination by removing corner points that do not correspond to the corners of the visual marker in the image.

FIG. 7 is a diagram illustrating an example in which coordinates in an image of a corner point 720 of a visual marker 710 are adjusted according to an embodiment of the present disclosure. According to one embodiment, the coordinates of the corner point 720 in the image may be adjusted to match the coordinates of the corner on the visual marker in the image. In other words, correction or refinement may be performed on the coordinates of the corner point 720 in the image.

As described above, the processor may extract the corner point 720 from the visual marker 710 and determine the coordinates of the extracted corner point 720 on a virtual plane. Additionally, the processor may determine the coordinates of the corner point 720 in the image based on the calculated homography and the coordinates of the corner point 720 on the virtual plane. In this case, the coordinates of the corner point 720 in the image may not match the coordinates of the corner of the visual marker.

In the illustrated example, the first point 730 may represent a point corresponding to the corner point 720 on the virtual plane. Additionally, the second point 740 may represent a point corresponding to the corner of the visual marker in the image. For example, the coordinates of the first point 730 may be determined using the corner point 720 and homography, and the coordinates of the second point 740 may be extracted using a corner detector. Here, the first point 730 and the second point 740 may represent the same corner on the visual marker, but the coordinates of the first point 730 and the coordinates of the second point 740 may not match.

According to one embodiment, the processor may adjust the coordinates of the first point 730 in the image so that the coordinates of the first corner point 730 match the coordinates of the second point 740 on the visual marker in the image. In this case, a predetermined algorithm for adjusting the coordinates of the point can be used. With this configuration, the processor can determine the pose of the camera using the adjusted coordinates, thereby reducing inaccuracy in determining the camera pose.

FIG. 8 is a diagram illustrating an example in which the processor 220 determines the posture 830 of a camera according to an embodiment of the present disclosure. As shown, the processor 220 may determine the pose 830 of the camera using the coordinates 810 on the virtual plane of the plurality of second corner points and the coordinates 820 in the image of the plurality of second corner points. there is.

As described above, the processor 220 uses a PnP algorithm to measure or determine the pose 830 of the camera based on the coordinates 810 on the virtual plane of the plurality of second corner points and the coordinates 820 in the image. You can. For example, the PnP algorithm may be an algorithm for calculating rotation and translation transformation relationships between a coordinate system on a virtual plane and a camera coordinate system. That is, coordinates 810 on the virtual plane may correspond to three-dimensional coordinates, and coordinates 820 in the image may correspond to two-dimensional pixel coordinates in the camera image.

According to one embodiment, the pose 830 of the camera determined using the PnP algorithm includes a rotation and movement transformation matrix representing the transformation relationship between the coordinates 810 on the virtual plane of the plurality of second corner points and the coordinates in the image. It can be included. In this case, the camera's internal parameters can be input to the PnP algorithm. For example, the internal parameters of the camera may include the camera's focal length, height above the ground, etc. Additionally, rotation and translation transformation matrices, internal parameters of the camera, etc. can be used to determine the camera's 6 Degrees of Freedom. That is, the processor 220 uses the coordinates 810 on the virtual plane of the plurality of second corner points and the coordinates 820 in the image of the plurality of second corner points to measure the camera's 6 degrees of freedom (position) 830) can be determined.

FIG. 9 is a diagram illustrating an example in which visual markers 910 are aligned on a grid map 920 according to an embodiment of the present disclosure. According to one embodiment, the visual marker 910 may include a first visual marker 910_1 including at least four reference corner points and a second visual marker 910_2 including at least one additional corner point. there is. In the illustrated example, the first visual marker 910_1 may represent a square marker created according to a predetermined standard/format, and the second visual marker 910_2 may be configured in the form of a checker board. . As described above, when the visual marker 910 is configured to include the first visual marker 910_1 and the second visual marker 910_2, there are more corner points ( first corner point, second corner point, third corner point, etc.) may be extracted. In this way, when more corner points are extracted and used to determine the camera pose, the accuracy of camera pose determination can be improved.

According to one embodiment, the grid map 920 may include a grid of a predetermined size. For example, the grid may be composed of a square or rectangular shape with predetermined horizontal and vertical lengths. In this case, the grid size of the grid map 920 and the grid sizes of the first visual marker 910_1 and the second visual marker 910_2 may be created to be the same. That is, the first color grid and the second color grid constituting the first visual marker 910_1 and/or the second visual marker 910_2 are created with the same size as the predetermined size of the grid on the grid map 920. It can be.

According to one embodiment, when the visual marker 910 and the grid map 920 are combined, the visual marker 930 aligned on the grid map 920 including a grid of a predetermined size may be generated. Here, the processor extracts a plurality of first corner points from the aligned visual marker 930 as described above, calculates a homography, determines the coordinates of the plurality of second corner points, and determines the pose of the camera. You can. In this case, the processor can effectively measure or determine the camera posture by using the visual marker 930 aligned on the grid map to identify the location of the corner point included in the visual marker 930 in advance.

In Figure 9, the first visual marker (910_1) and the second visual marker (910_2) are shown as connected, but this is not limited, and the first visual marker (910_1) and the second visual marker (910_2) are displayed spaced apart. It can be. In addition, in FIG. 9, the visual marker 930 and all grids aligned on the grid map 920 are shown as displayed, but the present invention is not limited thereto, and all grids of the grid map 920 are displayed or at least some of them (e.g. For example, only the outer portion of a grid map) may be displayed.

FIG. 10 is a diagram illustrating an example in which the visual marker 1020 is utilized according to an embodiment of the present disclosure. According to one embodiment, the computing device 1010 may be a device that extracts a corner point from an image of the visual marker 1020 captured by a built-in camera and measures or determines the posture of the camera. For example, the computing device 1010 may be a robot capable of autonomous driving based on images captured by a camera. Additionally, the visual marker 1020 may be a reference indicator for guiding the location and direction of the adapter 1030 for charging the computing device 1010, and may be displayed adjacent to the adapter 1030.

As shown, the computing device 1010 includes a visual marker 1020 to access the adapter 1030 for charging and couple the charging plug connected to the computing device 1010 to the adapter 1030. You can shoot and receive video. Then, the computing device 1010 may extract a plurality of first corner points on the visual marker 1020 from the received image. In addition, the computing device 1010 calculates homography based on the coordinates on the virtual plane of the plurality of first corner points and the coordinates in the image, and uses the homography to correspond to the coordinates on the virtual plane of the plurality of second corner points. The coordinates within the image can be determined. The computing device 1010 may determine the pose of the camera based on the coordinates of the plurality of second corner points on the virtual plane and the coordinates in the image.

According to one embodiment, the computing device 1010 may determine movement and rotation movements for coupling the charging plug to the adapter 1030 based on the determined posture of the camera. For example, in order to couple the charging plug to the adapter 1030, the computing device 1010 moves the charging plug connected to the computing device 1010 toward the adapter 1030 according to a movement transformation matrix among the posture information of the camera. You can move it, adjust the tilt or rotation angle of the charging plug, or modify its position according to the rotation transformation matrix among the camera's posture information. Then, the computing device 1010 may combine the charging plug with the adapter 1030 based on the determined operation and perform charging. In FIG. 10 , the visual marker 1020 is shown as the first visual marker described above in FIG. 9 , but the present invention is not limited thereto, and the visual marker may be configured to include a first visual marker and a second visual marker.

FIG. 11 is a flowchart illustrating an example of a method 1100 for determining a camera pose according to an embodiment of the present disclosure. According to one embodiment, the camera pose determination method 1100 may be performed by an information processing system (eg, at least one processor of a computing device). The camera pose determination method 1100 may be initiated by the processor extracting a plurality of first corner points on the visual marker from an image including the visual marker (S1110). For example, the plurality of first corner points may include at least four corner points.

The processor may calculate homography based on the coordinates of the plurality of first corner points on the virtual plane and the coordinates in the image (S1120). Additionally, the processor may use homography to determine coordinates in the image corresponding to coordinates on the virtual plane of the plurality of second corner points (S1130). For example, the plurality of second corner points may include at least 5 corner points. Additionally, the plurality of second corner points may include a plurality of first corner points.

The processor may determine the pose of the camera based on the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image (S1140). According to one embodiment, the processor may use a PnP algorithm to determine the pose of the camera based on the coordinates on the virtual plane of the plurality of second corner points and the coordinates in the image. Here, the pose of the camera may include a rotation and movement transformation matrix representing a transformation relationship between coordinates on a virtual plane of a plurality of second corner points and coordinates in the image and/or internal parameters of the camera.

The above-described camera pose determination method may be provided as a computer program stored in a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

The methods, operations, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed as any combination of those designed to perform the functions described in. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

For firmware and/or software implementations, techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and PROM ( on computer-readable media such as programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the disclosure is not limited thereto and may also be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Furthermore, aspects of the subject matter of this disclosure may be implemented in multiple processing chips or devices, and storage may be similarly effected across the multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure as can be understood by those skilled in the art. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

110: computing device
112: first movement
114: Second operation
116: Third movement
118: Fourth movement
120: camera
130: Video
140: Visual marker

Claims (20)

A method for determining a camera pose executed by at least one processor, comprising:
extracting a plurality of first corner points on the visual marker from an image including the visual marker;
calculating homography based on coordinates of the plurality of first corner points on a virtual plane and coordinates within the image;
determining coordinates in the image corresponding to coordinates on the virtual plane of a plurality of second corner points using the homography; and
Determining the pose of the camera based on the coordinates of the plurality of second corner points on the virtual plane and the coordinates in the image
Including,
The visual marker includes a predetermined pattern for guiding or indicating an object or object located in a specific location or specific area,
Determining the pose of the camera based on the coordinates of the plurality of second corner points on the virtual plane and the coordinates in the image includes:
Determining the posture of the camera based on the coordinates of the plurality of second corner points on the virtual plane and the coordinates in the image using a Perspective n-Point (PnP) algorithm.
Including,
The plurality of first corner points include at least four corner points,
The method of determining a camera pose, wherein the plurality of second corner points include at least five corner points.
According to paragraph 1,
The step of calculating homography based on coordinates on a virtual plane of the plurality of first corner points and coordinates in the image includes:
Calculating a matrix for converting coordinates of the plurality of first corner points on the virtual plane into coordinates in the image of the plurality of first corner points as the homography.
Including, a method for determining a camera pose.
delete According to paragraph 1,
The pose of the camera includes a rotation and movement transformation matrix representing a transformation relationship between coordinates of the plurality of second corner points on the virtual plane and coordinates in the image.
According to paragraph 1,
Determining coordinates in the image corresponding to coordinates on the virtual plane of a plurality of second corner points using the homography includes:
determining coordinates in the image corresponding to coordinates on the virtual plane of a plurality of third corner points using the homography;
determining whether coordinates of the plurality of third corner points in the image correspond to coordinates of a corner on a visual marker in the image; and
Determining the coordinates of the plurality of second corner points in the image by removing corner points that do not correspond to the coordinates of a corner on a visual marker in the image among the plurality of third corner points.
Including, a method for determining a camera pose.
According to clause 5,
The step of determining whether the coordinates of the plurality of third corner points in the image correspond to the coordinates of a corner on a visual marker in the image,
extracting the coordinates of a corner on a visual marker in the image using a corner detector; and
Determining whether coordinates of the plurality of third corner points in the image correspond to coordinates of a corner on a visual marker in the image
Including, a method for determining a camera pose.
According to paragraph 1,
Adjusting the coordinates of the plurality of second corner points in the image so that the coordinates of the plurality of second corner points in the image match the coordinates of a corner on a visual marker in the image.
A method for determining a camera pose, further comprising:
In clause 7,
Adjusting the coordinates of the plurality of second corner points in the image so that the coordinates of the plurality of second corner points in the image match the coordinates of a corner on a visual marker in the image, comprising:
extracting the coordinates of a corner on a visual marker in the image using a corner detector; and
Adjusting coordinates of the plurality of second corner points in the image to match the coordinates of corners on the visual marker in the image.
Including, a method for determining a camera pose.
According to paragraph 1,
The visual marker includes a first visual marker including at least four reference corner points and a second visual marker including at least one additional corner point.
According to clause 9,
The second visual marker is configured in the form of a checker board.
According to paragraph 1,
The visual marker is aligned on a grid map including a grid of a predetermined size.
delete According to paragraph 1,
The visual marker includes a grid of a plurality of first colors and a grid of a plurality of second colors.
A computer program stored in a computer-readable recording medium for executing the method for determining a camera posture according to any one of claims 1, 2, 4 to 11, and 13 on a computer.
As a computing device,
communication module;
Memory; and
At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory
Including,
The at least one program is,
Extracting a plurality of first corner points on the visual marker from an image including the visual marker,
Calculate homography based on coordinates on a virtual plane of the plurality of first corner points and coordinates in the image,
Determine coordinates in the image corresponding to coordinates on the virtual plane of a plurality of second corner points using the homography,
Includes instructions for determining a pose of the camera based on coordinates of the plurality of second corner points on the virtual plane and coordinates in the image,
The visual marker includes a predetermined pattern for guiding or indicating an object or object located in a specific location or specific area,
Determining the pose of the camera based on the coordinates of the plurality of second corner points on the virtual plane and the coordinates in the image includes:
Using a PnP (Perspective n-Point) algorithm, determining the posture of the camera based on the coordinates of the plurality of second corner points on the virtual plane and the coordinates in the image,
The plurality of first corner points include at least four corner points,
The computing device of claim 1, wherein the plurality of second corner points includes at least five corner points.
According to clause 15,
Calculating homography based on the coordinates on the virtual plane of the plurality of first corner points and the coordinates in the image,
Computing device comprising calculating a matrix for converting coordinates on the virtual plane of the plurality of first corner points into coordinates in the image of the plurality of first corner points as the homography.
delete According to clause 15,
The pose of the camera includes a rotation and translation transformation matrix representing a transformation relationship between coordinates on the virtual plane of the plurality of second corner points and coordinates within the image.
According to clause 15,
The at least one program is,
Computing device further comprising instructions for adjusting the coordinates of the plurality of second corner points in the image so that the coordinates of the plurality of second corner points in the image match the coordinates of a corner on a visual marker in the image. .
According to clause 15,
The visual marker includes a first visual marker including at least four reference corner points and a second visual marker including at least one additional corner point.

Images