WO2021057582A1 - Procédé, dispositif et système de mise en correspondance d'images, d'imagerie en 3d et de reconnaissance de pose - Google Patents

Procédé, dispositif et système de mise en correspondance d'images, d'imagerie en 3d et de reconnaissance de pose Download PDF

Info

Publication number
WO2021057582A1
WO2021057582A1 PCT/CN2020/115736 CN2020115736W WO2021057582A1 WO 2021057582 A1 WO2021057582 A1 WO 2021057582A1 CN 2020115736 W CN2020115736 W CN 2020115736W WO 2021057582 A1 WO2021057582 A1 WO 2021057582A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
current
deformed
matching
image block
Prior art date
Application number
PCT/CN2020/115736
Other languages
English (en)
Chinese (zh)
Inventor
陈森淼
Original Assignee
鲁班嫡系机器人（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 鲁班嫡系机器人（深圳）有限公司 filed Critical 鲁班嫡系机器人（深圳）有限公司
Publication of WO2021057582A1 publication Critical patent/WO2021057582A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images

Definitions

  • the invention relates to the technical field of image matching, in particular to methods, devices and systems for image matching, 3D imaging and gesture recognition.
  • the control unit matches a single image collected by a single image sensor with a reference image, or matches two images collected by the left and right image sensors, and then performs 3D imaging or performs posture recognition of the target object according to the matching result.
  • the present invention provides a method, device and system for image matching, 3D imaging and gesture recognition.
  • the first aspect of the present invention provides an image matching method.
  • the image matching method includes:
  • Detect the deformed area and/or non-deformed area in the initial image generate a matching result for the deformed area and/or generate a matching result for the non-deformed area.
  • the detecting the deformed area and/or the non-deformed area in the initial image includes:
  • the current image block is a current deformed image block or a current non-deformable image block; wherein the non-deformable area is a set of the current non-deformable image block; the deformed area Is the set of the current deformed image blocks.
  • the detecting the deformed area and/or the non-deformed area in the initial image includes:
  • the current image block is the current deformed image block; if yes, the current image block is the current non-deformed image block; wherein, the non-deformed area is a set of the current non-deformed image blocks; the deformed area Is the set of the current deformed image blocks.
  • the generating a matching result for the deformed region includes:
  • Matching is performed on the reference image to obtain a matching result for the deformed area.
  • the converting the deformed area into a reference image includes:
  • the current deformed image block is converted into the current reference image block.
  • the converting the deformed area into a reference image includes:
  • the deformed area is converted into a reference image based on Fourier transform.
  • the converting the deformed area into a reference image includes:
  • the current deformed image block is converted into a current reference image block.
  • the converting the deformed area into a reference image includes:
  • the generating a matching result for the deformed region includes:
  • Matching is performed on the image group to obtain a matching result.
  • the generating a matching result for the deformed region includes:
  • Matching is performed on the template image group to obtain a matching result.
  • the initial image is an image collected by an image sensor after an image is projected to the target object; wherein the projected image has a periodic gradual change pattern and is unique within a certain spatial range, or is within a certain spatial range It is unique.
  • the second aspect of the present invention provides a gesture recognition method, the gesture recognition method includes:
  • the posture information of the target object is generated.
  • a third aspect of the present invention provides a 3D imaging method, the 3D imaging method includes:
  • a fourth aspect of the present invention provides an image matching device, the image matching device includes:
  • Image acquisition module to acquire the initial image
  • the image matching module is used to detect the deformed area and/or non-deformed area in the initial image, generate a matching result for the deformed area and/or generate a matching result for the non-deformed area.
  • a fifth aspect of the present invention provides a gesture recognition device, and the gesture recognition device includes:
  • the posture generation module is used to generate posture information of the target object according to the matching result.
  • a sixth aspect of the present invention provides a 3D imaging device, the 3D imaging device includes:
  • the image generation module is used to generate a 3D image of the target object according to the matching result.
  • a seventh aspect of the present invention provides a system, which includes: an image projector, an image sensor, and a control unit;
  • the image projector is used to project an image to a target object
  • the image sensor is used to collect the initial image of the target object after the image is projected
  • the control unit is configured to implement the image matching method described in the first aspect; the gesture recognition method described in the second aspect; and/or the steps of the 3D imaging method described in the third aspect.
  • An eighth aspect of the present invention provides a computer device that includes a memory, a processor, and a computer program that is stored in the memory and can run on the processor.
  • the processor executes the computer program, Implement the steps of the image matching method described in the first aspect; the gesture recognition method described in the second aspect; and/or the 3D imaging method described in the third aspect.
  • a ninth aspect of the present invention provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image matching method described in the first aspect is implemented; The gesture recognition method described above; and/or the steps of the 3D imaging method described in the third aspect.
  • Fig. 1A is a first schematic structural diagram of an embodiment of a system provided by the present invention
  • Fig. 1B is a second schematic structural diagram of an embodiment of a system provided by the present invention
  • Fig. 1C is a third schematic structural diagram of an embodiment of the system provided by the present invention ;
  • FIG. 2 is a schematic flowchart of an embodiment of an image matching method provided by the present invention.
  • FIG. 3 is a schematic diagram of the first process of an embodiment of detecting a deformed area and/or a non-deformed area in an initial image provided by the present invention
  • FIG. 4 is a schematic diagram of a second process of an embodiment of detecting a deformed area and/or a non-deformed area in an initial image provided by the present invention
  • FIG. 5 is a schematic diagram of the first process of an embodiment of generating a matching result for a deformed area provided by the present invention
  • FIG. 6 is a schematic diagram of a second process of an embodiment of generating a matching result for a deformed area provided by the present invention.
  • FIG. 7 is a schematic diagram of a third process of an embodiment of generating a matching result for a deformed area provided by the present invention.
  • FIG. 8 is a schematic diagram of the first process of an embodiment of detecting that a current image block is a current deformed image block or a current non-deformable image block based on image features according to the present invention
  • FIG. 9 is a schematic diagram of a second process of an embodiment of detecting whether a current image block is a current deformed image block or a current non-deformable image block based on image characteristics according to the present invention.
  • FIG. 10 is a schematic diagram of the first process of an embodiment of converting a deformed area into a reference image provided by the present invention
  • FIG. 11 is a schematic diagram of the second process of an embodiment of converting a deformed area into a reference image provided by the present invention.
  • FIG. 12 is a schematic diagram of the third process of an embodiment of converting a deformed area into a reference image provided by the present invention.
  • FIG. 13 is a schematic flowchart of an embodiment of 3D imaging provided by the present invention.
  • FIG. 14 is a schematic flowchart of an embodiment of a gesture recognition method provided by the present invention.
  • FIG. 15 is a first structural block diagram of an embodiment of a matching device provided by the present invention.
  • FIG. 16 is a second structural block diagram of an embodiment of a matching device provided by the present invention.
  • FIG. 17 is a third structural block diagram of an embodiment of a matching device provided by the present invention.
  • FIG. 18 is a fourth structural block diagram of an embodiment of a matching device provided by the present invention.
  • FIG. 19 is a fifth structural block diagram of an embodiment of a matching device provided by the present invention.
  • FIG. 20 is a sixth structural block diagram of an embodiment of a matching device provided by the present invention.
  • 21 is a structural block diagram of an embodiment of a gesture recognition device provided by the present invention.
  • FIG. 22 is a structural block diagram of an embodiment of a 3D imaging device provided by the present invention.
  • Figure 23 is a structural block diagram of an embodiment of a computer device provided by the present invention.
  • FIG. 24 is a schematic diagram of an embodiment of image conversion provided by the present invention.
  • FIG. 25A is a first schematic diagram of an embodiment of an initial image provided by the present invention
  • FIG. 25B is a second schematic diagram of an embodiment of an initial image provided by the present invention
  • FIG. 25C is a third schematic diagram of an embodiment of an initial image provided by the present invention
  • Figure 25D is a fourth schematic diagram of an embodiment of an initial image provided by the present invention
  • Fig. 26A is a schematic diagram of an embodiment of a current preprocessed image block provided by the present invention
  • Fig. 26B is a schematic diagram of an embodiment of a current preprocessed image block after conversion provided by the present invention
  • FIG. 27 is a schematic diagram of intermediate results produced by the method for detecting deformed regions provided by the present invention.
  • an embodiment of the present invention provides a system, which includes an image sensor 11, an image projector 12, and a control unit 13;
  • the image projector 12 is used to project an image to a target object.
  • the image projector 12 can be used to project a periodic gradual and unique image within a certain spatial range to the target object.
  • the image projector 12 can be used to project a periodic gradual and unique image within a certain spatial range to the target object.
  • a periodic gradual and unique image within a certain spatial range to the target object.
  • the periodic gradual change rule may include, but is not limited to, periodically changing sine wave (or cosine wave, etc.) fringes.
  • the sine wave does not necessarily fully meet the sine wave standard, and may also be fringes close to sine.
  • incompletely regular sine fringes, or linearly changing sine fringes also called triangular waves.
  • being unique within a certain spatial range means that an image frame of a certain specification moves in any area of the image, and the image in the corresponding area within the image frame is always unique. For example: sine wave stripes with random patterns.
  • the image projector 12 can also be used to project a unique image within a certain spatial range, such as a scattered pattern, to the target object.
  • the image projector 12 can also project any other images that can meet the requirements of subsequent matching.
  • the image projector 12 may be a projector or a laser projector that can project the above-mentioned image that is currently or will be developed in the future.
  • the image sensor 11 is used to collect the initial image of the target object after the projected image.
  • the image sensor 11 sends the collected initial image to the control unit 13, the memory or the server, and so on.
  • the image sensor 11 may be, but is not limited to, a camera, a video camera, a scanner, or other devices with related functions (mobile phones, computers, etc.), and so on.
  • the image sensor 11 may be a group (as shown in FIG. 1A, 1B) or multiple groups (as shown in FIG. 1C) arranged around the target object; wherein, each group of image sensors may include one image sensor (as shown in FIG. 1A). (Shown), 2 image sensors (as shown in Figures 1B and 1C), or more than 2 image sensors (illustration omitted).
  • the image sensor 11 can be fixedly arranged relative to the target object, or can be movably arranged relative to the target object, which is not limited in this specific embodiment.
  • the control unit 13 respectively communicates with the image projector 11 and the image sensor 12 in a wired or wireless manner, and communicates with the image projector 11 and the image sensor 12.
  • Wireless methods may include, but are not limited to: 3G/4G, WIFI, Bluetooth, WiMAX, Zigbee, UWB (ultra wideband), and other wireless connection methods that are currently or will be developed in the future.
  • the control unit 13 may be a part of an independent computing device; it may also be a part of the image projector 11; and/or a part of the image sensor 12. In this specific embodiment, for convenience of description, the control unit 13 is a separate component Show.
  • control unit 13 can be Programmable Logic Controller (PLC), Field-Programmable Gate Array (FPGA), Computer (Personal Computer, PC), Industrial Control Computer (Industrial Personal Computer, IPC) , Smart phone, tablet or server, etc.
  • PLC Programmable Logic Controller
  • FPGA Field-Programmable Gate Array
  • PC Computer
  • IPC Industrial Control Computer
  • Smart phone Smart phone, tablet or server, etc.
  • the control unit generates program instructions according to a pre-fixed program, combined with manually input information, parameters, or data collected by an external image sensor.
  • the present invention provides an image matching method.
  • the image matching method includes the following method steps:
  • Step S110 acquiring an initial image
  • Step S120 detects the deformed area and/or the non-deformed area in the initial image, and generates a matching result for the deformed area and/or generates a matching result for the non-deformed area.
  • the deformed area and/or non-deformed area in the initial image are determined by detection, that is, the initial image may include both deformed areas and non-deformed areas at the same time, or all deformed areas, or all non-deformed areas.
  • the image matching method described in the following embodiment can be used to directly match the non-deformed area, and for the deformed area, the deformed area needs to be processed (further detailed description will be given in the following embodiment). Then, perform matching according to the image matching method described in the following embodiment.
  • Step S110 acquiring an initial image
  • a group of image sensors can be set, each group of image sensors includes 1 image sensor, the initial image is one, and a series of different distances are stored in advance. Download the reference image of the target object after the projected image collected by the image sensor, and subsequently match the initial image with a series of reference images.
  • a group of image sensors may be provided, each group of image sensors includes two image sensors 11, and each group of initial images includes a first initial image and a second initial image. Two images.
  • N groups of image sensors are set according to the above description, where N is an integer greater than or equal to 2, and each group of images is composed of two image sensors 11, that is, each group of initial The image includes the first initial image and the second initial image.
  • image splicing methods currently available or developed in the future can be used, such as: feature-based splicing and region-based splicing.
  • the region-based splicing method can start from the gray value of the image to be spliced, and calculate the gray value of an area of the image to be spliced with the same size in the reference image using other methods such as least squares. The difference is compared to the difference to determine the similarity of the overlapping area of the image to be spliced, so as to obtain the range and position of the overlapping area of the spliced image, and then realize image splicing.
  • the feature-based stitching is to derive the features of the image through pixels, and then use the image features as a standard to search and match the corresponding feature regions of the overlapping parts.
  • each group of initial images varies according to the number of image sensors.
  • the number of image sensors can be any number greater than or equal to 1. Since the imaging mode of more than 2 image sensors is also a combination of any two of them, the imaging method is The corresponding imaging methods under two image sensors are repeated. Therefore, this specific embodiment only uses a single group of image sensors, and the group of image sensors includes one image sensor or two image sensors as an example for description.
  • Step S120 detects the deformed area and/or the non-deformed area in the initial image, and generates a matching result for the non-deformed area and/or generates a matching result for the deformed area, respectively.
  • the deformed area refers to some areas in the initial image that are inconsistent with the reference image.
  • the deformed area is due to the target object (1 image sensor as an example) or part of the surface on the target object (2 image sensors as an example) relative to the reference surface (taking 2 image sensors as an example, the reference surface can be parallel to the image sensor)
  • the surface of the chip; taking an image sensor as an example, the reference surface can refer to the deflection, bending, etc.
  • the inconsistency may be manifested as: the deformation area is deflected, stretched, compressed, and/or bent relative to the reference image.
  • the sine wave image in the deformed area is deflected by 2 left (as shown in Fig. 25A), right deflection (as shown in Fig. 25B), and the sine wave in the deformed area Stretching (as shown in FIG. 25C), compression (as shown in FIG. 25D), or bending (illustration omitted), etc.
  • the non-deformed area refers to the area in the initial image that is consistent with the reference image.
  • the image matching method may include but is not limited to the following method steps:
  • the matching is performed between the two initial images.
  • the initial image includes a first initial image and a second initial image.
  • a fixed-size image frame is usually set with the matched pixel as the center. Match the image blocks in the image frame.
  • An n*n image block in the first initial image will be compared with the N image blocks of the same size in the second initial image along the epipolar direction of the two image sensors (N is one of the parallaxes of the two images Search scope).
  • the method of comparison is to calculate the absolute value of the brightness difference of the corresponding pixels of the two image blocks, and then sum the absolute value to obtain a matching score.
  • N matching scores can be obtained, and the minimum value of these N matching scores can be obtained, and the pixel point in the second initial image corresponding to this minimum value is the matched pixel point in the first initial image Correspondingly, and so on, so as to obtain the matching result of multiple pixels in the two images corresponding to each other.
  • one image sensor is taken as an example.
  • the above embodiment is to match between the first initial image and the second initial image, and in the embodiment of one image sensor, the initial The image is matched with the reference image, and the specific matching method can be referred to the above embodiment, which will not be repeated here.
  • step S120 may include the following method steps:
  • Step S121 acquiring the current image block in the initial image
  • the current image block is the image block for current matching.
  • Step S123 based on the image characteristics of the current image block, detect that the current image block is the current deformed image block or the current non-deformable image block;
  • the initial image includes a deformed area and/or a non-deformed area.
  • the deformed area refers to the current set of deformed image blocks;
  • the non-deformed area refers to the current set of non-deformed image blocks.
  • the deformed area can be detected by judging whether these characteristics are deformed, and then the deformed area can be converted to the image generated when the corresponding reference surface is generated.
  • the image includes the characteristic part of the fringe with a sinusoidal gradual change.
  • the sinusoidal fringe is parallel to the reference plane, the horizontal X is the repeated change of multiple cycles, and the vertical Y is aligned, which is Upright state.
  • the period of the stripe in the horizontal X direction may be stretched or compressed, and/or the vertical Y direction may be inclined, and the current image block may be deformed according to the changes of these characteristic parts in the image.
  • step S120 may include the following method steps:
  • Step S122 obtains the current image block in the initial image
  • Step S124 match the current image block; determine whether a matching result can be generated
  • step S126 use the current image block as the current deformed image block
  • step S128 use the current image block as the current non-deformable image block.
  • an image frame of a preset size is sequentially moved by a unit amount (e.g., the first initial image) along the initial image (e.g., the first initial image).
  • step S120 can be implemented through but not limited to the following method steps:
  • step S120 may include the following method steps:
  • Step S221 Obtain a deformed area
  • the deformed area may be a collection of multiple current deformed image blocks, so the current deformed image block can be obtained, and the current deformed image block can be converted into the current reference image block in the following step S222.
  • Step S222 Convert the deformed area into a reference image
  • Step S223 performs matching on the reference image to obtain a matching result for the deformed area.
  • the deformed area is replaced with a reference image, and the reference image is matched.
  • the matching method please refer to the relevant description in the image matching method in the above embodiment, which will not be repeated here.
  • step S120 may include the following method steps:
  • Step S321 obtains a template image group in which the pre-generated deformation area has a unit deformation in sequence
  • the template image group may be a pre-generated image template group obtained after the current deformed image block sequentially undergoes at least one unit deformation amount.
  • a current template image group is generated in advance corresponding to each current image block.
  • each template corresponds to the image block
  • the template image can be generated in advance according to the deflection unit, combined with the spatial position conversion information of the image projector and the image sensor; or according to the method of fitting function in the embodiment shown in FIG. 11, the corresponding Template image.
  • Step S322 performs matching on the template image group to obtain a matching result.
  • the template image corresponding to the current image block in the first initial image is sequentially obtained (for example: deflection -60, -40, -20, 0, 20, 40, and 60 template images), the corresponding second initial image is located on the same epipolar line, and it can include multiple image blocks to be matched.
  • Each template image and the second initial The sum of the absolute values of the gray-scale differences between the image blocks to be matched in the image corresponds to the second initial image with the smallest sum of the absolute values of the gray-scale differences and the current image block of the first initial image.
  • the pixels corresponding to the two matching image blocks are matched pixels, and the matching result is obtained.
  • the initial image and the reference image can be matched according to the method described in the above embodiment.
  • the reference image can be regarded as the second initial image to generate the matching result.
  • the template image group can be generated in advance according to the deflection unit, combined with the spatial position conversion information of the image projector and the image sensor; or according to the method of fitting function in the embodiment shown in FIG. 11, the corresponding Set of template images.
  • step S120 may include the following method steps:
  • Step S421 sequentially generates image groups after unit deformation of the deformed area occurs
  • Step S422 performs matching on the image group to obtain a matching result.
  • the deformed image block group formed after the current image block in the first initial image is deflected by a unit angle and the plurality of waiting images located on the same epipolar line in the second initial image are sequentially acquired.
  • the matching image blocks are matched, and the sum of the absolute values of the gray-scale differences between each deformed image block and the image block to be matched in the second initial image is calculated, and then the second one with the smallest sum of the absolute values of the gray-scale differences is calculated.
  • the to-be-matched image block of the initial image and the current image block of the first initial image are matched image blocks, so the pixels corresponding to the two matched image blocks are matched pixels, thereby obtaining the matching result.
  • the image group after unit deformation can be generated according to the deflection unit, combined with the spatial position conversion information of the image projector and the image sensor; it can also be based on the fitting function in the embodiment shown in FIG. 9 below. Method to generate the corresponding image group.
  • step S123 based on the image characteristics of the current image block, detecting that the current image block is the current deformed image block or the current non-deformable image block can be implemented by, but not limited to, the following method steps:
  • step S123 may include the following method steps:
  • the image frame can be moved along the epipolar direction of the image sensor on the initial image, and each time it moves one unit, the current image on the initial image corresponding to the image frame is the current image block.
  • the current deformed image block when it is necessary to further convert the current deformed image block into a reference image based on the detection method, there may be a blank part on the edge of the current deformed image block after the conversion (as shown in FIG. 26B As shown), in order to ensure the integrity of the content of the current image block after conversion, it is usually necessary to obtain a current image block whose size is larger than the actual matching image block (as shown in Figure 26A), and then perform the conversion. The latter image is cropped to obtain the current deformed image block completely converted into the reference image (as shown in FIG. 26C).
  • the peak or trough of each sine wave should be on the same reference line.
  • the pixel point at the peak or valley is the maximum value pixel point, for example, the gray value is the highest or the lowest.
  • S1235 fits the most value pixel points in each row to obtain the current fitting line of the current image block
  • S1237 detects the current image shape variable of the current fitted line relative to the reference line
  • a threshold can be preset. If the deformation is less than When it is equal to or less than the threshold (in one embodiment, the threshold may also be a theoretical value of zero), the current image block is considered to be the current non-deformed image block; if it is greater than or equal to a certain threshold, the current image block is determined Is the current deformed image block, and the shape variable of the fitted line relative to the reference line is the current image shape variable of the current image block.
  • step S123 may include the following method steps:
  • the current image block is fitted to obtain a fitting function.
  • A is the amplitude of the sine function;
  • 360/B is the period of the function's brightness change (unit: Pixels),
  • C represents the inclination of the pixel in the Y direction.
  • B pixels the inclination angle is arccotangent (C/B);
  • D is the translation amount of the function in the lateral X direction;
  • E is the translation amount of the function in the Z direction; where A, D and E are fixed values.
  • C represents the inclination of the pixel in the Y direction.
  • C is 0;
  • arccotangent is the image shape variable of the current deformed image block.
  • it can be determined whether the image deformation amount is greater than or equal to a certain threshold; if so, the current image block is a deformed image block; if not, the current image block is a non-deformable image block.
  • step S222 converting the deformed area into a reference image can be achieved through but not limited to the following method steps:
  • step S222 may include the following method steps:
  • Step S2221 obtains the current deformed image block
  • Step S2222 extracts the most value pixel points in each row of the current deformed image block
  • Step S2223 fits the most value pixel points in each row to obtain the current fitted line of the current image block
  • Step S2224 detects the current image shape variable of the current fitting line relative to the reference line
  • step S2221 to step S2224 can refer to the description of step S1231 to step S1237 in step S123 in the above embodiment.
  • steps S2221-step S2224 here can be omitted.
  • Step S2225 converts the current deformed image block into the current reference image block based on the current image shape variable.
  • step S222 may include the following method steps:
  • Step S2231 obtains the current image block
  • Step S2232 performs fitting on the current image block to obtain the current fitting function
  • step S2231 to step S2232 can be found in the description of step S1232 to step S1234 in the above embodiment.
  • steps S2231-step S2232 here can be omitted.
  • Step S2233 converts the current deformed image block into the current reference image block based on the current fitting function
  • the deformed area can be converted based on the fitting function, that is, the period of the deformed area is the reference period, and the C of the image is 0, so as to obtain the deformed area after conversion.
  • the size of the image block to be acquired is (width W: 33, height H: 33), since there will be blanks at the edges of the converted image (as shown in Figure 26B), in order to ensure the integrity of the image block content, it is usually necessary to intercept An initial image block larger than the size of the image block, for example, as shown in Figure 26A, a wider area is selected as the initial image block (W: 63, H: 33), where the center of the initial image block Form a rectangular frame, the size of the rectangular frame corresponds to the size of the image block (W: 33, H: 33); in addition, the initial image block can also be any other size, as long as it is larger than the preset image block And ensure that the content in the converted image block is complete.
  • the initial image block is fitted with a three-dimensional curve according to the above fitting function.
  • the transformation method is as follows: Since the height of the image is H: 33, take the middle row (that is, the 17th row) without transformation.
  • the middle row 17 is used as the 0 point reference, and the number of other rows is i, and the i-th row is translated along the positive direction of the X axis by -(i-17)*(C/B) pixels. After all rows are traversed, the transformation is completed. such as:
  • the initial image is converted, and the converted initial image becomes as shown in FIG. 26B.
  • the area within the rectangular frame (W: 33, H: 33) located in the middle of the deformed initial image is intercepted as the converted image block, and the converted image block after the interception is shown in FIG. 26C.
  • step S222 may include the following method steps:
  • Step S2241 obtains the current deformed image block
  • Step S2242 generates the current deformation of the target object based on the current deformation image block
  • Step S2243 based on the current deformation amount, converts the current deformed image block into the current reference image block;
  • the actual spatial position coordinates of point A in the image corresponding to the image sensor coordinate system can be obtained according to the initial image; according to the deformation of point A relative to the reference plane L'(for example: the deformation can be implemented according to the above
  • the image shape variable obtained in the example is converted based on the calibration result of the image sensor; or obtained based on the template image group described in the above embodiment)
  • the point A corresponding to the direction of the projected image of the image projector 12 can be calculated
  • the position coordinates of the virtual point A'projected on the reference plane O in the image sensor coordinate system According to the conversion between the first and second image sensor coordinate systems and the first and second image coordinate systems, the positions A'can be obtained respectively The coordinates in the first and second image coordinate systems, and so on, can obtain the corrected image of the deformed area.
  • step S222 may be based on Fourier transform to convert the deformed area into a reference image.
  • a 3D imaging method is further provided, and the 3D imaging method includes the matching method described in the above embodiment, and further includes the steps:
  • S130 generates a 3D image of the target object according to the matching result.
  • the posture information of the corresponding point of the matching point pair in the three-dimensional space can be obtained, based on the multiple matching point pairs included in the matching result , That is, you can draw a 3D image of the object in a three-dimensional space.
  • the posture information can be 3d coordinates in a preset coordinate system for the target, and the motion of a rigid body in a 3-dimensional space can be described by 3d coordinates (a total of 6 degrees of freedom). Specifically, it can be divided into rotation and translation, each It is 3 degrees of freedom.
  • the translation of a rigid body in a 3-dimensional space is an ordinary linear transformation, and a 3x1 vector can be used to describe the translation position; and the commonly used description methods of rotation pose include but are not limited to: rotation matrix, rotation vector, quaternion, Euler angle and Lie algebra.
  • a gesture recognition method is also provided.
  • the gesture recognition method includes the matching method described in the above embodiment, and further includes the steps:
  • S140 generates a posture recognition result of the target object according to the matching result.
  • each matching point pair is based on the triangulation algorithm or based on the corresponding reference image, and the posture information of the corresponding point in the three-dimensional space of the matching point pair can be obtained, based on the posture of one or more three-dimensional space points Information to obtain the result of the posture recognition of the target object.
  • the result of the posture recognition of the target object may be the posture information representing the position and posture of the entire object or the posture information of a certain target position associated with the target object (for example, the target position may be located on the target object or on the including frame of the target object).
  • the posture information of the target object can be obtained.
  • the Linemod method refers to storing the corresponding point cloud images at multiple angles based on the 3D model of the target object (such as: CAD model) in advance, and matching the point cloud image obtained in the above embodiment with the image in the image library to determine The posture information of the target object.
  • an image matching device is provided, and the image matching device includes:
  • the image acquisition module 110 is used to acquire an initial image
  • the result generating module 120 is configured to detect the deformed area and/or the non-deformed area in the initial image, generate a matching result for the deformed area and/or generate a matching result for the non-deformed area.
  • the above result generation module 120 includes:
  • the image acquisition unit 121 is configured to acquire the current image block in the initial image
  • the image detection unit 123 is configured to detect whether the current image block is a current deformed image block or a current non-deformable image block based on the image characteristics of the current image block;
  • the above result generation module 120 includes:
  • the image acquisition unit 122 is configured to acquire the current image block in the initial image
  • the image matching unit 124 is used to match the current image block, and the result judgment unit 126 is used to judge whether a matching result can be generated;
  • the image determining unit 128 is configured to, if not, the current image block is the current deformed image block; if so, the current image block is the current non-deformable image block.
  • the above result generation module 120 includes:
  • the deformation acquiring unit 221 is configured to acquire the deformation area
  • the deformation conversion unit 222 is configured to convert the deformation area into a reference image
  • the reference matching unit 223 is configured to perform matching on the reference image to obtain a matching result for the deformed area.
  • the above result generating module 120 includes:
  • the template obtaining unit 321 is configured to obtain a template image group in which the pre-generated deformation area has a unit deformation in sequence;
  • the template matching unit 322 is configured to perform matching on the template image group to obtain a matching result.
  • the above result generation module 120 includes:
  • the image generating unit 421 is configured to sequentially generate image groups after unit deformation of the deformed area occurs;
  • the image matching unit 422 is configured to perform matching on the image group to obtain a matching result.
  • the image detection unit 123 includes:
  • the image acquisition unit 1231 is used to acquire the current image block
  • the maximum value extraction unit 1233 is used to extract the maximum value pixel points in each row of the current image block
  • the best value fitting unit 1235 is used to fit the best value pixel points in each row to obtain the current fitting line of the current image block;
  • the deformation detection unit 1237 is used to detect the current image deformation of the current fitting line relative to the reference line;
  • the image detection unit 1239 is configured to detect whether the current image block is the current deformed image block or the current non-deformable image block according to the current image deformation.
  • the image detection unit 123 includes:
  • the image acquisition unit 1232 acquires the current image block
  • the function extraction unit 1234 fits the current image block to obtain a fitting function
  • the image detection unit 1236 detects whether the current image block is the current deformed image block or the current non-deformed image block according to the fitting function.
  • the deformation conversion unit 222 includes:
  • the image acquiring unit 2221 is used to acquire the current deformed image block in the deformed area
  • the maximum value extraction unit 2222 is configured to extract the maximum value pixel points in each row of the current deformed image block
  • the best value fitting unit 2223 is configured to fit the best value pixel points in each row to obtain the current fitted line;
  • the deformation calculation unit 2224 is used to calculate the current image deformation of the current fitting line relative to the reference line;
  • the image conversion unit 2225 is configured to convert the current deformed image block into the current reference image block based on the current image shape variable.
  • the deformation conversion unit 222 includes:
  • the image acquiring unit 2231 is used to acquire the current deformed image block in the deformed area
  • the function extraction part 2232 is used for fitting the current deformed image block to obtain the current fitting function
  • the image conversion unit 2233 is configured to convert the current deformed image block into the current reference image block based on the current fitting function.
  • the deformation conversion unit 222 includes:
  • the image acquiring unit 2241 is used to acquire the current deformed image block of the deformed area
  • the deformation generating unit 2242 is configured to generate the current deformation of the target object based on the current deformed image block;
  • the image conversion unit 2243 is configured to convert the current deformed image block into the current reference image block based on the current deformation amount.
  • the deformation conversion unit 222 includes:
  • the image conversion unit 2251 is configured to convert the deformed area into a reference image based on Fourier transform.
  • a gesture recognition device As shown in FIG. 21, in one embodiment, a gesture recognition device is provided, and the gesture recognition device includes:
  • the posture generation module 130 is configured to generate posture information of the target object according to the matching result.
  • a 3D imaging device is provided, and the 3D imaging device includes:
  • the 3D imaging module 140 is configured to generate a 3D image of the target object according to the matching result.
  • each module in each of the above-mentioned devices may be implemented in whole or in part by software, hardware, and a combination thereof.
  • the above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.
  • a computer-readable storage medium stores a computer program, and the computer program implements the image matching method described in the above embodiments when the computer program is executed by a processor, Steps of 3D imaging method and/or gesture recognition method.
  • a computer device is also provided.
  • the computer device includes a memory, a processor, and a computer program that is stored in the memory and can run on the processor.
  • the processor executes the computer program, the steps of the image matching method, 3D imaging method, and/or gesture recognition method described in the above embodiments are implemented.
  • Industrial control computers have important computer attributes and characteristics. Therefore, they all have computer central processing unit (CPU), hard disk, memory and other internal memories, as well as plug-in hard disks. External storage such as Smart Media Card (SMC), Secure Digital (SD) Card, Flash Card (Flash Card), etc., with operating system, control network and protocol, computing power, and friendly man-machine interface, It is to provide reliable, embedded, intelligent computers and industrial control computers for other structures/equipment/systems.
  • SMC Smart Media Card
  • SD Secure Digital
  • Flash Card Flash Card
  • the computer program may be divided into one or more modules/units, and the one or more modules/units are stored in the memory and executed by the processor to complete the present invention.
  • the one or more modules/units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the control unit.
  • the so-called processor can be a central processing unit (Central Processing Unit, CPU), other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the memory may be a storage device built into the terminal, such as a hard disk or a memory.
  • the memory may also be an external storage device of the control unit, such as a plug-in hard disk equipped on the control unit, a Smart Media Card (SMC), a Secure Digital (SD) card, and a flash memory. Card (Flash Card), etc.
  • the memory may also include not only an internal storage unit of the control unit, but also an external storage device.
  • the memory is used to store the computer program and other programs and data required by the terminal.
  • the memory can also be used to temporarily store data that has been output or will be output.
  • FIG. 23 is only an example of a computer device, and does not constitute a limitation on the computer device. It may include more or less components than shown in the figure, or a combination of certain components, or different components, such as
  • the control unit may also include input and output devices, network access devices, buses, and the like.
  • the disclosed devices and methods can be implemented in other ways.
  • the embodiments of each device described above are only illustrative.
  • the division of the modules or units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or Components can be combined or integrated into another system, or some features can be omitted or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
  • the integrated module/unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the present invention implements all or part of the processes in the above-mentioned embodiments and methods, and can also be completed by instructing relevant hardware through a computer program.
  • the computer program can be stored in a computer-readable storage medium. When the program is executed by the processor, it can implement the steps of the foregoing method embodiments. .
  • the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file, or some intermediate forms.
  • the computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electrical carrier signal, telecommunications signal, and software distribution media, etc.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • electrical carrier signal telecommunications signal
  • software distribution media etc.
  • the content contained in the computer-readable medium can be appropriately added or deleted according to the requirements of the legislation and patent practice in the jurisdiction.
  • the computer-readable medium Does not include electrical carrier signals and telecommunication signals.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne un procédé, un dispositif et un système de mise en correspondance d'images, d'imagerie en 3D et de reconnaissance de pose. Le procédé de mise en correspondance d'image consiste à : acquérir une image initiale ; et effectuer une détection sur l'image initiale pour obtenir une région déformée et/ou une région non déformée, et générer un résultat de mise en correspondance pour la région déformée et/ou un résultat de mise en correspondance pour la région non déformée. Dans la solution technique de la présente invention, la détection est effectuée sur une image initiale pour obtenir une région déformée et/ou une région non déformée, puis un schéma correspondant est utilisé pour effectuer une mise en correspondance pour la région déformée et/ou la région non déformée, de telle sorte qu'un résultat de mise en correspondance d'images précis peut être obtenu même lorsqu'un objet cible est décalé par rapport à des plans de référence respectifs.
PCT/CN2020/115736 2019-09-27 2020-09-17 Procédé, dispositif et système de mise en correspondance d'images, d'imagerie en 3d et de reconnaissance de pose WO2021057582A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910926826.3 2019-09-27
CN201910926826.3A CN112581512A (zh) 2019-09-27 2019-09-27 图像匹配、3d成像及姿态识别方法、装置及系统

Publications (1)

Publication Number Publication Date
WO2021057582A1 true WO2021057582A1 (fr) 2021-04-01

Family

ID=75110514

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/115736 WO2021057582A1 (fr) 2019-09-27 2020-09-17 Procédé, dispositif et système de mise en correspondance d'images, d'imagerie en 3d et de reconnaissance de pose

Country Status (2)

Country Link
CN (1) CN112581512A (fr)
WO (1) WO2021057582A1 (fr)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08305854A (ja) * 1995-05-01 1996-11-22 Nippon Telegr & Teleph Corp <Ntt> 射影パラメータ抽出処理方法
CN1489382A (zh) * 2002-07-23 2004-04-14 日本电气视象技术株式会社 投影仪
CN102393398A (zh) * 2002-09-30 2012-03-28 应用材料以色列公司 用于光学检测的照射系统
CN108305218A (zh) * 2017-12-29 2018-07-20 努比亚技术有限公司 全景图像处理方法、终端及计算机可读存储介质
CN108391145A (zh) * 2017-02-20 2018-08-10 南安市耀森智能设备有限公司 一种机器人
CN110139033A (zh) * 2019-05-13 2019-08-16 Oppo广东移动通信有限公司 拍照控制方法及相关产品

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2787453B2 (ja) * 1988-10-25 1998-08-20 大日本印刷株式会社 絵柄フィルムの絵柄割付領域決定方法
CN101198964A (zh) * 2005-01-07 2008-06-11 格斯图尔泰克股份有限公司 使用红外图案照射创建对象的三维图像
JP6251489B2 (ja) * 2013-03-28 2017-12-20 株式会社 資生堂 画像解析装置、画像解析方法、及び画像解析プログラム
AU2015202937A1 (en) * 2015-05-29 2016-12-15 Canon Kabushiki Kaisha Systems and methods for registration of images
CN105956997B (zh) * 2016-04-27 2019-07-05 腾讯科技（深圳）有限公司 图像形变处理的方法和装置
WO2018050223A1 (fr) * 2016-09-14 2018-03-22 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Détection de motifs
CN108133492B (zh) * 2016-12-01 2022-04-26 京东方科技集团股份有限公司 图像匹配方法、装置和系统
CN109427046B (zh) * 2017-08-30 2021-07-20 深圳中科飞测科技股份有限公司 三维测量的畸变校正方法、装置及计算机可读存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08305854A (ja) * 1995-05-01 1996-11-22 Nippon Telegr & Teleph Corp <Ntt> 射影パラメータ抽出処理方法
CN1489382A (zh) * 2002-07-23 2004-04-14 日本电气视象技术株式会社 投影仪
CN102393398A (zh) * 2002-09-30 2012-03-28 应用材料以色列公司 用于光学检测的照射系统
CN108391145A (zh) * 2017-02-20 2018-08-10 南安市耀森智能设备有限公司 一种机器人
CN108305218A (zh) * 2017-12-29 2018-07-20 努比亚技术有限公司 全景图像处理方法、终端及计算机可读存储介质
CN110139033A (zh) * 2019-05-13 2019-08-16 Oppo广东移动通信有限公司 拍照控制方法及相关产品

Also Published As

Publication number Publication date
CN112581512A (zh) 2021-03-30

Similar Documents

Publication Publication Date Title
EP3614340B1 (fr) Procédés et dispositifs d&#39;acquisition de visage 3d et support d&#39;enregistrement lisible par ordinateur
US10872227B2 (en) Automatic object recognition method and system thereof, shopping device and storage medium
US10008005B2 (en) Measurement system and method for measuring multi-dimensions
US9135710B2 (en) Depth map stereo correspondence techniques
US10726580B2 (en) Method and device for calibration
US10810718B2 (en) Method and device for three-dimensional reconstruction
US10455219B2 (en) Stereo correspondence and depth sensors
CN105335748B (zh) 图像特征提取方法和系统
EP3135033B1 (fr) Stéréo structurée
KR20200044676A (ko) 활성 깊이 센싱을 위한 방법 및 장치 그리고 이의 교정방법
WO2012096747A1 (fr) Établissement de cartes télémétriques à l&#39;aide de motifs d&#39;éclairage périodiques
US11468609B2 (en) Methods and apparatus for generating point cloud histograms
CN113362445B (zh) 基于点云数据重建对象的方法及装置
CN116129037B (zh) 视触觉传感器及其三维重建方法、系统、设备及存储介质
CN109375833B (zh) 一种触控指令的生成方法及设备
Wei et al. An accurate stereo matching method based on color segments and edges
Yin et al. Estimation of the fundamental matrix from uncalibrated stereo hand images for 3D hand gesture recognition
CN112197708B (zh) 测量方法及装置、电子设备及存储介质
JP7298687B2 (ja) 物体認識装置及び物体認識方法
CN116912417A (zh) 基于人脸三维重建的纹理贴图方法、装置、设备和存储介质
WO2021057582A1 (fr) Procédé, dispositif et système de mise en correspondance d&#39;images, d&#39;imagerie en 3d et de reconnaissance de pose
JP2021021577A (ja) 画像処理装置及び画像処理方法
JP2017162251A (ja) ３次元非接触入力装置
CN106651940B (zh) 一种用于3d交互的专用处理器
Hu et al. Active shape reconstruction using a novel visuotactile palm sensor

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20869735

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20869735

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 29.09.2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20869735

Country of ref document: EP

Kind code of ref document: A1