WO2023103559A1 - 三维识别装置、终端、标定方法、存储介质 - Google Patents

三维识别装置、终端、标定方法、存储介质 Download PDF

Info

Publication number
WO2023103559A1
WO2023103559A1 PCT/CN2022/123544 CN2022123544W WO2023103559A1 WO 2023103559 A1 WO2023103559 A1 WO 2023103559A1 CN 2022123544 W CN2022123544 W CN 2022123544W WO 2023103559 A1 WO2023103559 A1 WO 2023103559A1
Authority
WO
WIPO (PCT)
Prior art keywords
infrared
lens
camera
image set
rgb
Prior art date
Application number
PCT/CN2022/123544
Other languages
English (en)
French (fr)
Inventor
张永亮
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Priority to EP22902984.8A priority Critical patent/EP4436219A1/en
Publication of WO2023103559A1 publication Critical patent/WO2023103559A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/02—Services making use of location information
    • H04W4/029—Location-based management or tracking services

Definitions

  • the present application relates to but is not limited to the field of intelligent terminals, and in particular relates to a three-dimensional recognition device, a terminal, a calibration method, and a storage medium.
  • Embodiments of the present application provide a three-dimensional recognition device, a terminal, a calibration method, and a storage medium.
  • the embodiment of the present application provides a three-dimensional identification device, which is arranged inside the display screen of the terminal.
  • the three-dimensional identification device includes: an RGB component, the RGB component includes an RGB camera and a first lens, and the first A first optical channel is connected between the lens and the RGB camera; an infrared emission assembly, the infrared emission assembly includes an infrared floodlight illuminator and a second lens, between the infrared floodlight illuminator and the second lens A second optical channel is connected, the second lens is adjacent to the first lens; an infrared receiving assembly, the infrared receiving assembly includes a first infrared camera and a third lens, the first infrared camera and the first infrared camera A third optical channel is connected between the three lenses, and the third lens is adjacent to the first lens.
  • an embodiment of the present application provides a terminal, including: the three-dimensional identification device as described in the first aspect; a display screen, the three-dimensional identification device is arranged inside the display screen, and the display screen and the The area corresponding to the lens of the three-dimensional identification device is the area with enhanced light transmittance.
  • the embodiment of the present application provides a calibration method, which is applied to a three-dimensional recognition device, and the three-dimensional recognition device includes an RGB component, an infrared emitting component, and an infrared receiving component, wherein the RGB component includes an RGB camera and a first A lens, a first optical channel is connected between the first lens and the RGB camera head, the infrared emitting assembly includes an infrared floodlight illuminator and a second lens, and the infrared floodlight illuminator and the second lens A second optical channel is connected between them, the second lens is adjacent to the first lens, the infrared receiving assembly includes a first infrared camera and a third lens, and the first infrared camera and the third lens A third optical channel is connected therebetween, and the third lens is adjacent to the first lens.
  • the RGB component includes an RGB camera and a first A lens
  • a first optical channel is connected between the first lens and the RGB camera head
  • the infrared emitting assembly includes an infrared flood
  • the calibration method includes: when the infrared flood illuminator is in a working state, according to a first preset time sequence, the first image set is taken by the first infrared camera, and the second image set is taken by the RGB camera.
  • Image set wherein, the target objects captured by the first infrared camera at different times are different, the target objects captured by the RGB camera at different times are different, and the target objects captured by the first infrared camera and the RGB camera at the same time The objects are the same; perform the calibration between the first infrared camera and the infrared floodlight irradiator according to the first image set; fuse the first image set and the second image set respectively captured at the same time images to obtain a first fused image set, and perform calibration between the RGB camera and the first infrared camera according to the first fused image set.
  • an embodiment of the present application provides a terminal, including: a memory, a processor, and a computer program stored in the memory and operable on the processor, when the processor executes the computer program, the third The calibration method described in the aspect.
  • the embodiment of the present application provides a computer-readable storage medium, storing computer-executable instructions, and the computer-executable instructions are used to execute the calibration method as described in the third aspect.
  • FIG. 1 is a layout of a full-screen three-dimensional recognition device and an RGB camera in the prior art
  • FIG. 2 is a schematic diagram of a three-dimensional recognition device set in a terminal according to an embodiment of the present application
  • Fig. 3 is a schematic cross-sectional view of the three-dimensional recognition device provided by the present application.
  • Fig. 4 is a front view of the three-dimensional recognition device provided by the present application.
  • FIG. 5 is a schematic layout diagram of Embodiment 1 of the present application.
  • FIG. 6 is a schematic layout diagram of Embodiment 2 of the present application.
  • FIG. 7 is a schematic layout diagram of Embodiment 3 of the present application.
  • FIG. 8 is a flow chart of the calibration method applied to the three-dimensional recognition device in Embodiment 1 provided by the present application;
  • FIG. 9 is a flow chart of the calibration method applied to the three-dimensional recognition device in Embodiment 2 provided by the present application.
  • Fig. 10 is a flow chart of the calibration method applied to the three-dimensional recognition device of the third embodiment provided by the present application.
  • Fig. 11 is a flowchart of filtering laser speckle information provided by another embodiment of the present application.
  • Fig. 12 is a flowchart of global calibration provided by another embodiment of the present application.
  • Fig. 13 is an example diagram of a target object provided by the present application.
  • Fig. 14 is an apparatus diagram of a terminal provided by another embodiment of the present application.
  • the present application provides a three-dimensional recognition device, a terminal, a calibration method, and a storage medium.
  • the three-dimensional recognition device includes: an RGB component, the RGB component includes an RGB camera and a first lens, and the distance between the first lens and the RGB camera is Connected with a first optical channel; an infrared emitting assembly, the infrared emitting assembly includes an infrared flood illuminator and a second lens, a second optical channel is connected between the infrared flood illuminator and the second lens, the The second lens is adjacent to the first lens; an infrared receiving assembly, the infrared receiving assembly includes a first infrared camera and a third lens, and a third infrared camera is connected between the first infrared camera and the third lens In the optical channel, the third lens is adjacent to the first lens.
  • the RGB camera, the infrared floodlight irradiator and the first infrared camera can be arranged separately from the lens, and the transmission of light can be realized through the optical channel, so that the layout of multiple lenses under the screen can be more compact, thereby effectively Reduce the area of the area that is specially treated to improve light transmittance, effectively improve the display effect of the full screen, and improve user experience.
  • the embodiment of the present application provides a three-dimensional identification device, which is arranged inside the display screen of the terminal, and the three-dimensional identification device includes:
  • the RGB component includes an RGB camera 210 and a first lens 212, and the first lens 212 is connected to the RGB camera 210 through a first optical channel 211;
  • the infrared emitting assembly includes an infrared floodlight illuminator 310 and a second lens 312, a second optical channel 311 is connected between the infrared floodlight illuminator 310 and the second lens 312, the second lens 312 and the first lens 211 Adjacent;
  • the infrared receiving assembly includes a first infrared camera 410 and a third lens 412, a third optical channel 411 is connected between the first infrared camera 410 and the third lens 412, and the third lens 412 is adjacent to the first lens 212 .
  • FIG. 1 the layout of a common 3D recognition device is shown in FIG. 1.
  • the 3D recognition devices are arranged side by side on the display screen 110 of the terminal 10. Since the body size of the 3D recognition device is relatively large, there is a certain distance between each lens. The distance results in a larger area of the light transmittance enhanced region 120, which affects the display experience of the full screen.
  • the three-dimensional identification device of this embodiment can realize the separation of the main body and the lens of the three-dimensional identification device. In the manner shown in FIG. 2, only the size of the lens can be considered for a compact layout. The transmission of light through the light channel can effectively reduce the area of the light transmittance enhanced region 120 and improve user experience.
  • the size of the first lens 212 of the RGB camera 210 is large, and the size of the lenses of the infrared emitting component and the infrared receiving component is relatively small, usually a small-aperture lens. Therefore, as shown in FIG. 2 , the first lens 212 Surrounded by several small-aperture lenses, the compactness of the layout is further improved.
  • the small-aperture lenses are the second lens 312 and the third lens 412. Since the RGB camera 210 requires a large amount of light, the apertures of the second lens 312 and the third lens 412 are compared to The aperture of the first lens 212 is relatively small, therefore, arranging the second lens 312 and the third lens 412 adjacent to the first lens 212 will not interfere with the body of the RGB camera 210 .
  • the apertures of the second lens 312 and the third lens 412 can be adjusted according to the actual light input requirements, and the specific sizes are not limited here.
  • the first optical channel 211 , the second optical channel 311 and the third optical channel 411 can be in any shape, as long as they can ensure that light can enter or exit, and there is no limitation here.
  • FIG. 3 is a cross-sectional view taken along the vertical direction when the terminal is placed horizontally.
  • the RGB lens 210 is connected to the first lens 212 through the first optical channel 211, there is a certain distance between the RGB lens 210 and the first lens 212.
  • the second lens 312 and the third lens 412 It can be arranged adjacent to the first lens 212 in the manner shown in FIG. 5 , that is, it surrounds the first lens 212.
  • the specific position can be adjusted according to actual needs, as long as there is no interference between them, which is not the case in this embodiment. More limited.
  • the body size of the RGB camera 210, the infrared floodlight irradiator 310 and the first infrared camera 410 is larger than the corresponding lens, if the optical channels are all linear channels, in order to realize the lens layout shown in Figure 5, It is necessary to set a longer first optical channel 211 to ensure that the infrared flood illuminator 310 and the first infrared camera 410 do not interfere with the RGB camera 210, which will lead to an increase in the thickness of the terminal. Therefore, the second optical channel in this embodiment 311 and the third optical channel 411 both adopt the right-angle structure shown in FIG.
  • the reflector changes the propagation direction of the infrared light, so that the infrared flood irradiator 310 and the first infrared camera 410 can be laterally staggered from the RGB camera 210, and the internal layout obtained can refer to the front view shown in Figure 4, which effectively reduces the number of three-dimensional identification devices
  • the required longitudinal space can effectively reduce the thickness of the three-dimensional identification device.
  • three-dimensional recognition can be realized by using Time of Flight (TOF).
  • TOF Time of Flight
  • VCSEL Surface Emitting Laser
  • the inside of the first infrared camera 410 can also be adjusted according to actual needs, and it only needs to be able to realize accurate collection of TOF, and this embodiment does not limit the internal structure of the device.
  • the infrared emitting assembly of this embodiment also includes an infrared dot matrix projector 320 and a fourth lens 322, and a fourth optical channel 321 is connected between the infrared dot matrix projector 320 and the fourth lens 322,
  • the fourth lens 322 is adjacent to the first lens 212;
  • a third reflector (not shown) is also arranged in the fourth optical channel 321, and the third reflector is used for infrared dot matrix projector 320 emission The light is reflected to the fourth lens.
  • the setting principle of the infrared dot matrix projector 320, the fourth lens 321, the fourth light channel 321 and the third reflector can refer to the infrared flood illuminator 310, the second lens 312, the second light channel 311 and the first reflector 313 , will not be repeated here for the sake of brevity.
  • the fourth lens 322, the second lens 312, and the third lens 412 are arranged around the first lens 311, and the specific positions can be adjusted according to actual needs, which is not discussed in this embodiment. More limited.
  • the infrared flood irradiator 310 uses a high-power VCSEL, while the structure of this embodiment adopts a monocular structured light scheme, so the infrared flood irradiator 310 can use a relatively Low-power VCSEL, infrared dot matrix projector 320 adopts high-power VCSEL to transmit laser speckle, and the structure of the first infrared camera 410 can also be simplified compared with Embodiment 1, and the internal structure of the device is not described here. Too much to repeat.
  • the infrared receiving assembly of this embodiment includes a second infrared camera 420 and a fifth lens 422, a fifth optical channel 421 is connected between the second infrared camera 420 and the fifth lens 422, and the fifth lens 422 Adjacent to the first lens 212, in addition, a fourth reflector (not shown in the figure) is also arranged in the fifth optical channel 421, and the fourth reflector is used to reflect the light incident through the fifth lens 422 to the second Infrared camera 420.
  • the setting principles of the second infrared camera 420, the fifth lens 422, the fifth optical channel 421 and the fourth reflector can refer to the first infrared camera 410, the third lens 412, the third optical channel 411 and the second reflector, in order to describe It is convenient and will not be repeated here.
  • the fifth lens 422 , the fourth lens 322 , the second lens 312 and the third lens 412 are arranged around the first lens 311 , and the specific positions can be adjusted according to actual needs. The embodiment does not limit this much.
  • the second embodiment can realize three-dimensional recognition of monocular structured light, and this embodiment adds a second infrared camera 420, so it can realize three-dimensional recognition of binocular structured light, and the specific device parameters are selected according to actual needs. Yes, no more limitations here.
  • the main body of the identification device can be set in the internal space of the terminal, effectively reducing the area of the enhanced light transmittance area 120, thereby reducing the display area affected by the enhanced light transmittance area 120, and improving The overall display effect of the terminal is improved, thereby improving the user experience.
  • an embodiment of the present application also provides a terminal, and the terminal 10 includes:
  • the display screen 110 , the three-dimensional identification device is arranged inside the display screen 110 , and the area corresponding to the lens of the display screen 110 and the three-dimensional identification device is a light transmittance enhanced area 120 .
  • the number of lenses of the 3D recognition device depends on the scheme of the 3D recognition device, as shown in Figure 4, which is the layout of the 3D recognition device corresponding to Embodiment 3 Schematic diagram
  • the infrared dot matrix projector 320 and the second infrared camera 420 can be reduced on the basis of Figure 4.
  • the area of the enhanced translucency region 120 can be further reduced on the basis of FIG. 4 , and the specific area of the enhanced translucency region 120 can be adjusted according to the number of lenses corresponding to the structure required for three-dimensional recognition.
  • the present application also provides a calibration method applied to the three-dimensional recognition device described in Embodiment 1.
  • the calibration method includes but is not limited to the following steps:
  • Step S810 when the infrared floodlight illuminator is in the working state, according to the first preset time sequence, the first image set is taken by the first infrared camera, and the second image set is taken by the RGB camera, wherein the first infrared camera
  • the target objects photographed at different times are different, the target objects photographed by the RGB camera are different at different moments, and the target objects photographed by the first infrared camera and the RGB camera are the same at the same moment;
  • Step S820 performing calibration between the first infrared camera and the infrared flood illuminator according to the first image set;
  • Step S830 fusing images captured at the same time in the first image set and the second image set respectively to obtain a first fused image set, and performing calibration between the RGB camera and the first infrared camera according to the first fused image set.
  • the baselines 130 between multiple lenses coincide, so multiple photosensitive devices can be calibrated directly.
  • the first baseline 510 between the second lens 312 and the first lens 212 and the second baseline 520 between the third lens 412 and the first lens 212 cannot ensure On the same straight line, therefore, on the basis of the structure of the three-dimensional recognition device of the embodiment of the present application, two calibrations are required between the optical sensing devices, that is, the first infrared camera 410 and the RGB camera 210 are calibrated, and the infrared reflective illuminator 310 and the RGB camera 210 are calibrated.
  • the RGB camera 210 is calibrated to ensure that multiple optical sensor devices that are not distributed in parallel can be calibrated normally and used in conjunction with each other.
  • the target object can usually be calibrated with obvious contrast features such as a checkerboard diagram or a dot map, or the same pattern can be photographed from different angles through the photosensitive device.
  • this The embodiment can use the calibration object shown in FIG. 13 as the target object, the target object has 4 sides, each side has the same checkerboard figure, and the angles of the patterns of the checkerboard figure on each side are different, so that when shooting different sides The image content that can be obtained is different.
  • the target object shown in FIG. 13 can also be configured with a mechanical device to realize each side tilting forward or backward under command control, so as to capture more image content from different angles.
  • background infrared light sources and background visible light sources can also be added according to actual needs to ensure sufficient light, and details will not be repeated here.
  • the 3D recognition device can be fixedly placed, and the RGB camera 210 is facing the target object. Since the first lens 212 is adjacent to the second lens 312 and the third lens 412 and has a small aperture, it can be regarded as facing the calibration object.
  • the first preset time sequence is the interval between two adjacent shots.
  • the first preset time sequence can be set to be the same as the rotation period of the target object, for example, each side shot In the case of one sheet, it takes 2 seconds for the target object to turn from one side to the other, then the first preset time sequence can be 0 seconds, 2 seconds, 4 seconds, etc.
  • the first preset time sequence can be 0 seconds, 2 seconds, 4 seconds, etc.
  • the specific number can be determined according to the number of homography matrices corresponding to the number of internal parameters in the calibration parameters, for example, the internal parameters
  • the number of coefficients to be solved is n, and n (n is an even number) or n+1 (n is an odd number) equations that can be solved by the least square method need to be solved.
  • the number of homography matrices is n/2 (n is an even number) Or (n+1)/2 (n is an odd number), then the number of photos taken in the second round for target objects with different inclinations can be determined as n/2 (n is an even number) or (n+1)/2 (n is odd number).
  • the number of shots can also be increased according to actual needs, as long as calibration can be achieved, and there are no limitations here.
  • the image reflected by the infrared flood illuminator 310 is a two-dimensional image
  • the plane equation of the plane of the target object under the camera coordinates can be estimated from the two-dimensional image
  • the laser point cloud of the infrared reflective illuminator 310 Associated with the above plane transform the laser point cloud to the camera coordinate system through the mapping relationship between the laser coordinate system and the camera coordinate system, construct the minimum distance from each point in the laser point cloud to the plane, and then solve the minimum distance by the least square method
  • the distance value forms the calibration.
  • the above calibration method is only an example.
  • the calibration of the first infrared camera 410 and the RGB camera 210 can also be implemented in other ways, which will not be limited here.
  • the images in the first image set and the second image set are captured by different devices at the same time, so the images captured at the same moment can be fused.
  • the technical Personnel are familiar with how to complete the calibration of the RGB camera 210 , for example, by using methods such as Zhang Zhengyou method, OpenCV, and Matlab to obtain the respective internal parameters and distortion coefficients, and will not repeat them here.
  • the calibration method of the present application can also be applied to the three-dimensional recognition device described in the second embodiment above, which also includes but is not limited to the following steps:
  • Step S910 when the infrared dot matrix projector is in the working state, according to the second preset time sequence, the third image set is taken by the first infrared camera, and the fourth image set is taken by the RGB camera, wherein the first infrared camera
  • the target objects photographed at different times are different
  • the target objects photographed by the RGB camera are different at different moments
  • the target objects photographed by the first infrared camera and the RGB camera are the same at the same moment
  • Step S920 performing calibration between the first infrared camera and the infrared dot matrix projector according to the third image set;
  • Step S930 fusing images taken at the same time in the third image set and the fourth image set respectively to obtain a second fused image set, and performing calibration between the RGB camera and the first infrared camera according to the second fused image set.
  • the image capture of the third image set can be performed after the image capture of the first image set is completed.
  • alternately Start the infrared flood irradiator 310 and the infrared dot matrix projector 320 to shoot that is, alternately shoot the images of the first image set and the third image set, and select a specific working method according to the timing requirements, and there is not much to do here limited.
  • the infrared dot matrix projector 320 and the infrared flood light irradiator 310 work alternately, which can prevent the laser scatter plate of the infrared dot matrix projector 320 from interfering with the infrared light emitted by the infrared flood light irradiator 310 .
  • the infrared light emitted by the infrared dot matrix projector 320 is laser speckle
  • the first infrared camera 410 receives multiple three-dimensional light reflected from the target object irradiated by the infrared dot matrix projector 320 .
  • the laser speckle image after that, the three-dimensional laser speckle image is corresponding to the two-dimensional image of the first infrared camera 410 and converted.
  • This conversion process introduces homogeneous coordinates to realize the transformation from three-dimensional space points to two-dimensional images, so as to realize the transformation of each Each point is translated and rotated, and then the plane equations of multiple groups of target objects in the camera coordinates are obtained, and the distance error between the point and the plane is optimized to achieve calibration.
  • the calibration method of the present application can also be applied to the three-dimensional recognition device described in the third embodiment above, which also includes but is not limited to the following steps:
  • Step S1010 when the infrared dot matrix projector is in the working state, according to the second preset time sequence, the third image set is taken by the second infrared camera, and the fourth image set is taken by the RGB camera, wherein the second infrared camera
  • the target objects photographed at different times are different, the target objects photographed by the RGB camera are different at different moments, and the target objects photographed by the first infrared camera and the RGB camera are the same at the same moment;
  • Step S1020 performing calibration between the second infrared camera and the infrared dot matrix projector according to the third image set;
  • Step S1030 fusing images taken at the same time in the third image set and the fourth image set respectively to obtain a second fused image set, and performing calibration between the RGB camera and the second infrared camera according to the second fused image set.
  • the technical principle of this embodiment can refer to the principle of the embodiment described in FIG. 310 for matching work, the second infrared camera 420 and the infrared dot matrix projector 320 for matching work, of course, the matching relationship can also be exchanged, and no more limitations are made here.
  • the first image set can also be taken by the first infrared camera 410 and the second infrared camera 420 respectively under the condition that the infrared reflective illuminator 310 works, and two available images can be obtained respectively.
  • the images are cross-referenced, which can effectively improve the efficiency of shooting and calibration.
  • the third image set can also take a similar operation, which will not be repeated here.
  • the target object Assume that the time sequence is rotated, that is, the image capture of the first image set and the second image set is completed for the first side, and the image capture of the third image set and the fourth image set is completed for the second side, even if it can be done in the mechanical device It is difficult to adjust the inclination angle of the side of the target object to be photographed under the control of the target object, and it is also difficult to ensure that the image captured after the target object turns around is sufficient.
  • the target object after the target object completes one round of rotation, it can be increased by 90 degrees on the basis of the first rotation of the second round, that is, the image capture of the first image set and the second image set is performed for the second side , to ensure that each image set takes a different image than the first round.
  • the target object can also be rotated in a third round with an uneven angle or a small corner. side to take pictures simultaneously.
  • the uneven angle of the calibration object or the rotation of a small corner will cause the camera to take pictures across two sides or only take pictures of a single side part.
  • this wheel is only one of the first two rounds of rotation.
  • Supplementary photography on the one hand, only a small number of images are added for the correction of laser speckle interference, and on the other hand, it is also used to effectively supplement when the first two rounds of images of the target object are insufficient.
  • the above-mentioned adjustment method of the target object is only an example of this embodiment, and may also be adjusted according to the timing requirements, or a plurality of different target objects may be configured, which is not limited here.
  • step S910 shown in FIG. 9 or after executing step S1010 shown in FIG. 10 , it also includes but is not limited to the following steps:
  • Step S1110 filtering out the laser speckle information in the images of the third image set.
  • the image captured by the second infrared camera 420 contains laser speckle information.
  • it needs to be filtered out by denoising. For example, it can be used for Overcome the multiplicative distortion caused by speckle, convert the noise into an additive model with natural logarithm during iteration, then convert the image from RGB space to Hue Lightness Saturation (HLS) space, and extract the red space range, Then switch back to RGB space and further convert to gray space, perform histogram equalization and filtering processing, calculate the corner points to generate a checkerboard image and then perform calibration.
  • HLS Hue Lightness Saturation
  • step S1030 shown in FIG. 10 is executed, the following steps are also included but not limited to:
  • Step S1210 obtaining calibration parameters, where the calibration parameters include a first calibration parameter of the first infrared camera, a second calibration parameter of the second infrared camera, and a third calibration parameter of the RGB camera;
  • Step S1220 perform global calibration among the first infrared camera, the second infrared camera and the RGB camera according to the calibration parameters.
  • the three cameras need to be globally calibrated to realize cooperative work. Based on this, the respective calibration parameters need to be calculated, wherein the calibration parameters usually include internal parameters, distortion coefficients, external parameters and image scale factors.
  • the internal parameters and distortion coefficients can be obtained by fused calculations of one or more rounds of rotation of the target object captured by the three cameras, and those skilled in the art are familiar with how to calculate the corresponding parameters. More details.
  • the calculation of the extrinsic parameters requires three cameras to take a picture at the same time to calculate, that is, the three cameras need to take pictures of the same target object in a static state at the same time, and then perform feature calculations for their respective images.
  • the extrinsic parameters Calculation is a technique well known to those skilled in the art, and details are not repeated here.
  • the image scale factor represents the difference between the imaging of space objects in the two images due to the deviation of the optical center of the infrared camera and the RGB camera and the difference in the focal length of infrared and visible light. Therefore, the image scale factor can be obtained by comparing the infrared and RGB pixel differences of the two-dimensional calibration image taken on the side of the target object, so that the size of the spatial object can be unified on the infrared and RGB images.
  • the infrared and RGB images can be aligned.
  • the terminal 1400 includes: a memory 1410 , a processor 1420 , and a computer program stored in the memory 1410 and operable on the processor 1420 .
  • the processor 1420 and the memory 1410 may be connected through a bus or in other ways.
  • the non-transitory software programs and instructions required to realize the calibration method of the above-mentioned embodiment are stored in the memory 1410, and when executed by the processor 1420, the calibration method in the above-mentioned embodiment is executed, for example, the above-described execution in FIG. 8 Method step S810 to step S830, method step S910 to step S930 in FIG. 9 , method step S1010 to step S1030 in FIG. 10 , method step S1110 in FIG. 11 , method step S1210 to step S1220 in FIG. 12 .
  • the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • an embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor or a controller, for example, by the above-mentioned Execution by a processor in the terminal embodiment can cause the above-mentioned processor to execute the calibration method in the above-mentioned embodiment, for example, execute the method steps S810 to S830 in FIG. 8 and the method steps S910 to S910 in FIG. 9 described above. S930, method step S1010 to step S1030 in FIG. 10 , method step S1110 in FIG. 11 , method step S1210 to step S1220 in FIG. 12 .
  • Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer.
  • communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .
  • the embodiment of the present application includes: an RGB component, the RGB component includes an RGB camera and a first lens, a first optical channel is connected between the first lens and the RGB camera; an infrared emitting component, the infrared emitting component includes An infrared floodlight irradiator and a second lens, a second optical channel is connected between the infrared floodlight irradiator and the second lens, and the second lens is adjacent to the first lens; an infrared receiving assembly, The infrared receiving component includes a first infrared camera and a third lens, a third optical channel is connected between the first infrared camera and the third lens, and the third lens is adjacent to the first lens.
  • the RGB camera, the infrared floodlight irradiator and the first infrared camera can be arranged separately from the lens, and the transmission of light can be realized through the optical channel, so that the layout of multiple lenses under the screen can be more compact, thereby effectively Reduce the area of the area that is specially treated to improve light transmittance, effectively improve the display effect of the full screen, and improve user experience.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

本申请提供了一种三维识别装置、终端、标定方法、存储介质，三维识别装置包括：RGB组件，包括RGB摄像头(210)和第一镜头(212)，第一镜头(212)与RGB摄像头(210)之间连接有第一光通道(211)；红外发射组件，包括红外泛光照射器(310)和第二镜头(312)，红外泛光照射器(310)与第二镜头(312)之间连接有第二光通道(311)，第二镜头(312)与第一镜头(212)相邻；红外接收组件，包括第一红外摄像头(410)和第三镜头(412)，第一红外摄像头(410)和第三镜头(412)之间连接有第三光通道(411)，第三镜头(412)与第一镜头(212)相邻。

Description

三维识别装置、终端、标定方法、存储介质
相关申请的交叉引用
本申请基于申请号为202111516186.2、申请日为2021年12月07日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请涉及但不限于智能终端领域，尤其涉及一种三维识别装置、终端、标定方法、存储介质。
背景技术
随着显示屏技术的发展，屏下摄像头已经逐渐应用到各种智能终端，深受广大消费者的喜爱。由于镜头设置在显示屏内侧，需要在镜头对应的显示屏区域进行提升透光性的特殊处理，例如通过减少红绿蓝(Red Green Blue，RGB)像素或者缩小RGB像素，以增加通光量。
随着三维识别技术的发展，智能终端不仅仅需要在显示屏设置前置RGB摄像头，还需要设置各种三维识别器件，而每个三维识别器件的数量较多，目前主要采用图1所示的并列布局方式，这就导致显示屏为了提升透光性进行特殊处理的区域较大，而由于透光性不同，特殊处理的区域与显示屏的其他区域在显示效果上有一定的差异，影响全面屏的使用体验。
发明内容
以下是对本文详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。
本申请实施例提供了一种三维识别装置、终端、标定方法、存储介质。
第一方面，本申请实施例提供了一种三维识别装置，设置于终端的显示屏内侧，所述三维识别装置包括：RGB组件，所述RGB组件包括RGB摄像头和第一镜头，所述第一镜头与所述RGB摄像头之间连接有第一光通道；红外发射组件，所述红外发射组件包括红外泛光照射器和第二镜头，所述红外泛光照射器与所述第二镜头之间连接有第二光通道，所述第二镜头与所述第一镜头相邻；红外接收组件，所述红外接收组件包括第一红外摄像头和第三镜头，所述第一红外摄像头和所述第三镜头之间连接有第三光通道，所述第三镜头与所述第一镜头相邻。
第二方面，本申请实施例提供了一种终端，包括：如第一方面所述的三维识别装置；显示屏，所述三维识别装置设置于所述显示屏内侧，所述显示屏与所述三维识别装置的镜头所对应的区域为透光性增强区域。
第三方面，本申请实施例提供了一种标定方法，应用于三维识别装置，所述三维识别装置包括RGB组件、红外发射组件和红外接收组件，其中，所述RGB组件包括RGB摄像头和第一镜头，所述第一镜头与所述RGB摄像头之间连接有第一光通道，所述红外发射组件包括红外泛光照射器和第二镜头，所述红外泛光照射器与所述第二镜头之间连接有第二光通道，所述第二镜头与所述第一镜头相邻，所述红外接收组件包括第一红外摄像头和第三镜头，所述第一红外摄像头和所述第三镜头之间连接有第三光通道，所述第三镜头与所述第一镜头相邻。所述标定方法包括：在所述红外泛光照射器处于工作状态的情况下，根据第一预设时间序列，通过所述第一红外摄像头拍摄第一图像集，通过所述RGB摄像头拍摄第二图像集，其中，所 述第一红外摄像头在不同时刻拍摄的目标物体不同，所述RGB摄像头在不同时刻拍摄的目标物体不同，所述第一红外摄像头和所述RGB摄像头在相同时刻拍摄的目标物体相同；根据所述第一图像集进行所述第一红外摄像头和所述红外泛光照射器之间的标定；融合所述第一图像集和所述第二图像集中分别拍摄于相同时刻的图像，得到第一融合图像集，根据所述第一融合图像集进行所述RGB摄像头和所述第一红外摄像头之间的标定。
第四方面，本申请实施例提供了一种终端，包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如第三方面所述的标定方法。
第五方面，本申请实施例提供了一种计算机可读存储介质，存储有计算机可执行指令，所述计算机可执行指令用于执行如第三方面所述的标定方法。
本申请的其它特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本申请而了解。本申请的目的和其他优点可通过在说明书、权利要求书以及附图中所特别指出的结构来实现和获得。
附图说明
附图用来提供对本申请技术方案的进一步理解，并且构成说明书的一部分，与本申请的实施例一起用于解释本申请的技术方案，并不构成对本申请技术方案的限制。
图1是现有技术中全面屏的三维识别装置和RGB摄像头的布局方式；
图2是本申请实施例的三维识别装置设置在终端的示意图；
图3是本申请提供的三维识别装置的截面示意图；
图4是本申请提供的三维识别装置的正视图；
图5是本申请实施例一的布局示意图；
图6是本申请实施例二的布局示意图；
图7是本申请实施例三的布局示意图；
图8是本申请提供的应用于实施例一的三维识别装置的标定方法的流程图；
图9是本申请提供的应用于实施例二的三维识别装置的标定方法的流程图；
图10是本申请提供的应用于实施例三的三维识别装置的标定方法的流程图；
图11是本申请另一个实施例提供的滤除激光散斑信息的流程图；
图12是本申请另一个实施例提供的全局标定的流程图；
图13是本申请提供的目标物体的示例图；
图14是本申请另一个实施例提供的终端的装置图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本申请，并不用于限定本申请。
需要说明的是，虽然在装置示意图中进行了功能模块划分，在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于装置中的模块划分，或流程图中的顺序执行所示出或描述的步骤。说明书、权利要求书或上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。
本申请提供了一种三维识别装置、终端、标定方法、存储介质，三维识别装置包括：RGB 组件，所述RGB组件包括RGB摄像头和第一镜头，所述第一镜头与所述RGB摄像头之间连接有第一光通道；红外发射组件，所述红外发射组件包括红外泛光照射器和第二镜头，所述红外泛光照射器与所述第二镜头之间连接有第二光通道，所述第二镜头与所述第一镜头相邻；红外接收组件，所述红外接收组件包括第一红外摄像头和第三镜头，所述第一红外摄像头和所述第三镜头之间连接有第三光通道，所述第三镜头与所述第一镜头相邻。根据本实施例的技术方案，RGB摄像头、红外泛光照射器和第一红外摄像头能够与镜头分离布局，通过光通道实现光线的传播，使得多个镜头在屏下的布局可以更加紧凑，从而有效减少为了提升透光性进行特殊处理的区域的面积，有效提高全面屏的显示效果，提高用户体验。
下面结合附图，对本申请实施例作进一步阐述。
本申请实施例提供了一种三维识别装置，设置于终端的显示屏内侧，该三维识别装置包括：
RGB组件，RGB组件包括RGB摄像头210和第一镜头212，第一镜头212与RGB摄像头210通过第一光通道211连接；
红外发射组件，红外发射组件包括红外泛光照射器310和第二镜头312，红外泛光照射器310与第二镜头312之间连接有第二光通道311，第二镜头312与第一镜头211相邻；
红外接收组件，红外接收组件包括第一红外摄像头410和第三镜头412，第一红外摄像头410和第三镜头412之间连接有第三光通道411，第三镜头412与第一镜头212相邻。
需要说明的是，常见的三维识别装置的布局方式如图1所示，三维识别器件设置在终端10的显示屏110中并列布局，由于三维识别器件的本体尺寸较大，各个镜头之间有一定的距离，导致透光性增强区域120的面积较大，影响全面屏的显示体验。采用本实施例的三维识别装置，能够实现三维识别器件的本体与镜头的分离，能够以图2所示的方式，仅考虑镜头的尺寸进行紧凑布局，三维识别器件的本体在终端10的内部分布，通过光通道实现光线传播，能够有效减少透光性增强区域120的面积，提高用户体验。
值得注意的是，RGB摄像头210的第一镜头212尺寸较大，红外发射组件和红外接收组件的镜头尺寸较小，通常是小孔径镜头，因此，可以如图2所示，在第一镜头212的周围环绕若干个小孔径镜头，进一步提高布局的紧凑性。
需要说明的是，在本实施例中，小孔径镜头为第二镜头312和第三镜头412，由于RGB摄像头210所需要的通光量较大，第二镜头312和第三镜头412的孔径比起第一镜头212的孔径相对较小，因此，将第二镜头312和第三镜头412布局在第一镜头212相邻位置不会对RGB摄像头210的本体造成干涉。第二镜头312和第三镜头412的孔径可以根据实际的进光量需求调整，在此对具体尺寸不多作限定。
需要说明的是，参考图3，第一光通道211、第二光通道311和第三光通道411可以是任意形状，能够确保光线能够入射或者出射即可，在此不多作限定。
下面通过几个实施例对三维识别装置的各种实施例进行说明。
实施例一：
参照图3，图3为终端水平放置的情况下，沿竖直方向截取的截面图。在RGB镜头210通过第一光通道211与第一镜头212连接的情况下，RGB镜头210与第一镜头212之间有一定的距离，为了实现布局的紧凑，第二镜头312和第三镜头412可以以图5所示的方式与第一镜头212相邻设置，即环绕在第一镜头212周围，具体位置可以根据实际需求调整，互相 之间不会造成干涉即可，本实施例对此不多作限定。
在本实施例中，RGB摄像头210、红外泛光照射器310和第一红外摄像头410的主体尺寸比所对应的镜头大，若光通道均为直线通道，为了实现图5所示的镜头布局，需要设置较长的第一光通道211，以确保红外泛光照射器310和第一红外摄像头410与RGB摄像头210互不干涉，这会导致终端的厚度增加，因此，本实施例第二光通道311和第三光通道411均采用图3所示的直角结构，在第二光通道311的转角处设置第一反射镜313，在第三光通道411的转角处设置第二反射镜413，通过反射镜改变红外光的传播方向，使得红外泛光照射器310和第一红外摄像头410能够与RGB摄像头210横向错开，得到的内部布局可以参考图4所示的正视图，有效减少了三维识别装置所需要的纵向空间，有效减少三维识别装置的厚度。
值得注意的是，采用本实施例的结构，能够利用飞行时间(Time of Flight，TOF)实现三维识别，基于此，红外泛光照射器310采用大功率的垂直共腔振表面放射激光器(Vertical Cavity Surface Emitting Laser，VCSEL)，第一红外摄像头410的内部也可以根据实际需求调整，能够实现TOF的准确采集即可，本实施例对器件的内部结构不多作限定。
实施例二：
在实施例一的基础上，本实施例的红外发射组件还包括红外点阵投影器320和第四镜头322，红外点阵投影器320与第四镜头322之间连接有第四光通道321，第四镜头322与第一镜头212相邻；在第四光通道321中还设置有第三反射镜(图中未示出)，第三反射镜用于将红外点阵投影器320发射的红外光反射至第四镜头。红外点阵投影器320、第四镜头321、第四光通道321和第三反射镜的设置原理可以参考红外泛光照射器310、第二镜头312、第二光通道311和第一反射镜313，为了叙述简便在此不重复赘述。
需要说明的是，参照图4和图6，第四镜头322与第二镜头312、第三镜头412环绕设置于第一镜头311周围，具体的位置可以根据实际需求调整，本实施例对此不多作限定。
值得注意的是，实施例一为了实现TOF的三维识别，红外泛光照射器310采用大功率VCSEL，而本实施例的结构是采用单目结构光方案，因此红外泛光照射器310可以采用较低功率的VCSEL，红外点阵投影器320采用大功率VCSEL以透射激光散斑，并且，第一红外摄像头410的结构相比起实施例一也可以得到简化，在此对器件的内部结构不做过多赘述。
实施例三：
在实施例二的基础上，本实施例红外接收组件包括第二红外摄像头420和第五镜头422，第二红外摄像头420和第五镜头422之间连接有第五光通道421，第五镜头422与第一镜头212相邻，另外，第五光通道421中还设置有第四反射镜(图中未示出)，第四反射镜用于将通过第五镜头422入射的光反射至第二红外摄像头420。第二红外摄像头420、第五镜头422和第五光通道421和第四反射镜的设置原理可以参考第一红外摄像头410、第三镜头412和第三光通道411和第二反射镜，为了叙述简便在此不重复赘述。
需要说明的是，参照图4和图7，第五镜头422、第四镜头322、第二镜头312和第三镜头412环绕设置于第一镜头311周围，具体的位置可以根据实际需求调整，本实施例对此不多作限定。
值得注意的是，实施例二能够实现单目结构光的三维识别，而本实施例增加了第二红外摄像头420，因此能够实现双目结构光的三维识别，具体的器件参数根据实际需求选取即可，在此不多作限定。
通过上述三个实施例的三维识别装置，能够利用终端的内部空间设置识别器件的主体，有效减少透光性增强区域120的面积，从而减少了受到透光性增强区域120影响的显示区域，提高了终端的全面显示效果，从而提高用户体验。
另外，参照图2，本申请还实施例还提供了一种终端，该终端10包括：
如上实施例所述的三维识别装置；
显示屏110，所述三维识别装置设置于所述显示屏110内侧，所述显示屏110与所述三维识别装置的镜头所对应的区域为透光性增强区域120。
需要说明的是，参考实施例一至实施例三的描述，三维识别装置的镜头的数量取决于三维识别器件的方案，如图4所示，图4为与实施例三对应的三维识别装置的布局示意图，当需要采用实施例一的结构实现TOF的三维识别，可以在图4的基础上减少红外点阵投影器320和第二红外摄像头420，以及相对应的光通道和反射镜，在此情况下，透光性增强区域120的面积可以在图4的基础上进一步减小，透光性增强区域120的具体面积根据三维识别所需要的结构所对应的镜头数量进行调整即可。
另外，参照图8，本申请还提供了一种标定方法，应用于实施例一所述的三维识别装置，该标定方法包括但不限于有以下步骤：
步骤S810，在红外泛光照射器处于工作状态的情况下，根据第一预设时间序列，通过第一红外摄像头拍摄第一图像集，通过RGB摄像头拍摄第二图像集，其中，第一红外摄像头在不同时刻拍摄的目标物体不同，RGB摄像头在不同时刻拍摄的目标物体不同，第一红外摄像头和RGB摄像头在相同时刻拍摄的目标物体相同；
步骤S820，根据第一图像集进行第一红外摄像头和红外泛光照射器之间的标定；
步骤S830，融合第一图像集和第二图像集中分别拍摄于相同时刻的图像，得到第一融合图像集，根据第一融合图像集进行RGB摄像头和第一红外摄像头之间的标定。
需要说明的是，在图1所示的常规的排列布局方式下，多个镜头之间的基线130重合，因此可以多个光感器件直接进行标定，但是如图5所示，由于第二镜头312和第三镜头412与第一镜头212相邻，第二镜头312与第一镜头212之间的第一基线510和第三镜头412与第一镜头212之间的第二基线520并不能确保在同一条直线，因此，在本申请实施例的三维识别装置的结构基础上，光学传感器件之间需要两两标定，即第一红外摄像头410和RGB摄像头210进行标定，红外反光照射器310与RGB摄像头210进行标定，以确保多个不成并列分布的光学传感器件能够正常标定和相互配合使用。
需要说明的是，在光感器件的标定流程中，目标物体通常可以用棋盘图或者圆点图等对比特征明显的标定物，也可以通过光感器件从不同角度拍摄同一个图案，例如，本实施例可以利用图13所示的标定物作为目标物体，该目标物体具有4个侧面，每个侧面具有相同的棋盘图，且每个侧面的棋盘图的图案的角度不同，从而在拍摄不同侧面所能得到的图像内容不同。进一步地，图13所示的目标物体还可以配置机械装置，在指令控制下实现每个侧面的前倾或者后仰，从而拍摄到更多不同角度的图像内容。在采用图13所示的目标物体的情况下，还可以根据实际需求增加背景红外光源和背景可见光源，以确保具有充足的光线，在此不多做赘述。
需要说明的是，三维识别装置可以固定放置，RGB摄像头210正视目标物体，由于第一镜头212与第二镜头312和第三镜头412相邻，且孔径较小，因此可以认为是正视标定物。
值得注意的是，第一预设时间序列为相邻两次拍摄之间的间隔时间，为了实现自动拍摄，可以将第一预设时间序列与目标物体的转动周期设置相同，例如每个侧面拍摄一张的情况下，目标物体由一个侧面转到另一个侧面需要2秒，则第一预设时间序列可以0秒、2秒、4秒等，本领域技术人员能够设置相应参数，使得两个设备之间能够配合完成拍摄。
需要说明的是，为了确定标定参数，需要通过第一红外摄像头410和RGB摄像头210拍摄多张图像，具体的数量可以根据标定参数中的内参求解数量对应的单应性矩阵数量确定，例如，内参系数求解数量为n，需要求解n(n为偶数)或者n+1(n为奇数)个可以用最小二乘法求解的方程，此时的单应性矩阵数量为n/2(n为偶数)或者(n+1)/2(n为奇数),则次轮针对不同倾斜程度的目标物体的拍照数量即可确定为n/2(n为偶数)或者(n+1)/2(n为奇数)。当然，也可以根据实际需求增加拍摄数量，能够实现标定即可，在此不多做限定。
值得注意的是，红外泛光照射器310反射回来的图像为二维图像，可以通过二维图像估算出目标物体的平面在相机坐标下的平面方程，再将红外反光照射器310的激光点云关联至到上述平面，将激光点云通过激光坐标系到相机坐标系的映射关系转换到相机坐标系，构建激光点云中每个点到平面的最小距离，再通过最小二乘法求解出该最小距离值从而形成标定，上述标定方法仅为示例，在具备第一图像集的基础上，也可以通过其他方式实现第一红外摄像头410与RGB摄像头210的标定，在此不多做限定。
值得注意的是，第一图像集和第二图像集中的图像是同时通过不同器件拍摄得到，因此可以将拍摄于同一时刻的图像进行融合，在具备第一融合图像集的基础上，本领域技术人员熟知如何完成RGB摄像头210的标定，例如通过张正友法、OpenCV和Matlab等方法求解出各自的内参和畸变系数，在此不多做赘述。
另外，参照图9，本申请的标定方法还可以应用于上述实施例二所述的三维识别装置，还包括但不限于有以下步骤：
步骤S910，在红外点阵投影器处于工作状态的情况下，根据第二预设时间序列，通过第一红外摄像头拍摄第三图像集，通过RGB摄像头拍摄第四图像集，其中，第一红外摄像头在不同时刻拍摄的目标物体不同，RGB摄像头在不同时刻拍摄的目标物体不同，第一红外摄像头和RGB摄像头在相同时刻拍摄的目标物体相同；
步骤S920，根据第三图像集进行第一红外摄像头和红外点阵投影器之间的标定；
步骤S930，融合第三图像集和第四图像集中分别拍摄于相同时刻的图像，得到第二融合图像集，根据第二融合图像集进行RGB摄像头和第一红外摄像头之间的标定。
需要说明的是，在具备两个红外发射器件的情况下，可以在完成第一图像集的图像拍摄之后，再进行第三图像集的图像拍摄，当然，也可以出于提高效率的动机，交替启动红外泛光照射器310和红外点阵投影器320进行拍摄，即交替拍摄第一图像集的图像和第三图像集的图像，根据时机需求选取具体的工作方式即可，在此不多做限定。
值得注意的是，第四图像集的拍摄方式和原理可以参考图8所示实施例的第二图像集的拍摄方式和原理，在此不重复赘述。
值得注意的是，红外点阵投影器320与红外泛光照射器310交替工作，能够避免红外点阵投影器320的激光散板对红外泛光照射器310发射的红外光造成干扰。
值得注意的是，第二融合图像集的获取方法，以及RGB摄像头210与第一红外摄像头410 的标定方式可以参考图8所示实施例的描述，在此不重复赘述。
需要说明的是，在本实施例中，红外点阵投影器320所发射的红外光为激光散斑，第一红外摄像头410接收到红外点阵投影器320照射到目标物体反射回来的多个三维激光散斑图像，之后，将三维激光散斑图像对应到第一红外摄像头410的二维图像并进行转换，此转换过程引入齐次坐标实现三维空间点到二维图像的变换，从而实现对每个点进行平移和旋转操作，进而得到多组目标物体在相机坐标下的平面方程，优化点到平面间的距离误差实现标定。
另外，参照图10，本申请的标定方法还可以应用于上述实施例三所述的三维识别装置，还包括但不限于有以下步骤：
步骤S1010，在红外点阵投影器处于工作状态的情况下，根据第二预设时间序列，通过第二红外摄像头拍摄第三图像集，通过RGB摄像头拍摄第四图像集，其中，第二红外摄像头在不同时刻拍摄的目标物体不同，RGB摄像头在不同时刻拍摄的目标物体不同，第一红外摄像头和RGB摄像头在相同时刻拍摄的目标物体相同；
步骤S1020，根据第三图像集进行第二红外摄像头和红外点阵投影器之间的标定；
步骤S1030，融合第三图像集和第四图像集中分别拍摄于相同时刻的图像，得到第二融合图像集，根据第二融合图像集进行RGB摄像头和第二红外摄像头之间的标定。
需要说明的是，本实施例的技术原理可以参考图9所述实施例的原理，区别在于，第三图像集由第二红外摄像头420拍摄得到，即第一红外摄像头410与红外泛光照射器310匹配工作，第二红外摄像头420与红外点阵投影器320匹配工作，当然也可以互换匹配关系，在此不多做限定。
需要说明的是，除了上述区别，第一图像集也可以由在红外反光照射器310工作的情况下，由第一红外摄像头410和第二红外摄像头420分别执行一次拍摄，分别得到两张可用的图像并交叉引用，能够有效提高拍摄和标定的效率，第三图像集也可以采取类似的操作，在此不再赘述。
值得注意的是，在通过第一融合图像集完成第一红外摄像头410与RGB摄像头210的标定之后，还需要通过第二融合图像集完成第二红外摄像头420与RGB摄像头210之间的标定，由于各光感器件的基线不处在同一直线，因此需要两两进行标定，以确保三维识别装置能够正常运行。
值得注意的是，在具备两个红外摄像头和两个红外发射器的情况下，且红外点阵投影器320和红外泛光照射器310需要交替工作，根据上述实施例的描述，目标物体根据预设时间序列进行转动，即针对第一个侧面完成第一图像集和第二图像集的图像拍摄，针对第二个侧面完成第三图像集和第四图像集的图像拍摄，即使能够在机械装置的控制下调整目标物体被拍摄的侧面的倾角，也很难确保目标物体转动一圈之后拍摄的图像足够。在这种情况下，目标物体在完成一周转动之后，可以在第二周的第一次旋转的基础上增加90度，即针对第二个侧面进行第一图像集和第二图像集的图像拍摄，以确保每个图像集拍摄的图像与第一轮拍摄的图像不同。
当然，若第二轮拍摄的图像数量依然不足够，还可以将目标物体以不均匀角度或小转角开始转第三轮，第一红外摄像头410、第二红外摄像头420和RGB摄像头210对目标物体的侧面进行同步拍照。标定物不均匀角度或小转角旋转会导致摄像头跨两个侧面拍照或只拍摄到单个侧面局部的情况，和前两轮相比会出现较大的差异，因此此轮只是前两轮旋转的一个 补充拍照，一方面仅增补少量图像用于针对激光散斑干扰的矫正用途，另一方面也用于如只目标物体前两轮图像不足时做有效的增补补充。上述目标物体的调整方式仅为本实施例的示例，也可以根据时机需求调整，或者配置多个不同的目标物体，在此不多做限定。
另外，在一实施例中，参照图11，在执行完图9所示的步骤S910，或者执行完图10所示的步骤S1010之后，还包括但不限于有以下步骤：
步骤S1110，滤除第三图像集的图像中的激光散斑信息。
需要说明的是，在红外点阵投影器320工作的情况下，第二红外摄像头420拍摄的图像含有激光散斑信息，为了避免干扰标定，需要通过去噪方式将其滤除，例如，可以为了克服由散斑造成的乘法失真，在迭代时用自然对数将噪声转换成加法模型，再将图像由RGB空间转换为色度亮度饱和度(Hue Lightness Saturation，HLS)空间，提取红色空间范围，然后转回RGB空间并进一步转为灰度空间，进行直方图均衡化和滤波处理，计算角点生成棋盘格图像进而进行标定。
另外，在一实施例中，参照图12，在执行完图10所示的步骤S1030之后，还包括但不限于有以下步骤：
步骤S1210，获取标定参数，标定参数包括第一红外摄像头的第一标定参数、第二红外摄像头的第二标定参数和RGB摄像头的第三标定参数；
步骤S1220，根据标定参数对第一红外摄像头、第二红外摄像头和RGB摄像头之间进行全局标定。
需要说明的是，在完成第一红外摄像头410与RGB摄像头210，以及第二红外摄像头420与RGB摄像头210的两两标定之后，还需要将三个摄像头进行全局标定，以实现协同工作。基于此，需要计算出各自的标定参数，其中，标定参数通常包括内参、畸变系数、外参和图像尺度因子。
需要说明的是，对于内参和畸变系数，可以由三个摄像头各自所拍摄的目标物体旋转一轮或多轮的照片融合计算得以获取，本领域技术人员熟知如何计算出对应的参数，在此不多赘述。
需要的说明的是，外参的计算需要三个摄像头同时执行一次拍照来计算，即需要三个摄像头同时各自拍摄同一幅静止状态下目标物体后，针对各自的图像进行特征计算得到，外参的计算为本领域技术人员熟知的技术，在此不多做赘述。
需要说明的是，图像尺度因子是表示由于红外摄像头和RGB摄像头的光心存在偏差以及红外和可见光焦距不同导致的空间物体在两种图像成像时的差异。因此，图像尺度因子可以通过比对所拍的目标物体的侧面的二维标定图的红外和RGB像素差得到，从而使得空间物体在红外和RGB两种图像上实现尺寸统一。
进一步地，即便红外和RGB两种图像上实现尺寸统一，红外图像移到RGB图像仍有偏移，根据标定物的侧面标定图棋盘格或圆点的坐标位置以及红外和RGB图像像素坐标位置，计算对应的像素差，则可将红外和RGB像素对齐。
另外，参照图14，本申请的一个实施例还提供了一种终端，该终端1400包括：存储器1410、处理器1420及存储在存储器1410上并可在处理器1420上运行的计算机程序。
处理器1420和存储器1410可以通过总线或者其他方式连接。
实现上述实施例的标定方法所需的非暂态软件程序以及指令存储在存储器1410中，当被 处理器1420执行时，执行上述实施例中的标定方法，例如，执行以上描述的图8中的方法步骤S810至步骤S830、图9中的方法步骤S910至步骤S930、图10中的方法步骤S1010至步骤S1030、图11中的方法步骤S1110、图12中的方法步骤S1210至步骤S1220。
以上所描述的装置实施例仅仅是示意性的，其中作为分离部件说明的单元可以是或者也可以不是物理上分开的，即可以位于一个地方，或者也可以分别到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
此外，本申请的一个实施例还提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机可执行指令，该计算机可执行指令被一个处理器或控制器执行，例如，被上述终端实施例中的一个处理器执行，可使得上述处理器执行上述实施例中的标定方法，例如，执行以上描述的图8中的方法步骤S810至步骤S830、图9中的方法步骤S910至步骤S930、图10中的方法步骤S1010至步骤S1030、图11中的方法步骤S1110、图12中的方法步骤S1210至步骤S1220。本领域普通技术人员可以理解，上文中所公开方法中的全部或某些步骤、系统可以被实施为软件、固件、硬件及其适当的组合。某些物理组件或所有物理组件可以被实施为由处理器，如中央处理器、数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路。这样的软件可以分别在计算机可读介质上，计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的，术语计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于RAM、ROM、EEPROM、闪存或其他存储器技术、CD-ROM、数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外，本领域普通技术人员公知的是，通信介质通常包含计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。
本申请实施例包括：RGB组件，所述RGB组件包括RGB摄像头和第一镜头，所述第一镜头与所述RGB摄像头之间连接有第一光通道；红外发射组件，所述红外发射组件包括红外泛光照射器和第二镜头，所述红外泛光照射器与所述第二镜头之间连接有第二光通道，所述第二镜头与所述第一镜头相邻；红外接收组件，所述红外接收组件包括第一红外摄像头和第三镜头，所述第一红外摄像头和所述第三镜头之间连接有第三光通道，所述第三镜头与所述第一镜头相邻。根据本实施例的技术方案，RGB摄像头、红外泛光照射器和第一红外摄像头能够与镜头分离布局，通过光通道实现光线的传播，使得多个镜头在屏下的布局可以更加紧凑，从而有效减少为了提升透光性进行特殊处理的区域的面积，有效提高全面屏的显示效果，提高用户体验。
以上是对本申请的若干实施进行了具体说明，但本申请并不局限于上述实施方式，熟悉本领域的技术人员在不违背本申请本质的前提下还可作出种种的等同变形或替换，这些等同的变形或替换均包含在本申请权利要求所限定的范围内。

Claims (12)

  1. 一种三维识别装置，设置于终端的显示屏内侧，所述三维识别装置包括：
    红绿蓝RGB组件，所述RGB组件包括RGB摄像头和第一镜头，所述第一镜头与所述RGB摄像头之间连接有第一光通道；
    红外发射组件，所述红外发射组件包括红外泛光照射器和第二镜头，所述红外泛光照射器与所述第二镜头之间连接有第二光通道，所述第二镜头与所述第一镜头相邻；
    红外接收组件，所述红外接收组件包括第一红外摄像头和第三镜头，所述第一红外摄像头和所述第三镜头之间连接有第三光通道，所述第三镜头与所述第一镜头相邻。
  2. 根据权利要求1所述的三维识别装置，其中：
    所述第二光通道中还设置有第一反射镜，所述第一反射镜用于将所述红外泛光照射器发射的红外光反射至所述第二镜头；
    所述第三光通道还设置有第二反射镜，所述第二反射镜用于将通过所述第三镜头入射的光反射至所述第一红外摄像头。
  3. 根据权利要求2所述的三维识别装置，其中：
    所述红外发射组件还包括红外点阵投影器和第四镜头，所述红外点阵投影器与所述第四镜头之间连接有第四光通道，所述第四镜头与所述第一镜头相邻；
    所述第四光通道中还设置有第三反射镜，所述第三反射镜用于将所述红外点阵投影器发射的红外光反射至所述第四镜头。
  4. 根据权利要求3所述的三维识别装置，其中：
    所述红外接收组件还包括第二红外摄像头和第五镜头，所述第二红外摄像头和所述第五镜头之间连接有第五光通道，所述第五镜头与所述第一镜头相邻；
    所述第五光通道中还设置有第四反射镜，所述第四反射镜用于将通过所述第五镜头入射的光反射至所述第二红外摄像头。
  5. 一种终端，包括：
    如权利要求1至4任意一项所述的三维识别装置；
    显示屏，所述三维识别装置设置于所述显示屏内侧，所述显示屏与所述三维识别装置的镜头所对应的区域为透光性增强区域。
  6. 一种标定方法，应用于三维识别装置，所述三维识别装置包括RGB组件、红外发射组件和红外接收组件，其中，所述RGB组件包括RGB摄像头和第一镜头，所述第一镜头与所述RGB摄像头之间连接有第一光通道，所述红外发射组件包括红外泛光照射器和第二镜头，所述红外泛光照射器与所述第二镜头之间连接有第二光通道，所述第二镜头与所述第一镜头相邻，所述红外接收组件包括第一红外摄像头和第三镜头，所述第一红外摄像头和所述第三镜头之间连接有第三光通道，所述第三镜头与所述第一镜头相邻；
    所述标定方法包括：
    在所述红外泛光照射器处于工作状态的情况下，根据第一预设时间序列，通过所述第一红外摄像头拍摄第一图像集，通过所述RGB摄像头拍摄第二图像集，其中，所述第一红外摄像头在不同时刻拍摄的目标物体不同，所述RGB摄像头在不同时刻拍摄的目标物体不同，所述第一红外摄像头和所述RGB摄像头在相同时刻拍摄的目标物体相同；
    根据所述第一图像集进行所述第一红外摄像头和所述红外泛光照射器之间的标定；
    融合所述第一图像集和所述第二图像集中分别拍摄于相同时刻的图像，得到第一融合图像集，根据所述第一融合图像集进行所述RGB摄像头和所述第一红外摄像头之间的标定。
  7. 根据权利要求6所述的方法，其中，所述红外发射组件还包括红外点阵投影器和第四镜头，所述红外点阵投影器与所述第四镜头之间连接有第四光通道，所述第四镜头与所述第一镜头相邻；
    所述方法还包括：
    在所述红外点阵投影器处于工作状态的情况下，根据第二预设时间序列，通过所述第一红外摄像头拍摄第三图像集，通过所述RGB摄像头拍摄第四图像集，其中，所述第一红外摄像头在不同时刻拍摄的目标物体不同，所述RGB摄像头在不同时刻拍摄的目标物体不同，所述第一红外摄像头和所述RGB摄像头在相同时刻拍摄的目标物体相同；
    根据所述第三图像集进行所述第一红外摄像头和所述红外点阵投影器之间的标定；
    融合所述第三图像集和所述第四图像集中分别拍摄于相同时刻的图像，得到第二融合图像集，根据所述第二融合图像集进行所述RGB摄像头和所述第一红外摄像头之间的标定。
  8. 根据权利要求6所述的方法，其中，所述红外接收组件还包括第二红外摄像头和第五镜头，所述第二红外摄像头和所述第五镜头之间连接有第五光通道，所述第五镜头与所述第一镜头相邻；
    所述方法还包括：
    在所述红外点阵投影器处于工作状态的情况下，根据第二预设时间序列，通过所述第二红外摄像头拍摄第三图像集，通过所述RGB摄像头拍摄第四图像集，其中，所述第二红外摄像头在不同时刻拍摄的目标物体不同，所述RGB摄像头在不同时刻拍摄的目标物体不同，所述第一红外摄像头和所述RGB摄像头在相同时刻拍摄的目标物体相同；
    根据所述第三图像集进行所述第二红外摄像头和所述红外点阵投影器之间的标定；
    融合所述第三图像集和所述第四图像集中分别拍摄于相同时刻的图像，得到第二融合图像集，根据所述第二融合图像集进行所述RGB摄像头和所述第二红外摄像头之间的标定。
  9. 根据权利要求7或8所述的方法，其中，在所述通过所述RGB摄像头拍摄第四图像集之后，所述方法还包括：
    滤除所述第三图像集的图像中的激光散斑信息。
  10. 根据权利要求8所述的方法，其中，在所述根据所述第二融合图像集进行所述RGB摄像头和所述第二红外摄像头之间的标定之后，所述方法还包括：
    获取标定参数，所述标定参数包括所述第一红外摄像头的第一标定参数、所述第二红外摄像头的第二标定参数和所述RGB摄像头的第三标定参数；
    根据所述标定参数对所述第一红外摄像头、所述第二红外摄像头和所述RGB摄像头之间进行全局标定。
  11. 一种终端，包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，其中，所述处理器执行所述计算机程序时实现如权利要求6至10任意一项所述的标定方法。
  12. 一种计算机可读存储介质，存储有计算机可执行指令，其中，所述计算机可执行指令用于执行如权利要求6至10任意一项所述的标定方法。
PCT/CN2022/123544 2021-12-07 2022-09-30 三维识别装置、终端、标定方法、存储介质 WO2023103559A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP22902984.8A EP4436219A1 (en) 2021-12-07 2022-09-30 Three-dimensional recognition device, terminal, calibration method, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111516186.2 2021-12-07
CN202111516186.2A CN116249069A (zh) 2021-12-07 2021-12-07 三维识别装置、终端、标定方法、存储介质

Publications (1)

Publication Number Publication Date
WO2023103559A1 true WO2023103559A1 (zh) 2023-06-15

Family

ID=86628285

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/123544 WO2023103559A1 (zh) 2021-12-07 2022-09-30 三维识别装置、终端、标定方法、存储介质

Country Status (3)

Country Link
EP (1) EP4436219A1 (zh)
CN (1) CN116249069A (zh)
WO (1) WO2023103559A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106200249A (zh) * 2016-08-30 2016-12-07 辽宁中蓝电子科技有限公司 结构光和rgb传感器模组整体式集成系统3d相机
US20170186166A1 (en) * 2015-12-26 2017-06-29 Intel Corporation Stereo depth camera using vcsel with spatially and temporally interleaved patterns
CN108564613A (zh) * 2018-04-12 2018-09-21 维沃移动通信有限公司 一种深度数据获取方法及移动终端
CN111083453A (zh) * 2018-10-18 2020-04-28 中兴通讯股份有限公司 一种投影装置、方法及计算机可读存储介质
CN112927307A (zh) * 2021-03-05 2021-06-08 深圳市商汤科技有限公司 一种标定方法、装置、电子设备及存储介质
US20210266387A1 (en) * 2020-02-21 2021-08-26 Lg Electronics Inc. Mobile terminal

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170186166A1 (en) * 2015-12-26 2017-06-29 Intel Corporation Stereo depth camera using vcsel with spatially and temporally interleaved patterns
CN106200249A (zh) * 2016-08-30 2016-12-07 辽宁中蓝电子科技有限公司 结构光和rgb传感器模组整体式集成系统3d相机
CN108564613A (zh) * 2018-04-12 2018-09-21 维沃移动通信有限公司 一种深度数据获取方法及移动终端
CN111083453A (zh) * 2018-10-18 2020-04-28 中兴通讯股份有限公司 一种投影装置、方法及计算机可读存储介质
US20210266387A1 (en) * 2020-02-21 2021-08-26 Lg Electronics Inc. Mobile terminal
CN112927307A (zh) * 2021-03-05 2021-06-08 深圳市商汤科技有限公司 一种标定方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
EP4436219A1 (en) 2024-09-25
CN116249069A (zh) 2023-06-09

Similar Documents

Publication Publication Date Title
US10897609B2 (en) Systems and methods for multiscopic noise reduction and high-dynamic range
WO2018161466A1 (zh) 深度图像获取系统和方法
CN112634374B (zh) 双目相机的立体标定方法、装置、系统及双目相机
CN110572630B (zh) 三维图像拍摄系统、方法、装置、设备以及存储介质
WO2019100933A1 (zh) 用于三维测量的方法、装置以及系统
CN108765542B (zh) 图像渲染方法、电子设备和计算机可读存储介质
CN110111262A (zh) 一种投影仪畸变校正方法、装置和投影仪
US20200007736A1 (en) Exposure Control Method, Exposure Control Device and Electronic Device
US8334893B2 (en) Method and apparatus for combining range information with an optical image
US20170134713A1 (en) Image calibrating, stitching and depth rebuilding method of a panoramic fish-eye camera and a system thereof
JP4115801B2 (ja) ３次元撮影装置
CN107241592B (zh) 一种成像设备及成像方法
CN107370951B (zh) 图像处理系统及方法
JP2014522591A (ja) 角スライス実像３ｄディスプレイのためのアライメント、キャリブレーション、およびレンダリングのシステムおよび方法
JP2017220780A (ja) 撮影装置および車両
WO2022142139A1 (zh) 投影面选取和投影图像校正方法、装置、投影仪及介质
JP7489253B2 (ja) デプスマップ生成装置及びそのプログラム、並びに、デプスマップ生成システム
CN111757086A (zh) 有源双目相机、rgb-d图像确定方法及装置
CN108322726A (zh) 一种基于双摄像头的自动对焦方法
JP2020191624A (ja) 電子機器およびその制御方法
EP4443379A1 (en) Three-dimensional recognition apparatus, terminal, image enhancement method and storage medium
WO2023103559A1 (zh) 三维识别装置、终端、标定方法、存储介质
US12069227B2 (en) Multi-modal and multi-spectral stereo camera arrays
CN113301321A (zh) 成像方法、系统、装置、电子设备及可读存储介质
JP4543821B2 (ja) ３次元形状測定装置および方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22902984

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024529399

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2022902984

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022902984

Country of ref document: EP

Effective date: 20240619

NENP Non-entry into the national phase

Ref country code: DE