CN116912171A - Image data processing method, device, equipment and storage medium - Google Patents

Image data processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN116912171A
CN116912171A CN202310692865.8A CN202310692865A CN116912171A CN 116912171 A CN116912171 A CN 116912171A CN 202310692865 A CN202310692865 A CN 202310692865A CN 116912171 A CN116912171 A CN 116912171A
Authority
CN
China
Prior art keywords
image acquisition
depth data
target
image
pixel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310692865.8A
Other languages
Chinese (zh)
Inventor
杨成熙
盛骁杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Dajia Internet Information Technology Co Ltd
Original Assignee
Beijing Dajia Internet Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Dajia Internet Information Technology Co Ltd filed Critical Beijing Dajia Internet Information Technology Co Ltd
Priority to CN202310692865.8A priority Critical patent/CN116912171A/en
Publication of CN116912171A publication Critical patent/CN116912171A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/0007—Image acquisition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Quality & Reliability (AREA)
  • Image Processing (AREA)

Abstract

The present disclosure relates to an image data processing method, apparatus, device, and storage medium, the method comprising: acquiring object attribute characteristic images acquired by at least two image acquisition devices respectively for acquiring a target object and depth images corresponding to the at least two image acquisition devices respectively; acquiring depth data of a first pixel in a first depth image corresponding to target image acquisition equipment; mapping the first pixel to an object attribute characteristic image corresponding to the adjacent image acquisition equipment to obtain mapping position information corresponding to the first pixel; determining whether the depth data is abnormal depth data relative to the target image acquisition device and the adjacent image acquisition device according to the first pixel and the mapping position information; and generating an abnormality detection result of the depth data according to whether the depth data is abnormal depth data relative to the target image acquisition device and the adjacent image acquisition devices. The embodiment of the disclosure can improve the detection precision of the anomaly detection of the depth data.

Description

Image data processing method, device, equipment and storage medium
Technical Field
The present disclosure relates to the field of computers, and in particular, to an image data processing method, apparatus, device, and storage medium.
Background
The related art generally acquires an image by a single image acquisition device (e.g., a single camera), and performs depth data calculation and accuracy estimation of the depth data on the acquired single image.
However, in the related art, the manner of performing depth data calculation and precision estimation of depth data by using the image acquired by the single image acquisition device makes the accuracy of precision estimation of depth data of a partial image area (such as a blurred area, including but not limited to a repeated texture area, a smooth texture area, etc.) in the depth calculation process lower, so that the accuracy of depth data of an image in the whole system is reduced, and further, confirmation of image space consistency cannot be accurately performed.
Disclosure of Invention
The disclosure provides an image data processing method, an image data processing device and a storage medium, which at least solve the problems that in the related art, the accuracy of the accuracy estimation of depth data of a part of image areas in the depth calculation process is low in a mode of performing depth data calculation and accuracy estimation of the depth data by using images acquired by a single image acquisition device, so that the accuracy of the depth data of images in the whole system is reduced, and further, the consistency of image space cannot be accurately confirmed. The technical scheme of the present disclosure is as follows:
According to a first aspect of an embodiment of the present disclosure, there is provided an image data processing method including:
acquiring object attribute feature images acquired by at least two image acquisition devices respectively for acquiring a target object, and determining a depth image corresponding to each image acquisition device based on the object attribute feature images acquired by each image acquisition device;
acquiring depth data of a first pixel in a first depth image corresponding to target image acquisition equipment; the target image acquisition device is any one of at least two image acquisition devices;
mapping the first pixel to an object attribute characteristic image corresponding to an adjacent image acquisition device to obtain mapping position information corresponding to the first pixel; the adjacent image acquisition devices are the image acquisition devices except the target image acquisition device in at least two image acquisition devices;
determining whether the depth data is abnormal depth data relative to the target image acquisition device and the adjacent image acquisition device according to the first pixel and the mapping position information;
and generating an abnormality detection result of the depth data according to whether the depth data is abnormal depth data relative to the target image acquisition device and the adjacent image acquisition device.
In an optional embodiment, the number of adjacent image capturing devices is at least two, and the determining, according to the first pixel and the mapping position information, whether the depth data is abnormal depth data with respect to the target image capturing device and the adjacent image capturing device includes:
determining a first adjacent image acquisition device with mapping position information in the object attribute characteristic image from at least two adjacent image acquisition devices; and determining that the mapping position information does not exist in the object attribute feature image;
determining that the depth data is illegal depth data relative to the target image acquisition device and the second adjacent image acquisition device, and determining whether the depth data is abnormal depth data relative to the target image acquisition device and the first adjacent image acquisition device according to the similarity between the first pixel and the second pixel; the second pixel is a pixel corresponding to mapping position information existing in the object attribute feature image of the first adjacent image acquisition device.
In an alternative embodiment, the determining whether the depth data is abnormal depth data with respect to the target image capturing device and the first neighboring image capturing device according to the similarity between the first pixel and the second pixel includes:
Acquiring a first object attribute feature corresponding to the first pixel from an object attribute feature image corresponding to the target image acquisition device;
acquiring a second object attribute feature corresponding to the second pixel from an object attribute feature image corresponding to the first adjacent image acquisition device;
and determining whether the depth data is abnormal depth data relative to the target image acquisition device and the first adjacent image acquisition device according to the similarity between the first object attribute feature and the second object attribute feature.
In an optional embodiment, the number of the first neighboring image capturing device and the second object attribute feature is at least two, and the determining whether the depth data is abnormal depth data with respect to the target image capturing device and the first neighboring image capturing device according to the similarity between the first object attribute feature and the second object attribute feature includes:
determining second target object attribute characteristics with similarity meeting preset conditions between the second target object attribute characteristics and the first object attribute characteristics from at least two second object attribute characteristics; determining a target object attribute feature image in which the second target object attribute feature is located;
Determining first target adjacent image acquisition equipment for acquiring attribute characteristic images of the target object from at least two first adjacent image acquisition equipment;
the depth data is determined to be non-abnormal depth data relative to the target image acquisition device and the first target neighboring image acquisition device.
In an optional embodiment, the generating the anomaly detection result of the depth data according to whether the depth data is anomaly depth data with respect to the target image capturing device and the adjacent image capturing device includes:
and determining that the depth data is non-abnormal depth data under the condition that the number of the first target adjacent image acquisition devices is larger than or equal to a first preset number times of the number of the adjacent image acquisition devices and the number of the second adjacent image acquisition devices is smaller than a second preset number times of the number of the adjacent image acquisition devices, so as to obtain an abnormal detection result of the depth data.
In an optional embodiment, the generating the anomaly detection result of the depth data according to whether the depth data is anomaly depth data with respect to the target image capturing device and the adjacent image capturing device includes:
Determining that the depth data is abnormal depth data under the condition that the number of the first target adjacent image acquisition devices and the number of the second adjacent image acquisition devices meet any one of a first condition, a second condition and a third condition, and obtaining an abnormal detection result of the depth data;
the first condition is that the number of the first target adjacent image acquisition devices is greater than or equal to a first preset number times the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is greater than or equal to a second preset number times the number of the adjacent image acquisition devices;
the second condition is that the number of the first target adjacent image acquisition devices is smaller than or equal to a first preset number times the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is smaller than a second preset number times the number of the adjacent image acquisition devices;
the third condition is that the number of the first target adjacent image acquisition devices is smaller than or equal to a first preset number multiple of the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is larger than or equal to a second preset number multiple of the number of the adjacent image acquisition devices.
In an optional embodiment, the mapping the first pixel to the object attribute feature image corresponding to the adjacent image capturing device, to obtain mapping position information corresponding to the first pixel includes:
acquiring spatial position relation information between the target image acquisition equipment and the adjacent image acquisition equipment; the spatial position relation information is characterized; the mapping relation between the position information of the target object in the world coordinate system and the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition equipment and the adjacent image acquisition equipment respectively;
and mapping the first pixel to an object attribute characteristic image corresponding to the adjacent image acquisition equipment according to the spatial position relation information and the depth data to obtain mapping position information corresponding to the first pixel.
In an alternative embodiment, the acquiring spatial positional relationship information between the target image capturing device and the adjacent image capturing device includes:
acquiring the inner parameters and the outer parameters corresponding to the target image acquisition equipment and the inner parameters and the outer parameters corresponding to the adjacent image acquisition equipment;
Establishing position information of a target object in a world coordinate system according to the internal parameters and the external parameters corresponding to the target image acquisition equipment and the internal parameters and the external parameters corresponding to the adjacent image acquisition equipment, and obtaining the spatial position relation information according to the mapping relation between the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition equipment and the adjacent image acquisition equipment respectively
According to a second aspect of the embodiments of the present disclosure, there is provided an image data processing apparatus including:
the image acquisition module is configured to acquire object attribute feature images acquired by at least two image acquisition devices respectively on a target object, and determine a depth image corresponding to each image acquisition device based on the object attribute feature images acquired by each image acquisition device;
a depth data acquisition module configured to perform acquisition of depth data of a first pixel in a first depth image corresponding to a target image acquisition device; the target image acquisition device is any one of at least two image acquisition devices;
The mapping position information generation module is configured to perform mapping of the first pixel to an object attribute characteristic image corresponding to the adjacent image acquisition equipment to obtain mapping position information corresponding to the first pixel; the adjacent image acquisition devices are the image acquisition devices except the target image acquisition device in at least two image acquisition devices;
an abnormal depth data determining module configured to perform determining whether the depth data is abnormal depth data with respect to the target image capturing device and the adjacent image capturing device according to the first pixel and the mapping position information;
an abnormality detection result generation module configured to perform generation of an abnormality detection result of the depth data according to whether the depth data is abnormality depth data with respect to the target image capturing device and the adjacent image capturing device.
In an optional embodiment, the number of the adjacent image capturing devices is at least two, and the abnormal depth data determining module includes:
a neighboring image capturing device determining submodule configured to perform a first neighboring image capturing device that determines that mapping position information exists in an object attribute feature image from at least two of the neighboring image capturing devices; and determining that the mapping position information does not exist in the object attribute feature image;
An anomaly determination sub-module configured to perform determining that the depth data is illegal depth data with respect to the target image capturing device and the second neighboring image capturing device, and determining whether the depth data is anomaly depth data with respect to the target image capturing device and the first neighboring image capturing device according to a similarity between the first pixel and the second pixel; the second pixel is a pixel corresponding to mapping position information existing in the object attribute feature image of the first adjacent image acquisition device.
In an alternative embodiment, the anomaly determination sub-module includes:
a first object attribute feature acquiring unit configured to acquire a first object attribute feature corresponding to the first pixel from an object attribute feature image corresponding to the target image capturing device;
a second object attribute feature acquiring unit configured to perform acquiring a second object attribute feature corresponding to the second pixel from an object attribute feature image corresponding to the first neighboring image capturing device;
and a similarity judging unit configured to perform determination as to whether the depth data is abnormal depth data with respect to the target image capturing apparatus and the first neighboring image capturing apparatus, based on a similarity between the first object attribute feature and the second object attribute feature.
In an alternative embodiment, the similarity determining unit includes:
a target object attribute determination subunit configured to perform determining, from at least two second object attribute features, a second target object attribute feature whose similarity with the first object attribute feature satisfies a preset condition; determining a target object attribute feature image in which the second target object attribute feature is located;
a first target adjacent image capturing device determining sub-module configured to perform determining a first target adjacent image capturing device that captures an attribute feature image of the target object from at least two first adjacent image capturing devices;
an effective depth data generation subunit configured to perform a determination that the depth data is non-abnormal depth data with respect to the target image capture device and the first target neighboring image capture device.
In an alternative embodiment, the anomaly detection result generation module is configured to perform:
and determining that the depth data is non-abnormal depth data under the condition that the number of the first target adjacent image acquisition devices is larger than or equal to a first preset number times of the number of the adjacent image acquisition devices and the number of the second adjacent image acquisition devices is smaller than a second preset number times of the number of the adjacent image acquisition devices, so as to obtain an abnormal detection result of the depth data.
In an alternative embodiment, the anomaly detection result generation module is configured to perform:
determining that the depth data is abnormal depth data under the condition that the number of the first target adjacent image acquisition devices and the number of the second adjacent image acquisition devices meet any one of a first condition, a second condition and a third condition, and obtaining an abnormal detection result of the depth data;
the first condition is that the number of the first target adjacent image acquisition devices is greater than or equal to a first preset number times the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is greater than or equal to a second preset number times the number of the adjacent image acquisition devices;
the second condition is that the number of the first target adjacent image acquisition devices is smaller than or equal to a first preset number times the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is smaller than a second preset number times the number of the adjacent image acquisition devices;
the third condition is that the number of the first target adjacent image acquisition devices is smaller than or equal to a first preset number multiple of the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is larger than or equal to a second preset number multiple of the number of the adjacent image acquisition devices.
In an alternative embodiment, the mapping location information generating module includes:
a spatial positional relationship information acquisition unit configured to perform acquisition of spatial positional relationship information between the target image capturing apparatus and the adjacent image capturing apparatus; the spatial position relation information is characterized; the mapping relation between the position information of the target object in the world coordinate system and the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition equipment and the adjacent image acquisition equipment respectively;
and the mapping unit is configured to map the first pixel to the object attribute characteristic image corresponding to the adjacent image acquisition equipment according to the spatial position relation information and the depth data, so as to obtain the mapping position information corresponding to the first pixel.
In an alternative embodiment, the spatial position relation information obtaining unit includes:
a parameter acquisition subunit configured to perform acquisition of an internal parameter and an external parameter corresponding to the target image acquisition device, and an internal parameter and an external parameter corresponding to the adjacent image acquisition device;
and the establishing subunit is configured to establish the position information of the target object in the world coordinate system according to the inner parameter and the outer parameter corresponding to the target image acquisition equipment and the inner parameter and the outer parameter corresponding to the adjacent image acquisition equipment, and the mapping relation between the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition equipment and the adjacent image acquisition equipment respectively to obtain the spatial position relation information.
According to a third aspect of embodiments of the present disclosure, there is provided an electronic device for image data processing, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the image data processing method according to any of the above embodiments.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the image data processing method according to any one of the above-described embodiments.
According to a fifth aspect of the disclosed embodiments, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the image data processing method according to any of the above embodiments.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects: according to the embodiment of the disclosure, the object attribute characteristic images obtained by acquiring the target object by at least two image acquisition devices and the depth images corresponding to the at least two image acquisition devices are obtained; acquiring depth data of a first pixel in a first depth image corresponding to target image acquisition equipment; mapping the first pixel to an object attribute characteristic image corresponding to the adjacent image acquisition equipment to obtain mapping position information corresponding to the first pixel; determining whether the depth data is abnormal depth data relative to the target image acquisition device and the adjacent image acquisition device according to the first pixel and the mapping position information; and generating an abnormality detection result of the depth data according to whether the depth data is abnormal depth data relative to the target image acquisition device and the adjacent image acquisition devices. Therefore, under a system of at least two image acquisition devices, whether the depth data of the pixels in the depth image corresponding to each image acquisition device is abnormal or not is detected through mapping position information of the first pixels in the depth image acquired by different image acquisition devices to the object attribute feature images corresponding to the adjacent image acquisition devices, so that the abnormal detection precision of the depth data of the pixels in the depth image corresponding to each image acquisition device is improved, the accuracy of the depth data of the pixels in the depth image corresponding to each image acquisition device is further ensured, and accordingly, the confirmation of the image space consistency can be accurately carried out.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure and do not constitute an undue limitation on the disclosure.
Fig. 1 is an application environment diagram illustrating an image data processing method according to an exemplary embodiment.
Fig. 2 is a flowchart illustrating an image data processing method according to an exemplary embodiment.
Fig. 3 is a flow chart illustrating one way of deriving mapping location information for a first pixel according to an exemplary embodiment.
FIG. 4 is a flowchart illustrating the determination of the effectiveness of depth data with respect to a target image capture device and an adjacent image capture device, according to an example embodiment.
Fig. 5 is a block diagram of an image data processing apparatus according to an exemplary embodiment.
Fig. 6 is a block diagram of an electronic device for image data processing, according to an example embodiment.
Detailed Description
In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for presentation, analyzed data, etc.) related to the present disclosure are information and data authorized by the user or sufficiently authorized by each party.
Referring to fig. 1, fig. 1 is an application environment diagram illustrating an image data processing method according to an exemplary embodiment. The application environment may include a client 01 and a server 02. The server 02 may communicate with the client 01 by wired or wireless means, which is not limited by the present disclosure.
The server is used for acquiring object attribute characteristic images acquired by at least two image acquisition devices respectively for a target object and depth images corresponding to the at least two image acquisition devices respectively; the depth data of the first pixels in the first depth image corresponding to the target image acquisition device are acquired; the image processing device is used for mapping the first pixel to an object attribute characteristic image corresponding to the adjacent image acquisition equipment to obtain mapping position information corresponding to the first pixel; and determining whether the depth data is abnormal depth data relative to the target image acquisition device and the adjacent image acquisition device according to the first pixel and the mapping position information; and generating an abnormality detection result of the depth data according to whether the depth data is abnormal depth data with respect to the target image capturing device and the adjacent image capturing device. Optionally, the server 02 may be a server cluster or a distributed system including an independent physical server or a plurality of physical servers, and may also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
The client 01 may include at least two image capturing devices, and the client 01 captures respective object attribute feature images through the at least two image capturing devices. Alternatively, the client may include a smart phone, desktop computer, tablet computer, notebook computer, etc.
It should be noted that fig. 1 is only one application environment of the image data processing method provided in the present disclosure, and in practical application, other application environments may also be included.
Fig. 2 is a flowchart illustrating an image data processing method according to an exemplary embodiment, including the following steps, as shown in fig. 2.
In step S11, an object attribute feature image obtained by acquiring the target object by at least two image acquisition devices respectively is acquired, and a depth image corresponding to each image acquisition device is determined based on the object attribute feature image obtained by each image acquisition device.
Alternatively, the at least two image capturing devices may be cameras, scanners, and other devices with photographing functions. The types of the at least two image capturing devices may be the same or different, and the present disclosure is not particularly limited thereto.
Alternatively, the embodiment of the present disclosure does not specifically limit the installation positions of the at least two image capturing apparatuses. The at least two image capturing devices may or may not be mounted on the same horizontal line. The at least two image acquisition devices each need to be aligned to the target object.
Optionally, the object attribute feature image is acquired by an image acquisition device, and may be an image for reflecting attribute features of an object. The object attribute feature image may be various types of images, to which embodiments of the present disclosure are not particularly limited. For example, the object property feature image may be a texture image, which is a visual feature reflecting a homogeneity phenomenon in the object, that embodies a slowly varying or periodically varying surface structure organization arrangement property of the object surface. The texture image may be, for example, an RGB mode, a gray mode, a black back mode, etc., wherein the RGB mode refers to various colors obtained by changing three color channels of red (R), green (G), blue (B) and overlapping them with each other.
Optionally, the depth image of each image capturing device is obtained by performing depth calculation on the object attribute feature image captured by each image capturing device, where the manner of the depth calculation may be various, and the embodiment of the disclosure is not limited in particular. The depth image of each image acquisition device is obtained by carrying out depth calculation on the object attribute characteristic image acquired by each image acquisition device, so that the pixels in the depth image of each image acquisition device are in one-to-one correspondence with the pixels in the object attribute characteristic image acquired by each image acquisition device.
In step S13, depth data of a first pixel in a first depth image corresponding to a target image acquisition device is acquired; the target image capturing device is any one of the at least two image capturing devices.
Alternatively, any one of the at least two image capturing apparatuses may be used as the target image capturing apparatus. Since the first depth image corresponding to the target image acquisition device is known, the first depth image comprises a plurality of pixels, and the depth data of each pixel in the first depth image can be accurately known from the first depth image. Wherein the depth data of each pixel refers to: distance of each pixel relative to the shooting source (i.e., target image capturing device).
As an example, the embodiment of the present disclosure may detect the validity of the depth data of each pixel in the first depth image corresponding to the target image capturing device, in which case the first pixel may be any one of the pixels included in the first depth image.
As another example, the embodiment of the present disclosure may further divide the first depth image corresponding to the target image capturing device into a plurality of image areas, so as to detect the validity of the depth data of the pixels included in each image area in the dimension of the image area, in which case the first pixels may refer to the pixels included in any one of the image areas.
In step S15, mapping the first pixel to an object attribute feature image corresponding to the adjacent image acquisition device, so as to obtain mapping position information corresponding to the first pixel; the adjacent image pickup apparatuses are image pickup apparatuses other than the target image pickup apparatus among the at least two image pickup apparatuses.
Optionally, in the embodiment of the present disclosure, the first pixel may be mapped to an object attribute feature image corresponding to an adjacent image capturing device, so as to obtain mapping position information corresponding to the first pixel. The adjacent image pickup device is an image pickup device other than the target image pickup device among at least two image pickup devices.
As an example, the mapping position information is located within the field of view of the neighboring image capturing device, i.e. the mapping position information is located in the object attribute feature image corresponding to the neighboring image capturing device. As another example, the mapping location information may appear outside the field of view of the neighboring image capturing device, i.e. the mapping location information is not located in the object property feature image to which the neighboring image capturing device corresponds.
When the first pixel is any one of the pixels included in the first depth image, the mapping position information is position information corresponding to one pixel. In the case where the first pixel is a pixel included in any one of the image areas (the first depth image corresponding to the target image capturing apparatus is divided into a plurality of image areas), the map position information is a map area including: mapping position information of each pixel included in any one of the image areas.
In step S17, it is determined whether the depth data is abnormal depth data with respect to the target image capturing apparatus and the adjacent image capturing apparatus according to the first pixel and the mapping position information.
Alternatively, after the first pixel and the mapping position information are obtained, it may be determined whether the depth data is abnormal depth data with respect to the pair of image capturing devices (the target image capturing device and the adjacent image capturing device) based on the first pixel and the mapping position information. Illustratively, the abnormal depth data refers to valid depth data, and the non-abnormal depth data refers to invalid depth data. Further, valid depth data refers to higher generation accuracy or precision of the depth data (e.g., generation accuracy or precision greater than some preset threshold), and invalid depth data refers to reduced generation accuracy or precision (e.g., generation accuracy or precision less than or equal to the preset threshold).
It should be noted that, according to the first pixel and the mapping position information, there are various ways of determining whether the depth data is abnormal depth data relative to the target image capturing device and the adjacent image capturing device, and the embodiments of the present disclosure are not limited in particular.
In step S19, an abnormality detection result of the depth data is generated according to whether the depth data is abnormal depth data with respect to the target image capturing apparatus and the adjacent image capturing apparatus.
Alternatively, after obtaining whether the depth data is abnormal depth data with respect to the target image capturing apparatus and the adjacent image capturing apparatus, an abnormality detection result of the depth data may be generated according to whether the depth data is abnormal depth data with respect to the target image capturing apparatus and the adjacent image capturing apparatus. The abnormal detection result of the depth data refers to the depth data validity detection result of the first pixel. Further, the validity detection result refers to whether the generation precision or accuracy of the depth data is greater than a preset threshold or less than a preset threshold.
According to the embodiment of the disclosure, under a system of at least two image acquisition devices, whether the depth data of the pixels in the depth image corresponding to each image acquisition device are abnormal or not is detected through mapping position information of mapping a first pixel in the depth image acquired by different image acquisition devices to the object attribute feature image corresponding to the adjacent image acquisition device, so that the abnormality detection precision of the depth data of the pixels in the depth image corresponding to each image acquisition device is improved, the accuracy of the depth data of the pixels in the depth image corresponding to each image acquisition device is further ensured, and therefore confirmation of image space consistency can be accurately carried out. Wherein, spatial consistency refers to: the probability that a point in an image and a point in its surrounding neighborhood have the same category attribute is large, and this characteristic is referred to as the spatially consistent characteristic of the image.
Fig. 3 is a flowchart illustrating obtaining mapping location information corresponding to a first pixel according to an exemplary embodiment, as shown in fig. 3, in an optional embodiment, in the step S15, the mapping the first pixel to an object attribute feature image corresponding to an adjacent image capturing device to obtain mapping location information corresponding to the first pixel may include:
in step S151, spatial position relationship information between the target image capturing apparatus and the adjacent image capturing apparatus is acquired; representing the spatial position relation information; and the mapping relation between the position information of the target object in the world coordinate system and the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition device and the adjacent image acquisition devices respectively.
In step S153, the first pixel is mapped to the object attribute feature image corresponding to the adjacent image capturing device according to the spatial position relation information and the depth data, so as to obtain mapping position information corresponding to the first pixel.
Alternatively, in the above-described step S151, the server may previously establish spatial positional relationship information between the target image capturing apparatus and the adjacent image capturing apparatus. Since the position of the target object in the scene under the world coordinate system is fixed, but since the internal parameters, the external parameters, and the like of the respective image capturing apparatuses are different, the position of the target object in the image coordinate system in the respective image capturing apparatuses is different. Thus, the spatial positional relationship information may refer to: and the mapping relation between the position information of the target object in the world coordinate system and the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition device and the adjacent image acquisition devices respectively.
In one exemplary embodiment, the server may acquire the internal parameters and the external parameters corresponding to the target image capturing device, and the internal parameters and the external parameters corresponding to the neighboring image capturing devices. The internal parameters may refer to parameters related to the characteristics of the image capturing device, such as a focal length, a pixel size, and the like of the image capturing device. The external parameters may refer to parameters of the image acquisition device in the world coordinate system, such as the position, the rotation direction, etc. of the image acquisition device. The server performs calibration processing on the target image acquisition equipment and the adjacent image acquisition equipment according to the inner parameter and the outer parameter corresponding to the target image acquisition equipment and the inner parameter and the outer parameter corresponding to the adjacent image acquisition equipment so as to establish the position information of the target object in the world coordinate system, and the mapping relation between the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition equipment and the adjacent image acquisition equipment respectively, namely, the mapping relation between the world coordinates of the world coordinate system and the image coordinates in the image coordinate system is established, and the spatial position relation information is obtained.
Wherein, the image coordinate system refers to: the image acquired by the image acquisition device may be stored in the computer as an array, the value of each element (pixel) in the array being the brightness (gray) of the image point. A rectangular coordinate system u-v is defined on the image, and the coordinates (u, v) of each pixel are the number of columns and rows, respectively, of the pixel in the array. Therefore, (u, v) is an image coordinate system coordinate in units of pixels. The world coordinate system refers to: a reference coordinate system, called world coordinate system, is also chosen in the environment to describe the position of the image acquisition device and the object. The relationship between the image coordinate system and the world coordinate system can be described by a rotation matrix and a translation vector.
In this embodiment, since the internal parameter may refer to a parameter related to the self characteristic of the image capturing device, the external parameter may refer to a parameter of the image capturing device in the world coordinate system, and spatial position relationship information between different image capturing devices is established through the internal parameter and the external parameter, not only the parameter related to the self characteristic of the image capturing device, but also the parameter of the image capturing device in the world coordinate system is fully considered, and the establishment accuracy of the spatial position relationship information is improved.
Optionally, taking the first pixel as an example of any one of the pixels included in the first depth image, in step S153, after the server obtains the depth data of the first pixel, the position information of the first pixel in the first depth image may be accurately obtained according to the depth data, and since the pixels in the first depth image and the pixels in the object attribute feature image that generate the first pixel are in one-to-one correspondence, the position information of the pixels in the corresponding object attribute feature image may be obtained. Characterization by virtue of spatial position relation information; after the position information of the first pixel in the corresponding object attribute feature image is obtained, the position information in the world coordinate system corresponding to the position information of the first pixel in the corresponding object attribute feature image can be obtained according to the spatial position relation information, and the position information in the world coordinate system can obtain the mapping position information of the position information in the world coordinate system in the object attribute feature image corresponding to the adjacent image acquisition device according to the internal parameters and the external parameters of the adjacent image acquisition device, so as to obtain the mapping position information of the first pixel. The specific calculation formula can be:
q＝f(p,D p )；
Where q denotes mapping position information, f denotes spatial position relation information, p denotes a first pixel, D p Refers to depth data of p.
For example, the first pixel is p, its depth data is D p And forming position information 1 of the first pixel in the corresponding object attribute characteristic image according to the depth data and the first pixel. After the position information 1 is obtained, the position information 2 of the position information 1 in the world coordinate system can be obtained based on the above-described spatial position relationship information. The position information 2 obtains position information 3 according to the internal parameters and the external parameters of the adjacent image acquisition equipment, and the position information 3 is the mapping position information corresponding to the first pixel.
Taking the first pixel as an example of a pixel included in any one of the image areas (the first depth image corresponding to the target image capturing apparatus is divided into a plurality of image areas), the mapping position information is a mapping area, and the mapping area includes: mapping position information of each pixel included in any one of the image areas. Then, in the step S153, after the server obtains the depth data of the first pixel, the arbitrary image area may be accurately obtained according to the depth data, and the position information in the first depth image may be the pixel in the first depth image and the pixel in the object attribute feature image that generates the pixel are in one-to-one correspondence, so that the first target image area of the arbitrary image area in the corresponding object attribute feature image may be obtained. Characterization by virtue of spatial position relation information; after the first target image area is obtained, a second target image area in the world coordinate system corresponding to the first target image area can be obtained according to the spatial position relation information, and therefore a mapping area of the second target image area in the object attribute feature image corresponding to the adjacent image acquisition equipment is obtained according to the internal parameters and the external parameters of the second target image area and the adjacent image acquisition equipment, and further the mapping position information corresponding to the first pixel is obtained.
In this embodiment, due to the establishment of the spatial position relationship information, not only parameters related to the characteristics of the image acquisition device are fully considered, but also parameters of the image acquisition device in the world coordinate system are fully considered, and the establishment precision of the spatial position relationship information is high, and by using the spatial position relationship information and the depth data with high precision, the mapping position information after the mapping of the first pixel can be accurately obtained, so that the generation precision of the mapping position information is improved, and the determination precision of the abnormal detection result of the depth data is further improved.
Fig. 4 is a flowchart illustrating determining effectiveness of depth data with respect to a target image capturing device and an adjacent image capturing device according to an exemplary embodiment, and as shown in fig. 4, in an alternative embodiment, the number of adjacent image capturing devices is at least two, and in step S17, determining whether the depth data is abnormal depth data with respect to the target image capturing device and the adjacent image capturing device according to the first pixel and the mapping position information may include:
in step S171, determining, from at least two of the adjacent image capturing devices, a first adjacent image capturing device in which mapping position information exists in the object attribute feature image; and determining that the mapping position information does not exist in the object attribute feature image.
In step S173, determining that the depth data is illegal depth data with respect to the target image capturing device and the second neighboring image capturing device, and determining whether the depth data is abnormal depth data with respect to the target image capturing device and the first neighboring image capturing device according to the similarity between the first pixel and the second pixel; the second pixel is a pixel corresponding to mapping position information existing in the object attribute feature image of the first adjacent image capturing device.
The number of the adjacent image capturing devices is at least two. Taking the first pixel as any one pixel in the first depth image as an example, in the step S171, since the mapping position information may be located in the field of view of the neighboring image capturing device, that is, in the object attribute feature image corresponding to the neighboring image capturing device, when the first pixel is mapped to the object attribute feature image corresponding to each neighboring image capturing device, the mapping position information may also be located outside the field of view of the neighboring image capturing device, that is, not located in the object attribute feature image corresponding to the neighboring image capturing device, the server may determine, as the first neighboring image capturing device, the neighboring image capturing device in which the mapping position information exists in the object attribute feature image; and determining that the adjacent image acquisition equipment with mapping position information does not exist in the object attribute characteristic image as second adjacent image acquisition equipment. For example, the neighboring image capturing apparatuses are k: t (T) 0 ～T k . If the mapping position information is located at T 0 ～T k-2 In the corresponding object attribute feature image, T is taken as 0 ～T k-2 As the first neighboring image capturing device, if the mapping position information is located at T k-2 ～T k In the corresponding object attribute feature image, T is taken as k-2 ～T k As a second adjacent image acquisition device. The sum of the number of first neighboring image capturing devices and the second neighboring image capturing device is equal to k.
Taking the first pixel as an example of a pixel included in any one of the image areas (in which the first depth image corresponding to the target image capturing device is divided into a plurality of image areas), the mapping position information is a mapping area, and the mapping area includes: mapping position information of each pixel included in any one of the image areas. In the step S171, since the mapping area may be located within the field of view of the neighboring image capturing device, that is, within the object attribute feature image corresponding to the neighboring image capturing device, when the first pixel is mapped into the object attribute feature image corresponding to each neighboring image capturing device, the mapping area may be located outside the field of view of the neighboring image capturing device, that is, not within the object attribute feature image corresponding to the neighboring image capturing device. The server can determine adjacent image acquisition equipment with a mapping area in the object attribute characteristic image as first adjacent image acquisition equipment; and determining that the adjacent image acquisition equipment with no mapping area in the object attribute characteristic image is a second adjacent image acquisition equipment.
Optionally, in the above step S173, for the second adjacent image capturing device, since the depth data is not located within the field of view of the second adjacent image capturing device, the depth data is illegal depth data between the pair of image capturing devices (including the target image capturing device and the second adjacent image capturing device). For a first neighboring image capture device, the depth data is non-anomalous depth data between pairs of image capture devices (including the target image capture device and the first neighboring image capture device) because the depth data is within the field of view of the first neighboring image capture device. The server may acquire a second pixel of the pixel corresponding to the mapping position information existing in the object attribute feature image of the first neighboring image capturing device, and determine whether the depth data is abnormal depth data with respect to the target image capturing device and the first neighboring image capturing device according to the similarity between the first pixel and the second pixel.
It should be noted that if the first pixel is any one of the pixels included in the first depth image, the second pixel is any one of the pixels included in the object attribute feature image of the first neighboring image capturing device. If the first pixel is a pixel included in any one of the image areas (wherein the first depth image corresponding to the target image capturing device is divided into a plurality of image areas), the mapping position information is a mapping area, and the mapping area includes: the mapping position information of each pixel included in any one image area is: each pixel contained in the mapped region present in the object attribute feature image of the first neighboring image acquisition device.
In this embodiment, whether the depth data is abnormal depth data with respect to the pair of image capturing devices is determined according to whether the mapping position information is located in the object attribute feature image of the adjacent image capturing device, that is, whether the mapping position information is located in the field of view of the adjacent image capturing device, and whether the depth data is abnormal can be reflected to a great extent, so that determination accuracy of effectiveness of the depth data is improved.
It should be noted that, in step S173, according to the similarity between the first pixel and the second pixel, determining whether the depth data is abnormal depth data with respect to the target image capturing device and the first neighboring image capturing device may be implemented in a variety of ways, which is not specifically limited herein. In one possible embodiment, continuing with fig. 4, the step S173 may include:
in step S1731, a first object attribute feature corresponding to a first pixel is acquired from an object attribute feature image corresponding to a target image capturing apparatus.
In step S1733, second object attribute features corresponding to the second pixels are acquired from the object attribute feature images corresponding to the first neighboring image capturing devices.
In step S1735, it is determined whether the depth data is abnormal depth data with respect to the target image capture device and the first neighboring image capture device according to the similarity between the first object attribute feature and the second object attribute feature.
Optionally, the depth image of each image acquisition device is obtained by performing depth calculation on the object attribute feature image acquired by each image acquisition device, and the pixels in the depth image of each image acquisition device are in one-to-one correspondence with the pixels in the object attribute feature image acquired by each image acquisition device. Therefore, in step S1731-step S1733, the server may obtain the first object attribute feature corresponding to the first pixel from the object attribute feature image corresponding to the target image capturing device, that is, the first object attribute feature corresponding to the position of the first pixel in the object attribute feature image of the target image capturing device. The server may further obtain a second object attribute feature corresponding to the second pixel from the object attribute feature image corresponding to the first neighboring image capturing device, that is, a second object attribute feature corresponding to a position of the second pixel in the object attribute feature image of the first neighboring image capturing device. In the above-described step S1735, the server may determine whether the depth data is abnormal depth data with respect to the target image capturing device and the first neighboring image capturing device according to the similarity between the first object attribute feature and the second object attribute feature.
It should be noted that the similarity may be calculated by various ways, and is not limited herein, for example, cosine similarity, euclidean distance, and the like.
In one embodiment, if the first pixel is any one of the pixels contained in the first depth image, the second pixel is any one of the pixels contained in the object property feature image of the first neighboring image acquisition device. Accordingly, the first object attributeThe feature and the second object attribute feature are both attribute features corresponding to one pixel. Taking an object attribute feature image as a texture image as an example, wherein a first pixel is p, a second pixel is q, and the object attribute feature image corresponding to the target image acquisition equipment is a texture image I i The object attribute characteristic image corresponding to the first adjacent image acquisition equipment is texture image I j The first object property feature may beThe second object property feature may be +.>The server can calculate the texture map +.>And->Performing similarity calculation to obtain similarity ∈>
In another embodiment, if the first pixel is taken as an example of a pixel included in any one of the image areas (in which the first depth image corresponding to the target image capturing device is divided into a plurality of image areas), the mapping position information is a mapping area, and the mapping area includes: the mapping position information of each pixel included in any one image area is: each pixel contained in the mapped region present in the object attribute feature image of the first neighboring image acquisition device. Correspondingly, the first object attribute feature is any one image area, the attribute feature contained in the object attribute feature image corresponding to the target image acquisition device, the second object attribute feature is a mapping area, and the attribute feature contained in the object attribute feature image corresponding to the first adjacent image acquisition device. The server may calculate the similarity between the attribute features contained in any one of the image regions and the attribute features contained in the map region. For example, a first average value of the attribute features contained in any one image region may be calculated first, a second average value of the attribute features contained in the mapping region may be calculated, and a similarity between the first average value and the second average value may be calculated. And the similarity between the attribute features contained in any one image region and the corresponding attribute features in the attribute features contained in the mapping region can be calculated to obtain the similarity corresponding to each attribute feature contained in any one image region, and then the average value between the similarities corresponding to each attribute feature is calculated to obtain the final similarity.
In this embodiment, due to the similarity of the object attribute features of different pixels, the spatial consistency of the different pixels can be reflected to a great extent, so for the first adjacent image acquisition device existing in the mapping position information, according to the second object attribute feature corresponding to the mapping position information in the object attribute feature image, the similarity between the first object attribute features corresponding to the first pixels determines whether the depth data is abnormal depth data, so that the accuracy of determining the depth data can be further improved, and therefore the image can be ensured to maintain the spatial consistency, that is, the accuracy of determining the spatial consistency of the image is further improved.
In an exemplary embodiment, continuing to fig. 4, the determining whether the depth data is abnormal depth data with respect to the target image capturing device and the first neighboring image capturing device according to the similarity between the first object attribute feature and the second object attribute feature in the above step S1735 includes:
in step S17351, from at least two second object attribute features, second target object attribute features whose similarity with the first object attribute feature satisfies a preset condition are determined; and determining a target object attribute feature image in which the second target object attribute feature is located.
In step S17353, a first target neighboring image capturing device that captures a target object attribute feature image is determined from at least two first neighboring image capturing devices.
In step S17355, the depth data is determined to be non-abnormal depth data with respect to the target image capturing device and the first target adjacent image capturing device.
Alternatively, the number of the first neighboring image capturing devices may be at least two, and thus, the number of the second object attribute features may also be at least two. In the above step S17351, the server may determine a second target object attribute feature, of which the similarity with the first object attribute feature satisfies a preset condition, from among the at least two second object attribute features. The preset condition may be that the similarity is greater than a preset similarity threshold, or that the similarity is within a preset range, for example. After obtaining the second target object property feature, the server may determine a target object property feature image in which the second target object property feature is located.
Alternatively, in the above-described step S17353-step S17355, the server may determine a first target neighboring image capturing device that captures an attribute feature image of the target object from among at least two first neighboring image capturing devices, and determine the depth data as non-abnormal depth data with respect to the target image capturing device and the first target neighboring image capturing device.
It should be noted that, for the second object attribute feature whose similarity with the first object attribute feature does not satisfy the preset condition, the depth data is considered as abnormal depth data.
The similarity of object attribute characteristics of different pixels can reflect the consistency of the different pixels in space to a great extent, so that the similarity meets the preset condition as the basis for judging that the depth data is abnormal depth data, the determination accuracy of the abnormality of the depth data determination can be further improved, and the image can be further ensured to be kept consistent in space, namely, the determination accuracy of the spatial consistency of the image is further improved.
In an optional embodiment, in step S19, generating the anomaly detection result of the depth data according to the validity of the depth data with respect to the target image capturing device and the adjacent image capturing device may include:
and under the condition that the number of the first target adjacent image acquisition devices is larger than or equal to a first preset number times of the number of the adjacent image acquisition devices and the number of the second adjacent image acquisition devices is smaller than a second preset number times of the number of the adjacent image acquisition devices, determining that the depth data is non-abnormal depth data, and obtaining an abnormal detection result of the depth data.
In one embodiment, the first target neighboring image capture device may be referred to as an active image capture device S valid The second adjacent image acquisition device is taken as an abnormal image acquisition device S illegal And two constants alpha and beta are preset, and the alpha and beta can be set according to actual service requirements, which is not particularly limited. The first preset number of times may be represented by α and the second preset number of times may be represented by β. Illustratively, if S valid Not less than alpha.k and S illegal < β·k, where k refers to the number of all neighboring image acquisition devices, then the depth data D is considered p The abnormal detection result of the depth data is obtained, so that the number of effective image acquisition devices and the number of abnormal image acquisition devices are fully considered in the abnormality detection process of the depth data, and therefore the non-abnormal depth data of pixels in the target image acquisition device is efficiently and accurately detected through interaction between at least two image acquisition devices under the system of at least two image acquisition devices. In the case that the first pixel is any one of the pixels included in the first depth image, the embodiment of the disclosure can also realize whether the depth data of each pixel in the first depth image is abnormal or not, so that the abnormality detection range of the depth data is improved, and further, the spatial consistency of the images is ensured.
In another embodiment, the server may further compare the number of the first target neighboring image capturing devices and the number of the second neighboring image capturing devices with a certain threshold, and determine that the depth data is non-abnormal depth data if the comparison result of the two meets a certain preset number of conditions, so as to obtain an abnormal detection result of the depth data.
In a third embodiment, the server may further calculate a ratio between the number of the first target neighboring image capturing devices and the number of the second neighboring image capturing devices, and if the ratio satisfies a certain preset ratio condition, determine that the depth data is non-abnormal depth data, and obtain an abnormal detection result of the depth data.
In an optional embodiment, in step S19, generating the anomaly detection result of the depth data according to whether the depth data is the anomaly depth data with respect to the target image capturing device and the adjacent image capturing device may include:
and determining the depth data as abnormal depth data under the condition that the number of the first target adjacent image acquisition devices and the number of the second adjacent image acquisition devices meet any one of the first condition, the second condition and the third condition, and obtaining an abnormal detection result of the depth data.
The first condition is the number of first target adjacent image capturing devices being greater than or equal to a first preset number multiple of the number of adjacent image capturing devices and the number of second adjacent image capturing devices being greater than or equal to a second preset number multiple of the number of adjacent image capturing devices.
The second condition is that the number of the first target adjacent image capturing devices is less than or equal to a first preset number times the number of the adjacent image capturing devices, and the number of the second adjacent image capturing devices is less than a second preset number times the number of the adjacent image capturing devices.
The third condition is that the number of the first target adjacent image capturing devices is less than or equal to a first preset number times the number of the adjacent image capturing devices, and the number of the second adjacent image capturing devices is greater than or equal to a second preset number times the number of the adjacent image capturing devices.
In one embodiment, the first target neighboring image capture device may be considered as an active image capture deviceCollecting device S valid The second adjacent image acquisition device is taken as an abnormal image acquisition device S illegal And two constants alpha and beta are preset, and the alpha and beta can be set according to actual service requirements, which is not particularly limited. The first preset number of times may be represented by α and the second preset number of times may be represented by β. Illustratively, the first condition is: s is S valid Not less than alpha.k and S illegal And (3) not less than beta.k. The second condition is: s is S valid Alpha.k is less than or equal to S illegal < beta.k. The third condition is: s is S valid Alpha.k is less than or equal to S illwgal ≥β·k。
According to the embodiment, the number of effective image acquisition devices and the number of abnormal image acquisition devices are fully considered in the abnormality detection process of the depth data, so that abnormal depth data of pixels in the target image acquisition device is efficiently and accurately detected through interaction between at least two image acquisition devices under the system of at least two image acquisition devices.
In another embodiment, the server may further compare the number of the first target adjacent image capturing devices and the number of the second adjacent image capturing devices with a certain threshold respectively, and if the comparison result of the two does not meet the preset number condition, determine that the depth data is abnormal depth data, so as to obtain an abnormal detection result of the depth data.
In a third embodiment, the server may further calculate a ratio between the number of the first target neighboring image capturing devices and the number of the second neighboring image capturing devices, and if the ratio does not meet a preset ratio condition, determine that the depth data is abnormal depth data, and obtain an abnormal detection result of the depth data.
It should be noted that, after obtaining the non-abnormal depth data and the abnormal depth data, the non-abnormal depth data may be used as input of other algorithms, which is not specifically limited in the embodiments of the present disclosure.
In the following, the number of at least two image capturing devices is 6, and the target image capturing device is one of the 6 image capturing devicesPersonal (with C) i Indicated), the number of adjacent image acquisition devices is 5 (T) 1 、T 2 、T 3 、T 4 、T 5 ) The object attribute feature image is a texture image, the first pixel is any pixel contained in the first depth image, and the image data processing method is described as follows:
1) And obtaining texture images obtained by respectively collecting the target objects by the 6 image collecting devices. And depth images corresponding to the 6 image acquisition devices respectively.
2) Acquisition of C i Depth data D of any one first pixel p among pixels included in the corresponding first depth image p . C i Respectively with T 1 、T 2 、T 3 、T 4 、T 5 Spatial positional relationship information between the two.
3) Mapping the first pixel to T according to the spatial position relation information and the depth data 1 、T 2 、T 3 、T 4 、T 5 Respectively corresponding texture images to obtain a first pixel aiming at T 1 Mapping position information 1 for T 2 Mapping position information 2 for T 3 Mapping position information 3 for T 4 Mapping location information 4 for T 5 Mapping position information 5 of (a).
4) Assume that mapping position information 1 is located at T 1 In the corresponding texture image, the mapping position information 2 is located at T 2 In the corresponding texture image, the mapping position information 3 is located at T 3 In the corresponding texture image, the mapping position information 4 is not located at T 4 In the corresponding texture image, the mapping position information 5 is not located at T 5 And in the corresponding texture image. Then determine T 1 、T 2 、T 3 For the first adjacent image acquisition device, T 4 、T 5 Is a second adjacent image acquisition device. And determining D p For relative to the image acquisition device pair (C i And T 4 Image acquisition equipment pair and C i And T 5 A constituent pair of image acquisition devices) that can be considered an outlier mapImage acquisition equipment S illegal 。
5) Then determine D p Whether or not to compare with the target image capturing device and the first adjacent image capturing device (T 1 、T 2 、T 3 ) Is a part of the abnormal depth data:
acquisition of D p Corresponding first texture features, from T 1 Acquiring a second texture feature 1 of a second pixel corresponding to mapping position information 1 from the corresponding texture image, wherein the second texture feature 1 is represented by T 2 Acquiring a second texture feature 2 of a second pixel corresponding to the mapping position information 2 from the corresponding texture image, and obtaining a second texture feature from T 3 And acquiring a second texture characteristic 3 of a second pixel corresponding to the mapping position information 3 from the corresponding texture image.
The similarity 1 between the first texture feature and the second texture feature 1 is calculated, the similarity 2 between the first texture feature and the second texture feature 2 is calculated, and the similarity 3 between the first texture feature and the second texture feature 3 is calculated.
Assuming that the similarity 1 and the similarity 2 are larger than a preset threshold value, and the similarity 3 is smaller than or equal to a preset threshold value, taking the second texture feature 1 and the second texture feature 2 as second target object attribute features, determining a texture image in which the second texture feature 1 is positioned, determining a texture image in which the second texture feature 2 is positioned, and determining a T for acquiring the texture image in which the second texture feature 1 is positioned 1 And acquiring a T of a texture image in which the second texture feature 2 is located 2 Obtaining a first target adjacent image acquisition device, namely an effective image acquisition device S valid . At the same time determining the T of the texture image in which the second texture feature 3 is acquired 3 . Finally determining D p Is relative to C i And a first target adjacent image acquisition device (T 1 、T 2 ) Is not abnormal depth data of (a). Determining D p Is relative to C i And T 3 Is a part of the depth data of the object.
6) Constants α, β are preset, and the above values are substituted into the following formula:
If S valid Not less than alpha.k and S illegal < β.k, k is the number of neighboring image capturing devices (i.e., 5), then D is considered p Is non-outlier depth data.
If S valid Not less than alpha.k and S illegal Not less than beta.k, or S valid Alpha.k is less than or equal to S illegal < beta.k, or S valid Alpha.k is less than or equal to S illegal If ∈k is not less than β.k, then it is considered that D p Is the abnormal depth data.
Fig. 5 is a block diagram of an image data processing apparatus according to an exemplary embodiment. Referring to fig. 5, the apparatus includes:
an image acquisition module 21 configured to perform acquisition of object attribute feature images acquired by at least two image acquisition devices for each of the target objects, and determine a depth image corresponding to each of the image acquisition devices based on the object attribute feature images acquired by each of the image acquisition devices;
a depth data acquisition module 23 configured to perform acquisition of depth data of a first pixel in a first depth image corresponding to a target image capturing device; the target image acquisition device is any one of at least two image acquisition devices;
a mapping position information generating module 25 configured to perform mapping of the first pixel to the object attribute feature image corresponding to the adjacent image capturing device, so as to obtain mapping position information corresponding to the first pixel; the adjacent image acquisition devices are the image acquisition devices except the target image acquisition device in at least two image acquisition devices;
An abnormal depth data determining module 27 configured to perform determination of whether the depth data is abnormal depth data with respect to the target image capturing apparatus and the adjacent image capturing apparatus according to the first pixel and the mapping position information;
an abnormality detection result generation module 29 configured to perform generation of an abnormality detection result of the depth data according to whether the depth data is abnormal depth data with respect to the target image capturing device and the adjacent image capturing device.
In an alternative embodiment, the number of adjacent image capturing devices is at least two, and the abnormal depth data determining module includes:
a neighboring image capture device determination submodule configured to perform a first neighboring image capture device that determines that mapping location information exists in the object attribute feature image from at least two neighboring image capture devices; and determining that the mapping position information does not exist in the object attribute feature image;
an anomaly determination sub-module configured to perform determining that the depth data is illegal depth data with respect to the target image capturing device and the second neighboring image capturing device, and determining whether the depth data is anomaly depth data with respect to the target image capturing device and the first neighboring image capturing device according to a similarity between the first pixel and the second pixel; the second pixel is a pixel corresponding to mapping position information existing in the object attribute feature image of the first adjacent image capturing device.
In an alternative embodiment, the anomaly determination submodule includes:
a first object attribute feature acquiring unit configured to acquire a first object attribute feature corresponding to a first pixel from an object attribute feature image corresponding to a target image capturing device;
a second object attribute feature acquisition unit configured to perform acquisition of a second object attribute feature corresponding to a second pixel from an object attribute feature image corresponding to the first neighboring image acquisition device;
and a similarity judging unit configured to perform determination as to whether the depth data is abnormal depth data with respect to the target image capturing apparatus and the first neighboring image capturing apparatus, based on the similarity between the first object attribute feature and the second object attribute feature.
In an alternative embodiment, the similarity determining unit includes:
a target object attribute determination subunit configured to perform determining, from at least two second object attribute features, a second target object attribute feature whose similarity with the first object attribute feature satisfies a preset condition; determining a target object attribute feature image in which the second target object attribute feature is located;
A first target adjacent image capturing device determining sub-module configured to perform determining a first target adjacent image capturing device that captures a target object attribute feature image from at least two first adjacent image capturing devices;
and an effective depth data generation subunit configured to perform determining the depth data as non-abnormal depth data with respect to the target image capturing device and the first target neighboring image capturing device.
In an alternative embodiment, the abnormality detection result generation module is configured to perform:
and under the condition that the number of the first target adjacent image acquisition devices is larger than or equal to a first preset number times of the number of the adjacent image acquisition devices and the number of the second adjacent image acquisition devices is smaller than a second preset number times of the number of the adjacent image acquisition devices, determining that the depth data is non-abnormal depth data, and obtaining an abnormal detection result of the depth data.
In an alternative embodiment, the abnormality detection result generation module is configured to perform:
determining the depth data as abnormal depth data under the condition that the number of the first target adjacent image acquisition devices and the number of the second adjacent image acquisition devices meet any one of the first condition, the second condition and the third condition, and obtaining an abnormal detection result of the depth data;
The first condition is the number of the first target adjacent image acquisition devices being greater than or equal to a first preset number times the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices being greater than or equal to a second preset number times the number of the adjacent image acquisition devices;
the second condition is that the number of the first target adjacent image acquisition devices is smaller than or equal to a first preset number times the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is smaller than a second preset number times the number of the adjacent image acquisition devices;
the third condition is that the number of the first target adjacent image capturing devices is less than or equal to a first preset number times the number of the adjacent image capturing devices, and the number of the second adjacent image capturing devices is greater than or equal to a second preset number times the number of the adjacent image capturing devices.
In an alternative embodiment, the mapping location information generating module includes:
a spatial positional relationship information acquisition unit configured to perform acquisition of spatial positional relationship information between the target image capturing apparatus and the adjacent image capturing apparatus; representing the spatial position relation information; the position information of the target object in the world coordinate system and the mapping relation between the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition equipment and the adjacent image acquisition equipment respectively;
And the mapping unit is configured to map the first pixel to the object attribute characteristic image corresponding to the adjacent image acquisition equipment according to the spatial position relation information and the depth data, so as to obtain the mapping position information corresponding to the first pixel.
In an alternative embodiment, the spatial position relation information obtaining unit includes:
a parameter acquisition subunit configured to perform acquisition of an internal parameter and an external parameter corresponding to the target image acquisition device, and an internal parameter and an external parameter corresponding to the adjacent image acquisition device;
the establishing subunit is configured to establish the position information of the target object in the world coordinate system according to the inner parameter and the outer parameter corresponding to the target image acquisition device and the inner parameter and the outer parameter corresponding to the adjacent image acquisition device, and the mapping relation between the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition device and the adjacent image acquisition device respectively, so as to obtain the spatial position relation information.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
In an exemplary embodiment, there is also provided an electronic device including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of any of the image data processing methods of the above embodiments when executing instructions stored on the memory.
The electronic device may be a terminal, a server or similar computing device, which is exemplified by a server, fig. 6 is a block diagram of an electronic device for image data processing according to an exemplary embodiment, the electronic device 30 may vary considerably according to configuration or performance, and may include one or more central processing units (Central Processing Units, CPU) 31 (the central processing unit 31 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 33 for storing data, one or more storage media 32 (e.g., one or more mass storage devices) storing applications 323 or data 322. Wherein the memory 33 and the storage medium 32 may be transitory or persistent. The program stored on the storage medium 32 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Still further, the central processor 31 may be arranged to communicate with a storage medium 32, and to execute a series of instruction operations in the storage medium 32 on the electronic device 30. The electronic device 30 may also include one or more power supplies 36, one or more wired or wireless network interfaces 35, one or more input/output interfaces 34, and/or one or more operating systems 321, such as Windows ServerTM, mac OS XTM, unixTM, linuxTM, freeBSDTM, or the like.
The input-output interface 34 may be used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of electronic device 30. In one example, the input-output interface 34 includes a network adapter (Network Interface Controller, NIC) that can connect to other network devices through a base station to communicate with the internet. In one exemplary embodiment, the input-output interface 34 may be a Radio Frequency (RF) module for communicating with the internet wirelessly.
It will be appreciated by those of ordinary skill in the art that the configuration shown in fig. 6 is merely illustrative and is not intended to limit the configuration of the electronic device described above. For example, electronic device 30 may also include more or fewer components than shown in FIG. 6, or have a different configuration than shown in FIG. 6.
In an exemplary embodiment, a computer readable storage medium is also provided, which when executed by a processor of an electronic device, causes the electronic device to perform the steps of any of the image data processing methods of the above embodiments.
In an exemplary embodiment, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements the image data processing method provided in any of the above embodiments.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided by the present disclosure may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (11)

1. An image data processing method, comprising:
acquiring object attribute feature images acquired by at least two image acquisition devices respectively for acquiring a target object, and determining a depth image corresponding to each image acquisition device based on the object attribute feature images acquired by each image acquisition device;
acquiring depth data of a first pixel in a first depth image corresponding to target image acquisition equipment; the target image acquisition device is any one of at least two image acquisition devices;
Mapping the first pixel to an object attribute characteristic image corresponding to an adjacent image acquisition device to obtain mapping position information corresponding to the first pixel; the adjacent image acquisition devices are the image acquisition devices except the target image acquisition device in at least two image acquisition devices;
determining whether the depth data is abnormal depth data relative to the target image acquisition device and the adjacent image acquisition device according to the first pixel and the mapping position information;
and generating an abnormality detection result of the depth data according to whether the depth data is abnormal depth data relative to the target image acquisition device and the adjacent image acquisition device.
2. The method of claim 1, wherein the number of neighboring image capture devices is at least two, and wherein determining whether the depth data is abnormal depth data with respect to the target image capture device and the neighboring image capture device based on the first pixel and the mapping position information comprises:
determining a first adjacent image acquisition device with mapping position information in the object attribute characteristic image from at least two adjacent image acquisition devices; and determining that the mapping position information does not exist in the object attribute feature image;
Determining that the depth data is illegal depth data relative to the target image acquisition device and the second adjacent image acquisition device, and determining whether the depth data is abnormal depth data relative to the target image acquisition device and the first adjacent image acquisition device according to the similarity between the first pixel and the second pixel; the second pixel is a pixel corresponding to mapping position information existing in the object attribute feature image of the first adjacent image acquisition device.
3. The method of claim 2, wherein the determining whether the depth data is abnormal depth data with respect to the target image capture device and the first neighboring image capture device based on the similarity between the first pixel and the second pixel comprises:
acquiring a first object attribute feature corresponding to the first pixel from an object attribute feature image corresponding to the target image acquisition device;
acquiring a second object attribute feature corresponding to the second pixel from an object attribute feature image corresponding to the first adjacent image acquisition device;
and determining whether the depth data is abnormal depth data relative to the target image acquisition device and the first adjacent image acquisition device according to the similarity between the first object attribute feature and the second object attribute feature.
4. A method according to claim 3, wherein the number of the first neighboring image capturing device and the second object attribute feature is at least two, and the determining whether the depth data is abnormal depth data with respect to the target image capturing device and the first neighboring image capturing device according to the similarity between the first object attribute feature and the second object attribute feature comprises:
determining second target object attribute characteristics with similarity meeting preset conditions between the second target object attribute characteristics and the first object attribute characteristics from at least two second object attribute characteristics; determining a target object attribute feature image in which the second target object attribute feature is located;
determining first target adjacent image acquisition equipment for acquiring attribute characteristic images of the target object from at least two first adjacent image acquisition equipment;
the depth data is determined to be non-abnormal depth data relative to the target image acquisition device and the first target neighboring image acquisition device.
5. The method of claim 4, wherein generating the anomaly detection result for the depth data based on whether the depth data is anomaly depth data relative to the target image capture device and the neighboring image capture device comprises:
And determining that the depth data is non-abnormal depth data under the condition that the number of the first target adjacent image acquisition devices is larger than or equal to a first preset number times of the number of the adjacent image acquisition devices and the number of the second adjacent image acquisition devices is smaller than a second preset number times of the number of the adjacent image acquisition devices, so as to obtain an abnormal detection result of the depth data.
6. The method of claim 4, wherein generating the anomaly detection result for the depth data based on whether the depth data is anomaly depth data relative to the target image capture device and the neighboring image capture device comprises:
determining that the depth data is abnormal depth data under the condition that the number of the first target adjacent image acquisition devices and the number of the second adjacent image acquisition devices meet any one of a first condition, a second condition and a third condition, and obtaining an abnormal detection result of the depth data;
the first condition is that the number of the first target adjacent image acquisition devices is greater than or equal to a first preset number times the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is greater than or equal to a second preset number times the number of the adjacent image acquisition devices;
The second condition is that the number of the first target adjacent image acquisition devices is smaller than or equal to a first preset number times the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is smaller than a second preset number times the number of the adjacent image acquisition devices;
the third condition is that the number of the first target adjacent image acquisition devices is smaller than or equal to a first preset number multiple of the number of the adjacent image acquisition devices, and the number of the second adjacent image acquisition devices is larger than or equal to a second preset number multiple of the number of the adjacent image acquisition devices.
7. The method according to any one of claims 1 to 6, wherein mapping the first pixel to an object attribute feature image corresponding to an adjacent image capturing device, to obtain mapping location information corresponding to the first pixel, includes:
acquiring spatial position relation information between the target image acquisition equipment and the adjacent image acquisition equipment; the spatial position relation information is characterized; the mapping relation between the position information of the target object in the world coordinate system and the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition equipment and the adjacent image acquisition equipment respectively;
And mapping the first pixel to an object attribute characteristic image corresponding to the adjacent image acquisition equipment according to the spatial position relation information and the depth data to obtain mapping position information corresponding to the first pixel.
8. The method of claim 7, wherein the acquiring spatial positional relationship information between the target image capturing device and the neighboring image capturing device comprises:
acquiring the inner parameters and the outer parameters corresponding to the target image acquisition equipment and the inner parameters and the outer parameters corresponding to the adjacent image acquisition equipment;
and establishing the position information of the target object in a world coordinate system according to the internal parameters and the external parameters corresponding to the target image acquisition equipment and the internal parameters and the external parameters corresponding to the adjacent image acquisition equipment, and obtaining the spatial position relation information according to the mapping relation between the position information of the target object in the object attribute characteristic images corresponding to the target image acquisition equipment and the adjacent image acquisition equipment.
9. An image data processing apparatus, comprising:
the image acquisition module is configured to acquire object attribute feature images acquired by at least two image acquisition devices respectively on a target object, and determine a depth image corresponding to each image acquisition device based on the object attribute feature images acquired by each image acquisition device;
A depth data acquisition module configured to perform acquisition of depth data of a first pixel in a first depth image corresponding to a target image acquisition device; the target image acquisition device is any one of at least two image acquisition devices;
the mapping position information generation module is configured to perform mapping of the first pixel to an object attribute characteristic image corresponding to the adjacent image acquisition equipment to obtain mapping position information corresponding to the first pixel; the adjacent image acquisition devices are the image acquisition devices except the target image acquisition device in at least two image acquisition devices;
an abnormal depth data determining module configured to perform determining whether the depth data is abnormal depth data with respect to the target image capturing device and the adjacent image capturing device according to the first pixel and the mapping position information;
an abnormality detection result generation module configured to perform generation of an abnormality detection result of the depth data according to whether the depth data is abnormal depth data of the target image capturing device and the adjacent image capturing device.
10. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the image data processing method of any one of claims 1 to 8.
11. A computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the image data processing method of any one of claims 1 to 8.
CN202310692865.8A 2023-06-12 2023-06-12 Image data processing method, device, equipment and storage medium Pending CN116912171A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310692865.8A CN116912171A (en) 2023-06-12 2023-06-12 Image data processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310692865.8A CN116912171A (en) 2023-06-12 2023-06-12 Image data processing method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116912171A true CN116912171A (en) 2023-10-20

Family

ID=88361769

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310692865.8A Pending CN116912171A (en) 2023-06-12 2023-06-12 Image data processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116912171A (en)

Similar Documents

Publication Publication Date Title
JP6722323B2 (en) System and method for imaging device modeling and calibration
CN115294409B (en) Video processing method, system and medium for security monitoring
CN112258579B (en) Image processing method, image processing device, electronic equipment and computer readable storage medium
US20190188452A1 (en) Image processing method and apparatus, and electronic device
EP3598385B1 (en) Face deblurring method and device
US11282232B2 (en) Camera calibration using depth data
CN106981078B (en) Sight line correction method and device, intelligent conference terminal and storage medium
JP2020523703A (en) Double viewing angle image calibration and image processing method, device, storage medium and electronic device
CN112991180B (en) Image stitching method, device, equipment and storage medium
US20220138985A1 (en) Method, apparatus, electronic device and storage medium for image distortion calibrating
WO2020024576A1 (en) Camera calibration method and apparatus, electronic device, and computer-readable storage medium
EP4220547A1 (en) Method and apparatus for determining heat data of global region, and storage medium
CN111445487B (en) Image segmentation method, device, computer equipment and storage medium
WO2020133330A1 (en) Systems and methods for video surveillance
CN112257713A (en) Image processing method, image processing device, electronic equipment and computer readable storage medium
CN111507298A (en) Face detection method and device, computer equipment and storage medium
CN111383254A (en) Depth information acquisition method and system and terminal equipment
CN110111364B (en) Motion detection method and device, electronic equipment and storage medium
CN111915541A (en) Image enhancement processing method, device, equipment and medium based on artificial intelligence
CN111385461A (en) Panoramic shooting method and device, camera and mobile terminal
CN116912171A (en) Image data processing method, device, equipment and storage medium
CN111553850B (en) Three-dimensional information acquisition method and device based on binocular stereoscopic vision
CN110458768B (en) Image processing method, computer device, and readable storage medium
CN116152586A (en) Model training method and device, electronic equipment and storage medium
CN109003327B (en) Image processing method, image processing device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination