WO2021227360A1 - 一种交互式视频投影方法、装置、设备及存储介质 - Google Patents
一种交互式视频投影方法、装置、设备及存储介质 Download PDFInfo
- Publication number
- WO2021227360A1 WO2021227360A1 PCT/CN2020/121664 CN2020121664W WO2021227360A1 WO 2021227360 A1 WO2021227360 A1 WO 2021227360A1 CN 2020121664 W CN2020121664 W CN 2020121664W WO 2021227360 A1 WO2021227360 A1 WO 2021227360A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- dimensional
- camera
- feature
- video frame
- feature points
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000002452 interceptive effect Effects 0.000 title claims abstract description 52
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 73
- 238000009877 rendering Methods 0.000 claims abstract description 45
- 239000011159 matrix material Substances 0.000 claims abstract description 40
- 230000015654 memory Effects 0.000 claims description 20
- 238000005457 optimization Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 abstract description 10
- 238000012545 processing Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
Definitions
- the embodiments of the present application relate to the field of image processing, and in particular, to an interactive video projection method, device, equipment, and storage medium.
- Video projection technology combines surveillance video with a three-dimensional model, and projects the surveillance video of the area of interest into a three-dimensional model of a large scene, which can realize the combination of virtual and real static large scenes and dynamic key scenes.
- each channel of video requires the staff to spend a lot of time to configure the camera position and posture and other information, the configuration process is cumbersome, and there are situations where the video projection requirements cannot be met in time.
- the embodiments of the present application provide an interactive video projection method, device, equipment, and storage medium to meet the real-time requirements of video projection.
- an embodiment of the present application provides an interactive video projection method, including:
- the camera in the virtual scene is set according to the camera pose matrix, the focal length information and/or the distortion parameter, and the video frame is added to the rendering pipeline for video projection.
- the rendering of a three-dimensional map based on the initial position and posture of the camera determined in the virtual scene to obtain a two-dimensional picture corresponding to the initial position and posture, the initial position and posture being determined based on the video frames taken by the camera includes :
- the feature matching of the video frame captured by the camera with the two-dimensional picture includes:
- Feature matching is performed on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor.
- the method further includes:
- the method further includes:
- the viewing angle adjustment reminder is made when the feature matching fails to remind the operator to reselect the initial position and posture of the camera.
- the determining the three-dimensional feature points corresponding to the two-dimensional feature points on the video frame on the three-dimensional map according to the feature matching result includes:
- the coordinates of the three-dimensional feature points corresponding to the two-dimensional feature points in the three-dimensional map are determined.
- the determining the camera pose matrix, focal length information, and/or distortion parameters through a pose solving algorithm based on the two-dimensional feature points and the three-dimensional feature points includes:
- the two-dimensional feature point coordinates and the three-dimensional feature point coordinates are substituted into the PnP algorithm and the nonlinear optimization algorithm to obtain the camera pose matrix, focal length information and/or distortion parameters.
- the method further includes:
- the pose solution algorithm it is judged whether the camera pose solution is successful, and the viewing angle adjustment reminder is made when the camera pose solution fails to remind the operator to reselect the initial position and pose of the camera.
- an embodiment of the present application provides an interactive video projection device, including a two-dimensional rendering module, a feature corresponding module, a pose determination module, and a video projection module, wherein:
- a two-dimensional rendering module for rendering a three-dimensional map based on the initial position and posture of the camera determined in the virtual scene to obtain a two-dimensional picture corresponding to the initial position and posture, the initial position and posture being determined based on the video frame captured by the camera ;
- the feature corresponding module is configured to perform feature matching between the video frame captured by the camera and the two-dimensional picture, and determine the three-dimensional feature point corresponding to the two-dimensional feature point on the video frame on the three-dimensional map according to the feature matching result;
- the pose determination module is configured to determine the camera pose matrix, focal length information and/or distortion parameters through a pose solving algorithm based on the two-dimensional feature points and the three-dimensional feature points;
- the video projection module is configured to set a camera in a virtual scene according to the camera pose matrix, the focal length information and/or the distortion parameter, and add the video frame to the rendering pipeline for video projection.
- the two-dimensional rendering module is specifically used for:
- the feature corresponding module when the feature corresponding module performs feature matching between the video frame captured by the camera and the two-dimensional picture, it specifically includes:
- Feature matching is performed on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor.
- the feature corresponding module after the feature corresponding module performs feature matching on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor, it also screens the matched feature points based on the RANSAC algorithm.
- a matching error reminding module which is used to determine whether the feature matching is successful according to the feature matching result after the feature matching module performs feature matching on the video frame captured by the camera and the two-dimensional picture, and when the feature matching fails Remind the viewing angle adjustment at time to remind the operator to re-select the initial position and posture of the camera.
- the feature corresponding module determines the corresponding three-dimensional feature point of the two-dimensional feature point on the video frame on the three-dimensional map according to the feature matching result, it specifically includes:
- the coordinates of the three-dimensional feature points corresponding to the two-dimensional feature points in the three-dimensional map are determined.
- the pose determination module is specifically used for:
- the two-dimensional feature point coordinates and the three-dimensional feature point coordinates are substituted into the PnP algorithm and the nonlinear optimization algorithm to obtain the camera pose matrix, focal length information and/or distortion parameters.
- a pose error reminding module which is used to determine the camera pose matrix, focal length information, and/or the camera pose based on the two-dimensional feature point and the three-dimensional feature point by the pose determination module based on the pose solving algorithm. After distorting the parameters, judge whether the camera pose is successfully solved according to the result of the pose solving algorithm, and remind the operator to adjust the angle of view when the camera pose fails to solve, so as to remind the operator to reselect the initial position and posture of the camera.
- an embodiment of the present application provides a computer device, including: a memory and one or more processors;
- the memory is used to store one or more programs
- the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors implement the interactive video projection method as described in the first aspect.
- embodiments of the present application provide a storage medium containing computer-executable instructions, which are used to execute the interactive video projection method as described in the first aspect when the computer-executable instructions are executed by a computer processor .
- the initial position and posture of the camera in the virtual scene are determined according to the video frames taken by the camera, and a three-dimensional map is rendered based on the initial position and posture to obtain a two-dimensional picture corresponding to the range taken by the camera in the initial position and posture. , And then perform feature matching on the two-dimensional picture and the video frame shot by the camera. After the matching is completed, the three-dimensional feature points corresponding to the two-dimensional feature points on the video frame in the three-dimensional map are determined.
- the pose calculation algorithm can determine the camera pose matrix, Focus information and/or distortion parameters, set the camera in the virtual scene according to the above information, and add the video frame to the rendering pipeline for video projection, so as to achieve semi-automatic interactive fast video projection mapping, without the need for the staff to manually configure the camera parameters accurately, and improve Video projection efficiency, and by matching the video frame and the two-dimensional picture, the video frame can be projected on the correct position of the three-dimensional model, effectively improving the video projection effect.
- Fig. 1 is a flowchart of an interactive video projection method provided by an embodiment of the present application
- FIG. 2 is a flowchart of another interactive video projection method provided by an embodiment of the present application.
- FIG. 3 is a schematic structural diagram of an interactive video projection device provided by an embodiment of the present application.
- Fig. 4 is a schematic structural diagram of a computer device provided by an embodiment of the present application.
- Fig. 1 shows a flowchart of an interactive video projection method provided by an embodiment of the present application.
- the interactive video projection method provided by an embodiment of the present application may be executed by an interactive video projection device, which may Realized by means of hardware and/or software, and integrated in computer equipment.
- an interactive video projection device executes an interactive video projection method as an example.
- the interactive video projection method includes:
- the virtual scene can be a three-dimensional scene rendered based on a three-dimensional map.
- the displayed virtual scene picture can be determined based on the position, posture and angle of view of the camera in the virtual scene, and the adjustment operation (position, angle of view) can be performed through an input device (e.g., mouse, keyboard) , Focal length, etc.) to realize the position and posture of the camera in the virtual scene, so as to determine the position and posture of the camera in the target virtual scene.
- the virtual scene screen can be displayed on the display device (display screen).
- an initial position and posture confirmation button or a one-key mapping button can be set in the operation interface, and the position and posture of the camera in the virtual scene at this time can be determined in response to the confirmation operation of the button, and used as the initial position and posture of the camera.
- the three-dimensional map is established based on the world coordinate system. Even if there is an error between the coordinate system of the three-dimensional map and the world coordinate system, the offset caused by the error is within the error range (usually within a few meters). Rendering a three-dimensional map is not a big problem, and the rendered two-dimensional picture can still cover the target area (the area corresponding to the video frame).
- S102 Perform feature matching on the video frame captured by the camera and the two-dimensional picture, and determine the three-dimensional feature point corresponding to the two-dimensional feature point on the video frame on the three-dimensional map according to the feature matching result.
- feature matching is performed on the video frame captured by the camera and the two-dimensional picture, that is, the feature points in the video frame and the two-dimensional picture are extracted, and the similarity between the feature points The degree (feature vector distance) is matched and the matching result is generated.
- the feature points on the video frame are two-dimensional feature points
- the feature points on the two-dimensional screen are matching feature points.
- the correspondence between the three-dimensional map and the two-dimensional screen coordinate points can be recorded, and the three-dimensional features in the three-dimensional map corresponding to the matching feature points in the two-dimensional screen can be determined according to the record.
- Point coordinates are used to determine the three-dimensional feature points corresponding to the two-dimensional feature points on the video frame on the three-dimensional map.
- the RGB image (two-dimensional picture) and the depth map are simultaneously rendered when the two-dimensional picture is rendered.
- the three-dimensional feature points corresponding to the points in the two-dimensional picture can be inversely calculated to determine the two-dimensional image on the video frame.
- the feature point corresponds to the three-dimensional feature point on the three-dimensional map.
- S103 Based on the two-dimensional feature points and the three-dimensional feature points, determine a camera pose matrix, focal length information, and/or distortion parameters through a pose solving algorithm.
- the two-dimensional feature points and the three-dimensional feature points are substituted into the pose solving algorithm to obtain the camera pose matrix, focal length information, and/or Distortion parameters.
- the pose solution algorithm is a method to solve the motion of 3D to 2D point pairs. It describes how to determine the pose, focal length and distortion of the camera when the n 3D space points and their projection positions are known.
- whether the camera is distorted can be determined according to the specific type or parameters of the camera. For cameras with no distortion or less severe distortion, the distortion parameters can be set as default parameters (for example, if set to 0, the default camera has no distortion).
- S104 Set a camera in the virtual scene according to the camera pose matrix, the focal length information and/or the distortion parameter, and add the video frame to the rendering pipeline for video projection.
- the pose matrix and focal length information are input into the camera parameters of the virtual scene, and the position, posture and focal length of the camera are set. Then the video frame is added to the rendering pipeline, and the rendering pipeline performs real-time fusion projection of the video frame under the corresponding camera parameter settings.
- the mapped overlap area is processed for smooth transition, thereby fusing the video frame into the three-dimensional scene, and completing the video projection of the corrected video frame in the three-dimensional scene. It is understandable that the video projection of the video frame can be performed based on the existing video projection method, which will not be repeated here.
- the initial position and posture of the camera in the virtual scene are determined according to the video frames taken by the camera, and the three-dimensional map is rendered based on the initial position and posture to obtain a two-dimensional picture corresponding to the range taken by the camera in the initial position and posture, and then Perform feature matching on the two-dimensional picture and the video frame taken by the camera.
- Fig. 2 is a flowchart of another interactive video projection method provided by an embodiment of the application.
- the interactive video projection method is a concrete embodiment of the interactive video projection method.
- the interactive video projection method includes:
- S201 Determine the initial position and posture of the camera in the virtual scene based on the video frames shot by the camera.
- S202 Acquire a three-dimensional model tile corresponding to the rendering range according to the initial position and posture, and the three-dimensional map is stored in the form of a three-dimensional model tile.
- the three-dimensional map is stored in the form of three-dimensional model tiles.
- the data volume of the three-dimensional map is relatively large.
- each three-dimensional map is called a three-dimensional model tile, and the location range corresponding to each three-dimensional model tile is recorded.
- S203 Render the three-dimensional model tiles to obtain a two-dimensional picture corresponding to the initial position and posture.
- the three-dimensional model tiles are rendered by the GPU visualization engine and the two-dimensional picture corresponding to the initial position and posture is obtained. It can be understood that the display range of the two-dimensional picture is larger than the display range of the corresponding video frame.
- S204 Obtain video frames captured by the camera and feature points and descriptors of the two-dimensional picture based on the image feature extraction algorithm.
- image feature extraction is performed on video frames and two-dimensional images based on the GPU, and the image features include feature points and descriptors.
- the image feature extraction algorithm can be SIFT (Scale Invariant Feature Transform) algorithm, SURF (Speeded Up Robust Features, accelerated robust feature) algorithm, ORB (Oriented FAST and Rotated BRIEF) algorithm, etc. This embodiment does not Make a limit.
- the feature points of the image are some of the most representative points on the image.
- the so-called most representative means that these points contain most of the information expressed by the image. Even if you rotate, zoom, or even adjust the brightness of the image, these points still exist steadily and will not be lost. Finding these points is equivalent to confirming the image, and they can be used for meaningful work such as matching and recognition.
- Feature points are composed of two parts: Key-point and Descriptor.
- the BRIEF descriptor is a binary descriptor, usually a 128-bit binary string. Its calculation method is to randomly select 128 point pairs from around the key point p. For two points in each point pair, if the gray value of the previous point is greater than that of the next point, then it will be 1, otherwise it will be 0.
- extracting ORB features actually includes two things: extracting key points and calculating descriptors.
- Use FAST feature point detection algorithm or Harris corner detection algorithm or SIFT, SURF and other algorithms to detect the location of feature points, and then in the feature point neighborhood Use the BRIEF algorithm to establish feature descriptors.
- S205 Perform feature matching on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor.
- the similarity between the two feature points is judged according to the distance of the corresponding descriptor, and the smaller the distance, the higher the similarity.
- the distance of the descriptor can be Euclidean distance, Hamming distance, cosine distance, etc.
- the feature points are sorted according to distance, and the matching results of the first N features are displayed under a certain degree of confidence, that is, the two-dimensional The feature points between the picture and the video frame are matched.
- whether the feature matching is successful can be judged after the feature points are extracted or after the feature points are actually matched. For example, when calling the image feature extraction algorithm to obtain feature points, determine whether the feature points are normally obtained according to the algorithm results, or when matching the feature points according to the distance of the descriptor, judge whether the matching feature points reach the normal number according to the matching result Or the proportion, and use this to judge whether the feature matching is successful. When the matching is successful, the next step is to filter the feature points. If the matching fails, a viewing angle adjustment reminder is performed to remind the operator to return to the initial position and posture of the newly selected camera in step S201.
- the RANSAC Random Sample Consensus
- the basic matrix and homography matrix of the two-dimensional picture and video frame are obtained, and the matched feature points are screened by the RANSAC algorithm based on the basic matrix and the homography matrix to eliminate the feature points with matching errors.
- S207 Determine the coordinates of the matching feature point in the two-dimensional picture that matches the two-dimensional feature point on the video frame according to the feature matching result.
- the feature points that match each other in the two-dimensional picture and the video frame are respectively defined as the matching feature point and the two-dimensional feature point.
- the matching feature points in the two-dimensional picture that match the two-dimensional feature points on the video frame are determined, and the coordinates of these matching feature points are determined.
- S208 Determine the coordinates of the three-dimensional feature points corresponding to the two-dimensional feature points in the three-dimensional map according to the correspondence between the three-dimensional map and the two-dimensional screen coordinate points.
- the correspondence between the three-dimensional map and the coordinate points of the two-dimensional screen can be recorded.
- the matching feature points corresponding to the two-dimensional feature points are determined according to the matching result, and then the corresponding relationship between the three-dimensional map and the two-dimensional screen coordinate points is obtained and matched The coordinates of the three-dimensional feature point corresponding to the feature point.
- the RGB image (two-dimensional image) and the depth map are simultaneously rendered when the two-dimensional image is rendered.
- the three-dimensional feature points corresponding to the points in the two-dimensional image can be inversely calculated, and the matching features on the two-dimensional image can be obtained.
- the coordinates of the three-dimensional feature point corresponding to the point can be obtained.
- S209 Obtain the two-dimensional feature point coordinates on the video frame and the three-dimensional feature point coordinates on the three-dimensional map, and substitute the two-dimensional feature point coordinates and the three-dimensional feature point coordinates into the PnP algorithm and the nonlinear optimization algorithm to obtain the camera Pose matrix, focal length information and/or distortion parameters.
- the PnP (Perspective-n-Point) algorithm is a method for solving 3D to 2D point pair motion, which can be solved by P3P, direct linear transformation (DLT), EPnP and other algorithms.
- P3P algorithm is a 3D-2D pose solution method. It needs to know the matching 3D point and the image 2D point, that is, first find the corresponding 2D point (equivalent to the two-dimensional feature of this scheme Point) 3D coordinates in the current camera coordinate system (equivalent to the three-dimensional feature point coordinates of this solution), and then calculate the camera pose according to the 3D coordinates in the world coordinate system and the 3D coordinates in the current camera coordinate system.
- the nonlinear optimization algorithm is a method of further optimizing the reprojection error of the 3D point to the 2D point through the least squares given the initial value of the camera posture focal length. During the optimization process, the algorithm will further refine the camera posture and focal length. Adjustment.
- the LM (Levenberg-Marquardt) optimization algorithm is used as a non-linear optimization algorithm to optimize the initial value of the camera posture focal length to obtain the smallest reprojection error from 3D point to 2D point.
- the two-dimensional feature point coordinates on the video frame and the three-dimensional feature point coordinates on the three-dimensional map are obtained, the two-dimensional feature point coordinates and the three-dimensional feature point coordinates are substituted into the PnP algorithm and the nonlinear optimization algorithm, and the accurate solution is obtained by the PnP algorithm.
- the camera pose matrix and then optimize the camera parameters through a nonlinear optimization algorithm to obtain focal length information and/or distortion parameters.
- the distortion parameters can be determined according to the specific types or parameters of the camera. For cameras with no distortion or less severe distortion, the distortion parameters can be set as default parameters (for example, if set to 0, the default camera has no distortion), and the distortion parameters can be omitted. calculate.
- a viewing angle adjustment reminder will be made to remind the operator to reselect the initial position and attitude of the camera.
- whether the camera pose is successfully solved can be judged according to the solution result of the PnP algorithm. For example, according to the results of the PnP algorithm, it is judged whether the obtained camera pose data is normal or whether the deviation between the camera pose and the initial position and pose is within a reasonable range, and then it is judged whether the camera pose is solved successfully. When the camera pose is solved successfully, the next video projection operation is continued. If the camera pose fails to be solved, a viewing angle adjustment reminder is issued to remind the operator to return to the initial position and attitude of the newly selected camera in step S201.
- S210 Set a camera in the virtual scene according to the camera pose matrix, the focal length information, and/or the distortion parameter, and add the video frame to the rendering pipeline for video projection.
- the initial position and posture of the camera in the virtual scene are determined according to the video frames taken by the camera, and the three-dimensional map is rendered based on the initial position and posture to obtain a two-dimensional picture corresponding to the range taken by the camera in the initial position and posture, and then Perform feature matching on the two-dimensional picture and the video frame taken by the camera.
- the matching After the matching is completed, determine the three-dimensional feature point corresponding to the two-dimensional feature point on the video frame in the three-dimensional map, and determine the camera pose matrix and focal length information through the pose solving algorithm And/or distortion parameters, set the camera in the virtual scene according to the above information, and add the video frame to the rendering pipeline for video projection, so as to realize semi-automatic interactive fast video projection mapping, without the need for staff to manually configure camera parameters accurately, and improve video projection It is efficient, and the video frame can be projected on the correct position of the three-dimensional model through the matching of the video frame and the two-dimensional picture, which effectively improves the video projection effect.
- Two-dimensional images are rendered in the form of three-dimensional model tiles, which reduces the burden of GPU graphics processing and effectively improves the real-time performance of video projection.
- the operator only needs to adjust the viewing angle of the virtual scene to realize one-click mapping, without manual calculation of complex camera parameters, reducing the time spent on each channel of video projection parameter configuration, improving video projection efficiency, and facilitating the large-scale implementation of video projection technology Promotion.
- FIG. 3 is a schematic structural diagram of an interactive video projection device provided by an embodiment of the application.
- the interactive video projection device provided by this embodiment includes a two-dimensional rendering module 31, a feature corresponding module 32, a pose determination module 33 and a video projection module 34.
- the two-dimensional rendering module 31 is configured to render a three-dimensional map based on the initial position and posture of the camera determined in the virtual scene to obtain a two-dimensional picture corresponding to the initial position and posture, and the initial position and posture is based on the camera shooting.
- the pose determination module 33 is used to determine the camera pose matrix, focal length information and/or distortion parameters based on the two-dimensional feature points and the three-dimensional feature points through a pose solving algorithm; video projection module 34 , For setting a camera in a virtual scene according to the camera pose matrix, the focal length information and/or the distortion parameter, and adding the video frame to the rendering pipeline for video projection.
- the initial position and posture of the camera in the virtual scene are determined according to the video frames taken by the camera, and the three-dimensional map is rendered based on the initial position and posture to obtain a two-dimensional picture corresponding to the range taken by the camera in the initial position and posture, and then Perform feature matching on the two-dimensional picture and the video frame taken by the camera.
- the feature corresponding module 32 when the feature corresponding module 32 performs feature matching between the video frame captured by the camera and the two-dimensional picture, it specifically includes:
- Feature matching is performed on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor.
- the feature corresponding module 32 after the feature corresponding module 32 performs feature matching on the feature points between the video frame and the two-dimensional picture according to the distance of the descriptor, it further compares the matched feature points based on the RANSAC algorithm. To filter.
- it further includes a matching error reminding module, which is used to determine whether the feature matching is successful according to the feature matching result after the feature corresponding module 32 performs feature matching on the video frame captured by the camera and the two-dimensional picture , And when the feature matching fails, the viewing angle adjustment reminder will remind the operator to re-select the initial position and posture of the camera.
- a matching error reminding module which is used to determine whether the feature matching is successful according to the feature matching result after the feature corresponding module 32 performs feature matching on the video frame captured by the camera and the two-dimensional picture , And when the feature matching fails, the viewing angle adjustment reminder will remind the operator to re-select the initial position and posture of the camera.
- the feature corresponding module 32 determines the three-dimensional feature point corresponding to the two-dimensional feature point on the video frame on the three-dimensional map according to the feature matching result, it specifically includes:
- the coordinates of the three-dimensional feature points corresponding to the two-dimensional feature points in the three-dimensional map are determined.
- the pose determination module 33 is specifically configured to:
- the two-dimensional feature point coordinates and the three-dimensional feature point coordinates are substituted into the PnP algorithm and the nonlinear optimization algorithm to obtain the camera pose matrix, focal length information and/or distortion parameters.
- it further includes a pose error reminding module, configured to determine the camera pose matrix by the pose determination module 33 based on the two-dimensional feature points and the three-dimensional feature points. After the focal length information and/or distortion parameters, it is judged whether the camera pose is successfully solved according to the result of the pose solving algorithm, and the angle of view adjustment reminds when the camera pose solves failed to remind the operator to reselect the initial position and attitude of the camera.
- a pose error reminding module configured to determine the camera pose matrix by the pose determination module 33 based on the two-dimensional feature points and the three-dimensional feature points.
- Fig. 4 is a schematic structural diagram of a computer device provided by an embodiment of the present application.
- the computer equipment includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs It is executed by the one or more processors 41, so that the one or more processors 41 implement the interactive video projection method provided in the foregoing embodiment.
- the input device 43, the output device 44, the memory 42, and the processor 41 may be connected by a bus or other methods. In FIG. 4, the connection by a bus is taken as an example.
- the memory 42 can be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the interactive video projection method described in any embodiment of this application (for example, interactive The two-dimensional rendering module 31, the feature corresponding module 32, the pose determination module 33 and the video projection module 34 in the video projection device.
- the memory 42 may mainly include a program storage area and a data storage area.
- the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the device, and the like.
- the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
- the memory 42 may further include a memory remotely provided with respect to the processor 41, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
- the input device 43 can be used to receive inputted digital or character information, and generate key signal input related to user settings and function control of the device.
- the output device 44 may include a display device such as a display screen.
- the processor 41 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 42 to realize the above-mentioned interactive video projection method.
- the interactive video projection device and computer provided above can be used to execute the interactive video projection method provided in any of the above embodiments, and have corresponding functions and beneficial effects.
- the embodiment of the present application also provides a storage medium containing computer-executable instructions, when the computer-executable instructions are executed by a computer processor, they are used to execute the interactive video projection method provided in the above-mentioned embodiments.
- the method includes: rendering a three-dimensional map based on the initial position and posture of the camera determined in the virtual scene to obtain a two-dimensional picture corresponding to the initial position and posture, the initial position and posture being determined based on the video frames taken by the camera; Feature matching is performed on the video frame of the video frame and the two-dimensional picture, and the three-dimensional feature point corresponding to the two-dimensional feature point on the video frame on the three-dimensional map is determined according to the feature matching result; based on the two-dimensional feature point and the The three-dimensional feature points, the camera pose matrix, focal length information and/or distortion parameters are determined by a pose solving algorithm; the camera in the virtual scene is set according to the camera pose matrix, the focal length information and/or the distortion parameters, The video frame is added to the rendering pipeline for
- Storage medium any of various types of storage devices or storage devices.
- the term "storage medium” is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc. ; Non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements.
- the storage medium may further include other types of memory or a combination thereof.
- the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the Internet).
- the second computer system can provide the program instructions to the first computer for execution.
- storage media may include two or more storage media that may reside in different locations (for example, in different computer systems connected through a network).
- the storage medium may store program instructions (for example, embodied as a computer program) executable by one or more processors.
- the storage medium containing computer-executable instructions provided in the embodiments of the present application is not limited to the interactive video projection method described above, and can also execute the interactive video projection methods provided in any of the embodiments of the present application. Related operations in the video projection method.
- the interactive video projection apparatus, equipment, and storage medium provided in the above embodiments can perform the interactive video projection method provided in any embodiment of this application.
- the interactive video projection method provided by the example can perform the interactive video projection method provided in any embodiment of this application.
- the interactive video projection method provided by the example can perform the interactive video projection method provided in any embodiment of this application.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- Remote Sensing (AREA)
- Health & Medical Sciences (AREA)
- Signal Processing (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
Claims (11)
- 一种交互式视频投影方法,其特征在于,包括:基于在虚拟场景中确定的相机初始位置姿态,对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面,所述初始位置姿态基于摄像头拍摄的视频帧确定;对摄像头拍摄的视频帧和所述二维画面进行特征匹配,并根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点;基于所述二维特征点和所述三维特征点,通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数;根据所述相机位姿矩阵、所述焦距信息和/或所述畸变参数设置虚拟场景中的相机,并将所述视频帧加入渲染管线中进行视频投影。
- 根据权利要求1所述的交互式视频投影方法,其特征在于,所述基于在虚拟场景中确定的相机初始位置姿态,对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面,所述初始位置姿态基于摄像头拍摄的视频帧确定,包括:基于摄像头拍摄的视频帧在虚拟场景中确定相机初始位置姿态;根据所述初始位置姿态获取渲染范围对应的三维模型瓦片,所述三维地图通过三维模型瓦片的形式进行存储;对所述三维模型瓦片进行渲染以获得与所述初始位置姿态对应的二维画面。
- 根据权利要求1所述的交互式视频投影方法,其特征在于,所述对摄像头拍摄的视频帧和所述二维画面进行特征匹配,包括:基于图像特征提取算法获取摄像头拍摄的视频帧和所述二维画面的特征点以及描述子;根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配。
- 根据权利要求3所述的交互式视频投影方法,其特征在于,所述根据描述子的距离对所述视频帧和所述二维画面之间的特征点进行特征匹配之后,还包括:基于RANSAC算法对匹配的特征点进行筛选。
- 根据权利要求1所述的交互式视频投影方法,其特征在于,所述对摄像头拍摄的视频帧和所述二维画面进行特征匹配之后,还包括:根据特征匹配结果判断特征匹配是否成功,并在特征匹配失败时进行视角调整提醒,以提醒操作者重新选择相机的初始位置姿态。
- 根据权利要求1所述的交互式视频投影方法,其特征在于,所述根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点,包括:根据特征匹配结果确定所述二维画面中与所述视频帧上的二维特征点匹配的匹配特征点的坐标;根据三维地图和二维画面坐标点的对应关系,确定三维地图中与二维特征点对应的三维特征点的坐标。
- 根据权利要求1所述的交互式视频投影方法,其特征在于,所述基于所述二维特征点和所述三维特征点,通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数,包括:获取视频帧上的二维特征点坐标和三维地图上的三维特征点坐标;将所述二维特征点坐标和所述三维特征点坐标代入PnP算法和非线性优化算法,以得到相机位姿矩阵、焦距信息和/或畸变参数。
- 根据权利要求1所述的交互式视频投影方法,其特征在于,所述基于所述二维特征点和所述三维特征点,通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数之后,还包括:根据位姿求解算法结果判断相机位姿求解是否成功,并在相机位姿求解失败时进行视角调整提醒,以提醒操作者重新选择相机的初始位置姿态。
- 一种交互式视频投影装置,其特征在于,包括二维渲染模块、特征对应模块、位姿确定模块和视频投影模块,其中:二维渲染模块,用于基于在虚拟场景中确定的相机初始位置姿态,对三维地图进行渲染以获得与所述初始位置姿态对应的二维画面,所述初始位置姿态基于摄像头拍摄的视频帧确定;特征对应模块,用于对摄像头拍摄的视频帧和所述二维画面进行特征匹配,并根据特征匹配结果确定所述视频帧上的二维特征点在所述三维地图上对应的三维特征点;位姿确定模块,用于基于所述二维特征点和所述三维特征点,通过位姿求解算法确定相机位姿矩阵、焦距信息和/或畸变参数;视频投影模块,用于根据所述相机位姿矩阵、所述焦距信息和/或所述畸变 参数设置虚拟场景中的相机,并将所述视频帧加入渲染管线中进行视频投影。
- 一种计算机设备,其特征在于,包括:存储器以及一个或多个处理器;所述存储器,用于存储一个或多个程序;当所述一个或多个程序被所述一个或多个处理器执行,使得所述一个或多个处理器实现如权利要求1-8任一所述的交互式视频投影方法。
- 一种包含计算机可执行指令的存储介质,其特征在于,所述计算机可执行指令在由计算机处理器执行时用于执行如权利要求1-8任一所述的交互式视频投影方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010408983.8A CN111640181A (zh) | 2020-05-14 | 2020-05-14 | 一种交互式视频投影方法、装置、设备及存储介质 |
CN202010408983.8 | 2020-05-14 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021227360A1 true WO2021227360A1 (zh) | 2021-11-18 |
Family
ID=72332004
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/121664 WO2021227360A1 (zh) | 2020-05-14 | 2020-10-16 | 一种交互式视频投影方法、装置、设备及存储介质 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN111640181A (zh) |
WO (1) | WO2021227360A1 (zh) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114445541A (zh) * | 2022-01-28 | 2022-05-06 | 北京百度网讯科技有限公司 | 处理视频的方法、装置、电子设备及存储介质 |
CN114827569A (zh) * | 2022-04-24 | 2022-07-29 | 咪咕视讯科技有限公司 | 画面显示方法、装置、虚拟现实设备及存储介质 |
CN114915727A (zh) * | 2022-05-12 | 2022-08-16 | 北京国基科技股份有限公司 | 一种视频监控画面构建方法及装置 |
CN115022613A (zh) * | 2022-05-19 | 2022-09-06 | 北京字节跳动网络技术有限公司 | 一种视频重建方法、装置、电子设备及存储介质 |
CN115396644A (zh) * | 2022-07-21 | 2022-11-25 | 贝壳找房(北京)科技有限公司 | 基于多段外参数据的视频融合方法及装置 |
WO2023116430A1 (zh) * | 2021-12-23 | 2023-06-29 | 奥格科技股份有限公司 | 视频与城市信息模型三维场景融合方法、系统及存储介质 |
CN116580099A (zh) * | 2023-07-14 | 2023-08-11 | 山东艺术学院 | 一种基于视频与三维模型融合的林地目标定位方法 |
CN117218320A (zh) * | 2023-11-08 | 2023-12-12 | 济南大学 | 基于混合现实的空间标注方法 |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111640181A (zh) * | 2020-05-14 | 2020-09-08 | 佳都新太科技股份有限公司 | 一种交互式视频投影方法、装置、设备及存储介质 |
CN112437276B (zh) * | 2020-11-20 | 2023-04-07 | 埃洛克航空科技(北京)有限公司 | 一种基于WebGL的三维视频融合方法及系统 |
CN113096003B (zh) * | 2021-04-02 | 2023-08-18 | 北京车和家信息技术有限公司 | 针对多视频帧的标注方法、装置、设备和存储介质 |
CN113793379A (zh) * | 2021-08-12 | 2021-12-14 | 视辰信息科技(上海)有限公司 | 相机姿态求解方法及系统、设备和计算机可读存储介质 |
CN113870163B (zh) * | 2021-09-24 | 2022-11-29 | 埃洛克航空科技(北京)有限公司 | 基于三维场景的视频融合方法以及装置、存储介质、电子装置 |
CN114237537B (zh) * | 2021-12-10 | 2023-08-04 | 杭州海康威视数字技术股份有限公司 | 头戴式设备、远程协助方法及系统 |
CN114095662B (zh) * | 2022-01-20 | 2022-07-05 | 荣耀终端有限公司 | 拍摄指引方法及电子设备 |
CN114677572B (zh) * | 2022-04-08 | 2023-04-18 | 北京百度网讯科技有限公司 | 对象描述参数的生成方法、深度学习模型的训练方法 |
CN115100327B (zh) * | 2022-08-26 | 2022-12-02 | 广东三维家信息科技有限公司 | 动画立体视频生成的方法、装置及电子设备 |
CN115866254A (zh) * | 2022-11-24 | 2023-03-28 | 亮风台(上海)信息科技有限公司 | 一种传输视频帧及摄像参数信息的方法与设备 |
CN116758157B (zh) * | 2023-06-14 | 2024-01-30 | 深圳市华赛睿飞智能科技有限公司 | 一种无人机室内三维空间测绘方法、系统及存储介质 |
CN117237438A (zh) * | 2023-09-18 | 2023-12-15 | 共享数据(福建)科技有限公司 | 一种三维模型和无人机视频数据的范围匹配方法与终端 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20010043738A1 (en) * | 2000-03-07 | 2001-11-22 | Sawhney Harpreet Singh | Method of pose estimation and model refinement for video representation of a three dimensional scene |
CN105844696A (zh) * | 2015-12-31 | 2016-08-10 | 清华大学 | 基于射线模型三维重构的图像定位方法以及装置 |
CN108830894A (zh) * | 2018-06-19 | 2018-11-16 | 亮风台(上海)信息科技有限公司 | 基于增强现实的远程指导方法、装置、终端和存储介质 |
CN111586360A (zh) * | 2020-05-14 | 2020-08-25 | 佳都新太科技股份有限公司 | 一种无人机投影方法、装置、设备及存储介质 |
CN111640181A (zh) * | 2020-05-14 | 2020-09-08 | 佳都新太科技股份有限公司 | 一种交互式视频投影方法、装置、设备及存储介质 |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103136738A (zh) * | 2011-11-29 | 2013-06-05 | 北京航天长峰科技工业集团有限公司 | 一种复杂场景下固定摄像机监控视频与三维模型配准方法 |
CN102622747B (zh) * | 2012-02-16 | 2013-10-16 | 北京航空航天大学 | 一种用于视觉测量的摄像机参数优化方法 |
CN102663767B (zh) * | 2012-05-08 | 2014-08-06 | 北京信息科技大学 | 视觉测量系统的相机参数标定优化方法 |
CN103400409B (zh) * | 2013-08-27 | 2016-08-10 | 华中师范大学 | 一种基于摄像头姿态快速估计的覆盖范围3d可视化方法 |
CN103716586A (zh) * | 2013-12-12 | 2014-04-09 | 中国科学院深圳先进技术研究院 | 一种基于三维空间场景的监控视频融合系统和方法 |
CN104182982B (zh) * | 2014-08-27 | 2017-02-15 | 大连理工大学 | 双目立体视觉摄像机标定参数的整体优化方法 |
CN105118061A (zh) * | 2015-08-19 | 2015-12-02 | 刘朔 | 用于将视频流配准至三维地理信息空间中的场景的方法 |
CN105678748B (zh) * | 2015-12-30 | 2019-01-15 | 清华大学 | 三维监控系统中基于三维重构的交互式标定方法和装置 |
CN105678839A (zh) * | 2015-12-30 | 2016-06-15 | 天津德勤和创科技发展有限公司 | 基于计算机三维场景模拟技术的安防设备分布设计方法 |
CN107564098A (zh) * | 2017-08-17 | 2018-01-09 | 中山大学 | 一种大区域网络三维噪声地图的快速渲染方法 |
-
2020
- 2020-05-14 CN CN202010408983.8A patent/CN111640181A/zh active Pending
- 2020-10-16 WO PCT/CN2020/121664 patent/WO2021227360A1/zh active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20010043738A1 (en) * | 2000-03-07 | 2001-11-22 | Sawhney Harpreet Singh | Method of pose estimation and model refinement for video representation of a three dimensional scene |
CN105844696A (zh) * | 2015-12-31 | 2016-08-10 | 清华大学 | 基于射线模型三维重构的图像定位方法以及装置 |
CN108830894A (zh) * | 2018-06-19 | 2018-11-16 | 亮风台(上海)信息科技有限公司 | 基于增强现实的远程指导方法、装置、终端和存储介质 |
CN111586360A (zh) * | 2020-05-14 | 2020-08-25 | 佳都新太科技股份有限公司 | 一种无人机投影方法、装置、设备及存储介质 |
CN111640181A (zh) * | 2020-05-14 | 2020-09-08 | 佳都新太科技股份有限公司 | 一种交互式视频投影方法、装置、设备及存储介质 |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2023116430A1 (zh) * | 2021-12-23 | 2023-06-29 | 奥格科技股份有限公司 | 视频与城市信息模型三维场景融合方法、系统及存储介质 |
CN114445541A (zh) * | 2022-01-28 | 2022-05-06 | 北京百度网讯科技有限公司 | 处理视频的方法、装置、电子设备及存储介质 |
CN114827569A (zh) * | 2022-04-24 | 2022-07-29 | 咪咕视讯科技有限公司 | 画面显示方法、装置、虚拟现实设备及存储介质 |
CN114827569B (zh) * | 2022-04-24 | 2023-11-10 | 咪咕视讯科技有限公司 | 画面显示方法、装置、虚拟现实设备及存储介质 |
CN114915727A (zh) * | 2022-05-12 | 2022-08-16 | 北京国基科技股份有限公司 | 一种视频监控画面构建方法及装置 |
CN114915727B (zh) * | 2022-05-12 | 2023-06-06 | 北京国基科技股份有限公司 | 一种视频监控画面构建方法及装置 |
CN115022613A (zh) * | 2022-05-19 | 2022-09-06 | 北京字节跳动网络技术有限公司 | 一种视频重建方法、装置、电子设备及存储介质 |
CN115396644A (zh) * | 2022-07-21 | 2022-11-25 | 贝壳找房(北京)科技有限公司 | 基于多段外参数据的视频融合方法及装置 |
CN115396644B (zh) * | 2022-07-21 | 2023-09-15 | 贝壳找房(北京)科技有限公司 | 基于多段外参数据的视频融合方法及装置 |
CN116580099A (zh) * | 2023-07-14 | 2023-08-11 | 山东艺术学院 | 一种基于视频与三维模型融合的林地目标定位方法 |
CN117218320A (zh) * | 2023-11-08 | 2023-12-12 | 济南大学 | 基于混合现实的空间标注方法 |
CN117218320B (zh) * | 2023-11-08 | 2024-02-27 | 济南大学 | 基于混合现实的空间标注方法 |
Also Published As
Publication number | Publication date |
---|---|
CN111640181A (zh) | 2020-09-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021227360A1 (zh) | 一种交互式视频投影方法、装置、设备及存储介质 | |
WO2021227359A1 (zh) | 一种无人机投影方法、装置、设备及存储介质 | |
WO2018214365A1 (zh) | 图像校正方法、装置、设备、系统及摄像设备和显示设备 | |
WO2020001168A1 (zh) | 三维重建方法、装置、设备和存储介质 | |
KR101923845B1 (ko) | 이미지 처리 방법 및 장치 | |
CN110070564B (zh) | 一种特征点匹配方法、装置、设备及存储介质 | |
WO2019042419A1 (zh) | 图像跟踪点获取方法、设备及存储介质 | |
CN109389555B (zh) | 一种全景图像拼接方法及装置 | |
WO2019052534A1 (zh) | 图像拼接方法及装置、存储介质 | |
WO2010028559A1 (zh) | 图像拼接方法及装置 | |
US11620730B2 (en) | Method for merging multiple images and post-processing of panorama | |
CN111915483B (zh) | 图像拼接方法、装置、计算机设备和存储介质 | |
WO2021035627A1 (zh) | 获取深度图的方法、装置及计算机存储介质 | |
CN110781823A (zh) | 录屏检测方法、装置、可读介质及电子设备 | |
US20220405968A1 (en) | Method, apparatus and system for image processing | |
CN111866523A (zh) | 全景视频合成方法、装置、电子设备和计算机存储介质 | |
TW202244680A (zh) | 位置姿勢獲取方法、電子設備及電腦可讀儲存媒體 | |
CN112288878B (zh) | 增强现实预览方法及预览装置、电子设备及存储介质 | |
CN112215749A (zh) | 基于柱面投影的图像拼接方法、系统、设备及存储介质 | |
WO2023066143A1 (zh) | 全景图像的图像分割方法、装置、计算机设备和存储介质 | |
CN112085842A (zh) | 深度值确定方法及装置、电子设备和存储介质 | |
CN115514887A (zh) | 视频采集的控制方法、装置、计算机设备和存储介质 | |
CN115086625A (zh) | 投影画面的校正方法、装置、系统、校正设备和投影设备 | |
Xu et al. | Real-time keystone correction for hand-held projectors with an RGBD camera | |
CN111161148B (zh) | 一种全景图像生成方法、装置、设备和存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20934903 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20934903 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 28.04.2023) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20934903 Country of ref document: EP Kind code of ref document: A1 |