WO2019154169A1 - 跟踪交互装置的方法、存储介质以及电子设备 - Google Patents
跟踪交互装置的方法、存储介质以及电子设备 Download PDFInfo
- Publication number
- WO2019154169A1 WO2019154169A1 PCT/CN2019/073578 CN2019073578W WO2019154169A1 WO 2019154169 A1 WO2019154169 A1 WO 2019154169A1 CN 2019073578 W CN2019073578 W CN 2019073578W WO 2019154169 A1 WO2019154169 A1 WO 2019154169A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- marker
- target image
- target
- feature point
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
Definitions
- the application number submitted to the Chinese Patent Office on February 6, 2018 is CN201810119387.0.
- the name is “image processing method and device”.
- the application number submitted to the Chinese Patent Office on February 6, 2018 is CN201810119323.0 entitled "Virtual Scene Processing Method, Device, Interactive System” Head-mounted display device priority, a visual interaction device, and a computer-readable medium "in the Chinese patent application in its entirety by reference in the present application.
- the present application relates to the field of interaction technologies, and in particular, to a method for tracking an interaction device, a storage medium, and an electronic device.
- augmented reality is a technology that increases user perception of the real world through information provided by a computer system. It superimposes computer-generated virtual objects, scenes, or system prompt information into real scenes to enhance or modify the real world. The perception of the environment or data representing the real world environment.
- the embodiment of the present application provides an image processing method, including: acquiring a target image collected by an image capturing device, where the target image includes a marker disposed on the interaction device, the interaction device is located in a real scene; and according to the target image Determining position and posture information of the interaction device in the real scene; determining a virtual scene corresponding to the interaction device according to the position and posture information.
- the embodiment of the present application provides an image processing method, including: acquiring a first threshold image corresponding to a current frame image of a continuous multi-frame image except a first frame image, where the first threshold image is processed by processing a historical frame image. And a grayscale image having the same resolution as the current frame image; for each pixel of the current frame image, the pixel of the corresponding position in the first threshold image is used as a binarization threshold, and the current frame image is binarized.
- An embodiment of the present application provides an image processing method, including: acquiring a target image including a marker; processing the target image, and acquiring an enclosing relationship between the plurality of connected domains in the target image; An enclosing relationship between the plurality of connected domains in the image, and a feature of the pre-stored tag, determining identity information of the tag in the target image as identity information of the corresponding pre-stored tag.
- An embodiment of the present application provides an image processing method, including: acquiring a target image having an interaction device, and pixel coordinates of feature points in the interaction device in the target image, the interaction device including a plurality of sub-markers, each of the sub-markers
- the marker includes one or more feature points
- An embodiment of the present application provides an image processing method, including: acquiring a target image having a marker, the marker being distributed on one or more faces of the interaction device; and confirming a marker in the target image Identity information; determining, according to the marker information of the target image and the identity information of the marker, a tracking method adopted by the interaction device corresponding to the marker; acquiring the interaction device and the image collection according to a corresponding tracking method Position and attitude information between devices.
- the embodiment of the present invention provides an image processing method, including: acquiring a target image with an interaction device collected by an image collection device, where the target image includes a plurality of coplanar target feature points in the interaction device; The pixel coordinates of the target feature point in the target image in the image coordinate system corresponding to the target image; acquiring the image collection according to the pixel coordinates of the target feature point and the physical coordinates corresponding to the target feature point acquired in advance Position and posture information between the device and the interaction device, wherein the physical coordinate is a coordinate of a target feature point acquired in advance in a physical coordinate system corresponding to the interaction device.
- the embodiment of the present invention provides an image processing method, including: acquiring a target image with an interaction device collected by an image collection device, where the target image includes target feature points distributed on at least two faces in the interaction device; Obtaining the pixel coordinates of the target feature point in the target image in the image coordinate system corresponding to the target image; acquiring the image according to the pixel coordinates of the target feature point and the physical coordinates of the target feature point acquired in advance a position and posture information between the collection device and the interaction device, wherein the physical coordinate is a coordinate of the target feature point acquired in advance in a physical coordinate system corresponding to the interaction device.
- An embodiment of the present application provides a computer readable storage medium storing one or more computer programs, when the one or more computer programs are executed by one or more processors, for performing the following steps: acquiring an image acquisition device a captured target image, the target image includes a marker disposed on the interaction device, the interaction device is located in a real scene; and determining location and posture information of the interaction device within the real scene according to the target image; Determining a virtual scene corresponding to the interaction device according to the location and posture information.
- An embodiment of the present application provides an electronic device including one or more processors and a memory, the memory storing one or more computer programs, the one or more computer programs being executed by the one or more processors And performing the following steps: acquiring a target image acquired by the image capturing device, the target image including a marker disposed on the interaction device, the interaction device being located in a real scene; determining the interaction device according to the target image Position and posture information in the real scene; determining a virtual scene corresponding to the interaction device according to the position and posture information.
- FIG. 1 is an architectural diagram of an identification tracking system in an embodiment
- FIGS. 2a, 2b are schematic views of markers in the embodiment of the present application.
- Figure 3a is a structural diagram of an interaction device in an embodiment
- Figure 3b is a structural diagram of an interaction device in another embodiment
- Figure 3c is a structural diagram of an interaction device in another embodiment
- Figure 3d is a structural diagram of an interaction device in another embodiment
- Figure 3e is a structural diagram of an interaction device in another embodiment
- Figure 4 is a structural view of a multi-sided marking structure in one embodiment
- Figure 5 is a structural view of the multi-faceted marking structure shown in Figure 4 in another embodiment in another embodiment;
- Figure 6 is a structural view of a planar marking object in one embodiment
- Figure 7 is a schematic illustration of a marker in another embodiment
- Figure 8 is a flow chart of an image processing method in an embodiment
- FIG. 9 is a schematic diagram showing the position and posture between the first marker board and the twenty-six-face marker structure observed by the user in one embodiment
- FIG. 10 is an effect diagram of a virtual scene displayed in one embodiment
- 11 is a schematic diagram of different virtual scenes displayed based on different position and posture information between the interaction device and the image collection device in one embodiment
- FIG. 12 is a schematic diagram of different virtual scenes displayed based on different position and posture information between multiple interaction devices in one embodiment
- Figure 13 is a flow chart of an image processing method in another embodiment
- FIG. 14 is a flowchart of acquiring a first threshold image P1 in one embodiment
- FIG. 15 is a flowchart of acquiring a second threshold image P2 in one embodiment
- Figure 16a is a schematic diagram of calculating pixel values in one embodiment
- Figure 16b is a schematic diagram of calculating pixel values in another embodiment
- Figure 17 is a schematic illustration of bilinear interpolation in one embodiment
- Figure 18 is a schematic illustration of a marker in yet another embodiment
- Figure 19 is a tree diagram of the enclosing relationship of connected domains in an embodiment
- 20 is a flowchart of tracking and positioning an interactive device by a plane positioning and tracking method in an embodiment
- Figure 21a is a schematic illustration of an image coordinate system in one embodiment
- Figure 21b is a schematic diagram of a physical coordinate system in one embodiment
- 22 is a flow chart of obtaining physical coordinates of a target feature point in an embodiment
- FIG. 23 is a schematic diagram of expanding a new centroid in a target image in one embodiment
- 24 is a schematic diagram of expanding a new centroid in a preset marker model in one embodiment
- 25 is a schematic diagram of mapping a feature point of a target image to a coordinate system of a preset marker model in an embodiment, and acquiring a corresponding model feature point;
- 26 is a flow chart of tracking and positioning an interactive device by a stereo tracking method in an embodiment
- Figure 27 is a schematic illustration of an image coordinate system in another embodiment
- FIG. 1 illustrates an identification tracking system 10 provided by an embodiment of the present application, including a head mounted display device 100 and an interaction device 200, wherein the interaction device 200 has at least one marker thereon.
- the head mounted display device 100 may collect an image of the marker including the interaction device 200, and perform identification tracking of the marker of the interaction device 200 according to the acquired image to acquire the position and rotation information of the interaction device 200, thereby according to the interaction device 200.
- Location and rotation information shows virtual content.
- the head mounted display device 100 includes a housing (not labeled), an image capture device 110, a display device 120, an optical assembly 130, a processor 140, and a lighting device 150.
- the display device 120 and the image capture device 110 are both electrically connected to the processor 140.
- the illumination device 150 and the image capture device 110 are both disposed through a filter (not labeled) and covered in a housing, the filter can filter ambient light and the like, for example, the illumination device 150 emits infrared light, the filter A component that filters out light other than infrared light.
- the image capture device 110 is configured to acquire an image of the object and send it to the processor 140. Specifically, the image capture device 110 captures an image including at least one of the above-described planar mark plate or multi-face mark structure and transmits it to the processor 140.
- the image capture device 110 is a monocular camera adopting an infrared receiving method, which is not only low in cost, does not require external parameters between binocular cameras, and has low power consumption, and has a higher frame rate under the same bandwidth.
- the processor 140 is configured to output the corresponding display content to the display device 120 according to the image, and perform an operation of identifying and tracking the interaction device 200.
- Processor 140 may comprise any suitable type of general purpose or special purpose microprocessor, digital signal processor or microcontroller.
- the processor 140 can be configured to receive data and/or signals from various components of the system via, for example, a network.
- Processor 140 may also process data and/or signals to determine one or more operating conditions in the system. For example, when the processor 140 is applied to the head mounted display device, the processor performs an identification tracking operation on the interaction device 200 according to the image acquired by the image collection device, generates corresponding virtual display content, and transmits the display content to the display device 120 for display. And projecting the display content to the user through the optical component 130.
- the processor 140 is not limited to being installed in the head mounted display device 100.
- the head mounted display device 100 further includes a visual range camera 160 disposed on the housing and coupled to the processor 140 for capturing a scene image of an outside real scene and the scene image Send to processor 140.
- the processor 140 uses the visual mileage technology to acquire the position and posture relationship of the user's head in the real scene according to the scene image captured by the visual range camera 160.
- the processor 140 obtains a change in the specific position and direction of the head mounted display device 100 through the image sequence acquired by the visual mileage camera 160, through feature extraction, feature matching and tracking, and motion estimation, etc., thereby obtaining a header.
- the relative position and posture relationship of the display device 100 with the real scene and the position of the head mounted display device 100 in the real world are realized to achieve navigation positioning.
- the processor 140 can calculate the relative position and attitude relationship between the interaction device 200 and the real scene, thereby achieving a deeper interaction form and experience.
- the display device 120 is configured to display the display content output by the processor 140.
- display device 120 can be part of a smart terminal that is coupled to head mounted display device 100, ie, a display screen of the smart terminal, such as a display screen for a cell phone and a tablet.
- display device 120 can be a stand-alone display (eg, LED, OLED or LCD), etc., where the display device is fixedly mounted on the housing.
- the housing is provided with a mounting structure for mounting the smart terminal, and the smart terminal is installed on the housing through the mounting structure during use, and the processor 140 can It is a processor in the smart terminal, and may be a processor independently disposed in the casing, and connected to the smart terminal through a data line or a communication interface; when the display device 120 is a display device separated from a terminal such as a smart terminal, the display device 120 can be fixed to the housing.
- the optical component 130 is configured to project the light emitted by the display device 120 to a preset position, which may be an observation position of the user's eyes when the user wears the head mounted display device 100.
- Illumination device 150 is used to provide light when image capture device 110 is acquiring an image of an object. Specifically, the illumination angle of the illumination device 150 and the number of illumination devices 150 can be set according to actual use so that the emitted illumination light can cover the target object. Wherein, the illumination device 150 adopts an infrared illumination device capable of emitting infrared light. At this time, the image acquisition device 110 is a near-infrared camera and can receive infrared light. The number of the illumination devices 150 is not limited and may be one or plural. In some embodiments, the illumination device 150 is disposed adjacent to the image capture device 110, for example, a plurality of illumination devices 150 can be disposed adjacent to the image capture device 110. The application can improve the image quality of the target image collected by the image acquisition device 110 by means of active illumination.
- the interaction device 200 can be a planar marker object or a multi-faceted marker structure.
- the planar marking object includes a first marking plate 310 and a second marking plate 320.
- the multi-sided marking structure includes a six-sided marking structure 410 and a twenty-six-sided marking structure 420, and may also be other surface numbers. Marking structures are not listed here.
- the planar marking object has a marking surface on which the marking is disposed, which may be the first marking plate 310 or the second marking plate 320.
- the first marking plate 310 is provided with a plurality of markers, the contents of each of the markers are different, all the markings are disposed on the marking surface of the first marking plate 310, and the markings of the first marking plate 310 are The feature points are on the marked surface.
- a mark is disposed on the second marking plate 320, and the feature points of the markings on the second marking plate 320 are also all on the marking surface.
- the number of the second marking plates 320 may be plural, and the contents of the markings of each of the second marking plates 320 are different from each other, and the plurality of second marking plates 320 may be associated with the identification tracking system 10 Used in combination in areas such as augmented reality or virtual reality.
- the multi-faceted marking structure has a plurality of marking faces, and at least two non-coplanar marking faces are provided with markers.
- the multi-sided marking structure may be a six-sided marking structure 410, a twenty-six-sided marking structure 420, or the like.
- the six-sided marking structure 410 includes six marking surfaces, each of which is provided with a marking, and the patterns of the markings on each surface are different from each other.
- the twenty-six-sided marking structure 420 includes twenty-six faces, wherein twenty-six faces can be provided with 17 marking faces, each of which is provided with a marker, and the markings on each face The patterns are different from each other.
- the total number of faces of the multi-faceted mark structure and the description of the mark surface and the setting of the mark may be determined according to actual use, and are not limited herein.
- the interaction device is not limited to the above-mentioned planar marker object and multi-faceted marker structure, and the interaction device may be any carrier with a marker, and the carrier may be set according to an actual scene, such as a model gun such as a toy gun or a game gun.
- a corresponding marker is set on the interactive device such as the model gun.
- the interaction device 200 includes a first background and at least one marker distributed over the first background according to a particular rule.
- the marker includes a second background and a plurality of sub-markers distributed to the second background according to a particular rule, each sub-marker having one or more feature points.
- the first background and the second background have a certain degree of discrimination.
- the first background may be black and the second background may be white.
- the distribution rules of the sub-markers in each marker are different, and therefore, the images corresponding to each marker are different from each other.
- the sub-marker may be a pattern having a shape, and the color of the sub-marker has a certain degree of discrimination from the second background in the marker, for example, the second background is white, and the sub-marker is black.
- the sub-marker may be composed of one or more feature points, and the shape of the feature point is not limited, and may be a dot, a ring, or other shapes such as a triangle.
- the marker 210 includes a plurality of sub-markers 212, each sub-tag 212 being comprised of one or more feature points 214, each of which is a feature point 214 in Figure 2a.
- the shape of the marker 210 is a rectangle. Of course, the shape of the marker may be other shapes. The shape of the marker is not limited.
- the white area of the rectangle (ie, the second background) and the plurality of sub-markers 212 in the white area constitute the marker. 210. As shown in FIG.
- the marker 210' includes a plurality of sub-markers 212', each of the sub-markers 212' being composed of one or more feature points 214', which may be black dots or White dots.
- one sub-marker 212' may include one or more black dots 214', and one sub-marker 212' may also include one or more white dots 214'.
- the image capture device 110 collects the target image including the interaction device;
- the target image and the related information are obtained by the operation 140, and the interaction device is recognized, and the position and rotation relationship between the marker in the target image and the image acquisition device are acquired, thereby obtaining the interaction device relative to the head mounted display device 100.
- the position and attitude relationship make the virtual scene viewed by the user at the corresponding position and posture angle.
- the user can also enhance the user's experience by using a combination of multiple interactive devices to further generate new virtual images within the virtual scene.
- the user can also interact with the virtual scene through the interaction device.
- the identification tracking system 100 can also acquire the position and rotation relationship between the head mounted display device 100 and the real scene through the visual range camera 160, thereby acquiring the position and rotation relationship of the interaction device and the real scene, and the head mounted display device 100 is In the real world position, when the virtual scene has a certain correspondence with the real scene, a virtual scene similar to the real scene can be constructed to further improve the user experience.
- the interaction device includes a device body and one or more markers disposed on a surface of the device body.
- the marker may be disposed on one surface of the planar marking object, as shown in FIG. 3a
- the first marking plate 310 includes the device body 311, and one or more devices disposed on the surface of the device body 311.
- Marker 210 When the interaction device is a multi-faceted mark structure, the mark may be disposed on one or more surfaces of the multi-face mark structure, as shown in FIG. 3b, the six-sided mark structure 410 includes the device body 411, and is disposed on the device body 411.
- a surface marker 210 as shown in FIG.
- the device body 411 of the six-sided marking structure 410 includes a plurality of surfaces, and the marker 210 is disposed at the boundary of two adjacent surfaces in the device body 411, that is, one The markers are disposed on the surfaces of adjacent plurality of planes.
- the markers may also be disposed on the same surface of the device body having different planes, such as on a spherical surface, a curved surface, etc., as shown in Figure 3e, the marker 210 is disposed on the spherical surface of the device body 431.
- the manner in which the device body and the marker disposed on the device body in the interaction device are not limited to the above descriptions, the device body may have other shapes, and the marker may be set in other manners, and is not used herein. limited.
- one or more markers in the interaction device may be prominently disposed on the body of the device, i.e., the marker is a layer structure disposed on the surface of the device body.
- the surface of the device body may be provided with a groove corresponding to the number of the markers, and the marker is correspondingly disposed in the groove of the surface of the device body, and the depth of the groove may be equal to the thickness of the marker, so that the marker is outside The surface is flush with the top of the groove.
- the depth of the groove is not limited in the embodiment of the present application.
- the multi-faceted marker structure 400 has markers 210 to allow identification and tracking by the external image capture device 110.
- the multi-faceted indicia structure 400 includes a device body 401 and a handle 402 coupled to the device body 401.
- the handle 402 is provided with a connection (not shown) and the device body 401 is coupled to the connection.
- the device body 401 is provided with a marker 210.
- the image capture device 110 acquires an image including the marker 210, and the processor acquires information carried by the multi-face marker structure 400 according to the image, including identity information of the multi-faceted marker structure 400, and
- the identification tracking of the multi-sided marking structure 400 is realized with respect to the position and rotation information of the head-mounted display device, and the virtual content of the head-mounted display device is determined based on the positional rotation information.
- the specific configuration of the device body 401 is not limited.
- the device body 401 is a hexahedron, which includes eighteen square faces and eight triangular faces.
- the device body 401 includes a first surface 403 and a second surface 404 that are not coplanar with the first surface 403.
- the first surface 403 is provided with a first marker 220
- the second surface 404 is provided with a second marker 230 different from the first marker 220.
- the image capture device recognizes either or both of the first marker 220 and the second marker 230, and acquires position and orientation information of the multi-face marker structure 400 to identify and track the multi-face marker structure 400.
- first surface 403 and the second surface 404 may be disposed adjacent to each other, spaced apart from each other, or the first surface 402 and the first surface
- the two surfaces 404 may be eighteen square faces and any two of the eight triangular faces, and are not limited to the description herein.
- the device body 401 further includes any one or more of a third surface, a fourth surface, a fifth surface, a twenty-sixth surface (not identified), and correspondingly, the surfaces may be provided with corresponding
- the marker 210 has different information for the marker 210 on each surface.
- the planar marking object 300 includes a device body (not shown) having a base layer 302 on the main body 302 and one or more markers 210 disposed on the base layer 302. When the plurality of markers 210 are plural, the plurality of markers 210 are dispersedly disposed on the base layer 302.
- the base layer 302 may be made of a soft material such as cloth, plastic, etc.; the base layer 302 may also be made of a hard material such as cardboard, metal material, or the like.
- the base layer 302 can be provided with a fold to provide the base layer 302 with a folding function to facilitate folding storage.
- the planar marker object 300 is provided with two folds perpendicular to each other, and the two folds can divide the planar marker object 300 into four regions, and the four markers of the plane mark the object 300 by two folds. After the regions are folded, the planar marker objects 300 can be stacked into one region size.
- the shape of the base layer 302 is not limited, and may be, for example, a circle, a triangle, a square, a rectangle, an irregular polygon, or the like.
- the marker 210 includes a plurality of sub-markers 212 that are separated from each other, and each feature point 214 in each of the sub-markers 212 is separated from each other.
- the number of feature points 214 included in each sub-marker 212 is not limited and may be determined according to the actual identification requirement and the size of the area occupied by the marker 210.
- the shape of each feature point 214 is not limited and may be a triangle, a quadrangle or a circle.
- the sub-marker 212 can be a hollow pattern comprising one or more hollow portions, wherein each hollow portion can serve as a feature point 214, such as a black sub-segment including three white dots 214 in FIG. Marker 212a is shown.
- a solid figure may be further disposed on any hollow portion of the sub-marker 212, and the solid figure is used as the feature point 214 corresponding to the hollow portion of the sub-marker 212, as shown in FIG. Marker 212b is shown.
- a hollow pattern such as a circular ring, may be provided in the hollow portion of the sub-marker 212, with a hollow pattern of the hollow portion as a corresponding one of the feature points 214 in the sub-marker 212.
- a layered hollow pattern such as a nested circle, is placed in the sub-marker, with the last nested hollow circle as the feature point 214.
- the number of nesting layers of the hollow pattern in the sub-marker 212 can be set according to actual identification requirements or determined according to the resolution of the image capturing device.
- each sub-marker 212 of the marker 210 there may be one sub-marker 212 consisting of a solid pattern separated from each other, each solid pattern being a feature point 214.
- each solid pattern being a feature point 214.
- the respective black solid circles 214 separated from each other constitute one sub-marker 212c, and each black solid circle is a feature point 214 in the sub-marker 212c.
- the identity information of each of the markers 210 is determined, and the contents of each of the markers 210 are different from each other in one virtual scene.
- the number of sub-markers 212 included in the marker 210 is different from the number of sub-markers included in the other markers.
- the number of the sub-markers 212 of the three markers 210 are x, y, and z, respectively, wherein x, y, and z may be integers greater than or equal to 1, x, y, z is not equal.
- the type of feature points 214 that may be at least one sub-marker 212 in the marker 210 is different than the type of feature points 214 of the sub-marker 212 in other markers 210, such as one of the markers 210.
- the sub-marker 212 includes a feature point 214 that is a solid circle, and none of the other markers 210 includes a sub-marker 212 whose feature point 214 is a solid circle.
- the number of nesting layers of the hollow pattern in at least one of the sub-markers 212 in the marker 210 may be different from the number of nesting layers of the sub-marker 212 of the other markers 210.
- only one hollow portion of one of the markers 210 is provided with a solid dot that serves as the feature point 214 of the subtag 212.
- the processor recognizes the sub-marker 212 with a solid dot disposed in the hollow portion, the identity of the marker 210 corresponding to the sub-marker 212 can be determined, and the marker 210 is in the preset marker model.
- the hollow portion is provided with a marker 210 corresponding to the sub-marker 212 of a solid dot.
- the number combination of the markers 210 is different from the number combination corresponding to the other markers 210.
- the number of feature points 214 of each sub-marker 212 in each marker 210 constitutes a quantity combination in the marker 210.
- the marker 210 includes four sub-markers 212, wherein the number of feature points of the sub-marker 212a is 3, the number of feature points of the sub-marker 212b is 2, and the number of feature points of the sub-marker 212c 5, the number of feature points of the sub-marker 212d is 1, and the number of feature points 214 of the four sub-markers forms a quantity combination in the marker 210.
- the number combination can be a combination of numbers that arrange the sub-markers in a certain direction.
- the number combination of the sub-markers arranged in the sequential needle direction may be 3152, and the number combination in the counterclockwise direction may be 3251 or the like, wherein the sub-marker which is the starting point of the quantity combination may be any selected one of the sub-marks.
- the number combination corresponding to the marker 210 can also be expressed in other ways, and is not limited to the manner described above.
- the interaction device may be a planar marker object, a surface marker object or a stereo marker structure, etc., and may be designed according to different virtual scenes.
- FIG. 8 shows an image processing method of the present application, which is applied to the above-described identification tracking system, with the processor of the head mounted display device as an execution subject.
- the identification tracking system includes an image acquisition device and an interaction device having a marker.
- the method may include steps S110 to S130.
- Step S110 The processor acquires a target image with a marker collected by the image collection device.
- the interaction device is located in a real scene.
- the target image is an image of the interaction device collected by the image acquisition device, and the target image includes the marker of the interaction device.
- the interaction device may be any of the interaction devices mentioned in the above embodiments, or may be other structural forms. Interactive device.
- Step S120 The processor determines position and posture information of the interaction device in the real scene according to the target image.
- the position and posture information of the interaction device in the real scene may include information such as a position and a rotation angle of the interaction device in the real scene.
- the location information may refer to spatial location information of the interaction device in the real scene
- the posture information may refer to rotation information of the interaction device, where the location and posture information may be a location between the interaction device and the image collection device.
- the interaction device in the captured target image may be one or more. When there are multiple interaction devices in the acquired target image, the processor may acquire position and posture information between each interaction device and the image collection device within the target image.
- the processor acquires the target image and identifies the markers contained in the target image to determine the identity information of the markers in the target image.
- the processor may determine the interaction device corresponding to the marker according to the identity information of the marker, and generate a corresponding virtual object; and determine whether the interaction device is a planar marker object or a multi-face marker structure, to use the corresponding location tracking method to the interaction device. Tracking is performed to obtain information such as the position and posture between the interactive device and the image capturing device.
- Step S130 The processor determines a virtual scene corresponding to the interaction device according to the position and posture information.
- the processor can determine the display content corresponding to the interaction device according to the position and posture information of the interaction device, and present the display content to the user through the display device and the optical component of the head display device to generate an effect that the virtual scene is superimposed on the real scene.
- the correspondence between the different position and posture information and the display content may be pre-stored in the head-mounted display device, and after the processor acquires the position and posture information between the interaction device and the image collection device, according to the corresponding The relationship is to find the display content corresponding to the position and posture information between the current interaction device and the image acquisition device.
- the processor sends the display content to the display device, instructing the display device to display the display content corresponding to the position and posture information, and the display device displays the display content and projects the corresponding position through the optical component.
- the corresponding position may be the user's binocular position, and the user's eyes can observe the display content.
- the optical component has a certain degree of transparency, the real environment is also observed by the user, and the user can observe the visual effect that the display content is superimposed with the real environment.
- the user may use multiple interaction devices to generate more display content within the virtual scene, further improving the user's experience; the user may also interact with the displayed virtual content through the interaction device.
- the planar marking object and the multi-sided marking structure are included in the visual range of the image capturing device.
- the planar marking object may be a first marking plate
- the multi-sided marking structure may be twenty-six surfaces. Mark the structure.
- the image acquisition device collects an image of the interaction device in the field of view of the user
- the processor analyzes the image, determines the identity information of the first marker board, and the position and posture information between the head mounted display device and the first marker board, and generates and generates a corresponding correspondence.
- the display content is displayed on the display device, and the display content is projected to the user's glasses through the optical component.
- the user can view the real scene through the optical component, thereby observing the visual effect of the overlay of the display content with the real scene of the outside world.
- the virtual object w1 represents a water cup
- the virtual object w2 represents a soup spoon
- the soup spoon contains food
- the virtual object w3 represents a table.
- the display position of the virtual object w1 ie, the position seen by the user
- the display position of the virtual object w1 may correspond to the position of the marker 210A in the real scene in FIG. 9
- the display position of the virtual object w2 may correspond to the first marker panel in FIG.
- the display position of the virtual object w3 may correspond to the position of the twenty-six-sided mark structure in FIG. 9 in the real scene.
- the user can hold the twenty-six-sided marking structure and move to the position of the marker 210A on the first marking plate to reach the position shown by the twenty-six-sided marking structure in FIG. Wearing the display device sees a virtual image as shown in FIG.
- the display content displayed by the display device is reasonably set, so that when the virtual image shown in FIG. 10 is observed, the virtual image can be accurately superimposed on the first marking plate and the twenty-six-sided marking structure, and the visual effect is obtained. Better.
- the virtual reality scene of the augmented reality displayed in the head mounted display device also changes accordingly.
- the processor acquires the amount of change of the posture information between the interaction device and the image collection device, and adjusts the displayed display content according to the amount of change, so that the augmented reality scene changes correspondingly according to the amount of change of the posture information.
- FIG. 11 is a schematic diagram of different virtual scenes displayed based on different position and posture information between the interaction device and the image acquisition device in one embodiment.
- the posture information between the interaction device and the image acquisition device is S1
- the information changes, and the virtual scene displayed by the head mounted display device may change as the posture information changes.
- the posture information between the interactive device and the image capturing device becomes S2
- the virtual scene displayed by the head mounted display device may become a virtual scene as shown in FIG. 11(b).
- the virtual object w1 shown in FIG. 11( a ) may be a front side, and the virtual object w1 shown in FIG.
- 11( b ) may be a back side, so that the posture between the interaction device and the image capturing device can be seen.
- the user can observe the virtual object at different visual angles, for example, the change process from the front side of the virtual object w1 to the back side of the virtual object w1 can be observed.
- the rectangular frame in FIG. 11 is only used to indicate the size of the image, and the user may not see the rectangular frame while observing.
- the processor determines the position between the interaction devices according to the position and posture information of each interaction device.
- the posture information determines the displayed virtual scene according to the position and posture information between each interaction device and the image collection device, and the position and posture information between the interaction devices, and determines a virtual image corresponding to each interaction device, and multiple virtual images.
- the image is used to form a virtual scene.
- the processor may determine whether the location and posture information between the at least two interaction devices meet a preset criterion. When the preset criterion is met, the processor may modify the virtual image corresponding to the at least two interaction devices to Make the displayed virtual scene change.
- the preset standard is a standard set according to needs, for example, a preset angle or a preset distance value.
- FIG. 12 is a schematic diagram of different virtual scenes displayed based on different position and posture information between a plurality of interactive devices in one embodiment.
- the image capturing device collects an image of the first marking plate
- the head mounted display device displays as shown in FIG. 12 .
- the virtual scene shown in (a) wherein the virtual scene superimposes a candle on the table, wherein the position of the candle may be the position of a certain marker of the first marker panel, and the user holds the twenty-six-sided marker structure.
- the image capturing device simultaneously collects images of the first marking plate and the twenty-six marking structure, and the head mounted display device displays the virtual scene as shown in FIG. 12(b).
- the burning matchstick may be a virtual image corresponding to the twenty-six-sided mark structure, and the position and posture between the first mark plate and the twenty-six mark mark structure are changed, for example, the twenty-six mark mark structure gradually Near the marker 210A of the first marker panel, the virtual scene displayed may be a burning matchstick that gradually approaches the candle.
- the virtual scene displayed by the head-mounted display device may be a burning matchstick to ignite the candle, and the twenty-six-sided marking structure disappears in the field of view of the image capturing device.
- the virtual scene displayed by the head-mounted display device can be as shown in FIG. 12(c), and the virtual scene becomes a lit candle set on the table.
- FIG. 13 illustrates an image processing method in an embodiment of the present application, which is applicable to the identification tracking system illustrated in FIG. 1 with the processor of the head mounted display device as an execution subject.
- the method may include steps S110 to S130, wherein step S120 includes steps S122 to S126.
- Step S110 The processor acquires a target image with a marker collected by the image collection device.
- Step S120 The processor determines position and posture information of the interaction device in the real scene according to the target image.
- Step S120 includes steps S122 to S126.
- Step S122 the processor confirms the identity information of the marker in the target image.
- the processor acquires a target image with a marker collected by the image acquisition device, and the target image includes at least one marker having a plurality of sub-markers.
- the number of sub-markers included in the marker may be greater than or equal to four.
- the target image may also include a portion between the markers, that is, a portion of the first background.
- the processor can obtain the identity information of the marker according to the characteristics of the marker in the target image.
- the processor may pre-process the target image to obtain a processed target image that reflects various feature information in the target image.
- the processor preprocesses the target image, and distinguishes the first background, the second background, and the connected domains corresponding to the sub-markers and the feature points from the target image.
- pre-processing the target image may be performing binarization processing on the target image, so that there is a distinction between the first background and the sub-marker in the target image, and the sub-marker and the second background There is a distinction between them.
- the processor may perform binarization processing on the target image by using a fixed threshold method or an adaptive threshold method, and may also perform binarization by other methods, which is not limited herein.
- the first frame target image may be binarized by a global fixed threshold method, an inter-frame fixed threshold, an adaptive threshold method, or the like to obtain the first frame target image. Binarized image.
- the first frame target image captured by the processor after the image capture device is turned on is used as the first frame target image, and the target image of any frame in the image capture device may be used as the first frame target image.
- the continuous multi-frame target image is a multi-frame target image that is subsequently captured from the first frame target image, and the continuous multi-frame target image may be a target image sequentially adjacent to the time captured by the image acquisition device, or may be in time.
- the target image having a frame interval with each other is not limited in the embodiment of the present application, and may be determined according to actual needs.
- binarization processing can be performed in the manner described in the following embodiments.
- the processor may acquire a first threshold image P1 corresponding to the current frame target image except the first frame target image in the continuous multi-frame target image, and binarize the current frame target image according to the first threshold image P1.
- the processor binarizes any frame target image other than the first frame target image in the continuous multi-frame target image, the frame target image binarized may be used as the current frame target image.
- the processor may acquire a first threshold image P1 corresponding to the current frame target image, where the first threshold image P1 corresponding to the current frame target image is obtained by performing image processing on the historical frame target image and is distinguished from the current frame target image.
- the historical frame target image refers to the target image of the continuous multi-frame target image before the current frame
- the historical frame target image may be one or more frames.
- the resolution of the current frame target image is m*n
- the resolution of the first threshold image P1 is also m*n.
- the resolution of the current frame target image may be the resolution of the current frame target image acquired by an image acquisition device such as a camera.
- the processor processes the historical frame target image to obtain a first threshold image P1
- the pixel value of each pixel in the first threshold image P1 may be that the processor passes each of the historical frame target images.
- the pixel values obtained after processing are processed by other pixels around the pixel, and the obtained pixel values are taken as the pixel values of the corresponding pixel points in the first threshold image P1.
- the pixel value of each pixel in the first threshold image P1 is determined by a plurality of pixel points around the corresponding pixel in the history frame target image.
- the processor acquires a first threshold image P1 corresponding to a current frame target image other than the first frame target image in the continuous multi-frame target image.
- the method may include steps S221 to S223.
- Step S221 The processor acquires a second threshold image P2 processed by the historical frame target image, where the resolution of the second threshold image P2 is a first preset resolution, and the first preset resolution is lower than the current frame target image. Resolution.
- the first preset resolution of the second threshold image P2 may be a resolution within a required range of other external components such as hardware, for example, may be supported by the hardware end memory for storing the second threshold.
- the smaller the memory space the smaller the first preset resolution.
- the method of processing the historical frame target image by the processor and acquiring the second threshold image P2 may include steps S221a to S221c.
- Step S221a The processor downsamples the historical frame target image to obtain a downsampled image having a second preset resolution.
- the size of the second preset resolution is not limited, that is, the coefficient of down sampling is not limited.
- the N*N pixels in the historical frame target image are reduced to 1*1 pixels, wherein the size of N is not limited.
- the N1*N2 pixel points are reduced to 1*1 pixel points, and the values of N1 and N2 may be different, and the specific values of N1 and N2 are not limited.
- the historical frame target image may be downsampled to an image of the second preset resolution according to actual processing requirements.
- the specific implementation method for the processor to downsample the historical frame target image is not limited.
- the processor may obtain a pixel mean value for an N*N region with a row-column interval of N pixels in the historical frame target image, and compare the pixel mean as the N*N pixel points in the downsampled image.
- the pixel value of the pixel when the pixel mean is less than the preset minimum pixel value t, the pixel average may be set to the preset minimum pixel value t; or, the processor may be in the historical frame target In the image, the row and column take one pixel every N pixel points as the corresponding pixel point in the downsampled image; or, the processor can select the N1 pixel point in the historical frame target image, and the column interval is N2 The pixel value is obtained as the pixel value of the pixel corresponding to the N1*N2 pixel points after downsampling.
- Step S221b The processor calculates and acquires a third threshold image P3 having a second preset resolution according to the downsampled image.
- the processor may determine a pixel value of each pixel in the third threshold image P3 according to a pixel value of each pixel in each of the pixels in the downsampled image to obtain the second preset resolution.
- the pixel value of each pixel in the third threshold image P3 may be obtained according to pixel values of all pixels in the preset window range of the corresponding pixel in the downsampled image.
- the processor may determine a preset window corresponding to the pixel point of the xth row and the yth column in the downsampled image, and obtain a third threshold image P3 according to the pixel value of all the pixel points in the corresponding preset window range.
- the processor may perform an adaptive threshold operation on the downsampled image with a window size of W*W (the value of W is generally small) to obtain a third threshold image P3.
- the processor may perform an adaptive threshold operation on the pixel of the xth row and the yth column in the downsampled image on the W*W size window to obtain the pixel of the xth row and the yth column in the third threshold image P3. Pixel values.
- the integral map information of the downsampled image may be employed.
- the processor may obtain an integration map of the downsampled image.
- the pixel value of any pixel (x, y) in the integration map may be from the upper left corner of the downsampled image to the pixel (x, y) The sum of the gray values of all the pixels in the rectangular area formed.
- the processor may calculate and acquire a third threshold image P3 having a second preset resolution according to the integral map, and determine a third threshold image according to pixel values of each pixel in each preset pixel in the preset window. The pixel value of each pixel in P3 to obtain a third threshold image P3 having a second preset resolution.
- Each pixel point in the third threshold image P3 obtained by the processor may be obtained according to the pixel value of all the pixel points in the preset window range of the corresponding pixel point in the integration map.
- the processor may determine a preset window corresponding to the pixel of the xth row and the yth column in the integral map, and obtain the xth of the third threshold image P3 according to the pixel value of all the pixels in the corresponding preset window range.
- the pixel value of the pixel of the yth column may perform an adaptive threshold operation on the integral map with a window size of W*W (the value of W is generally small) to obtain a third threshold image P3.
- the processor may perform an adaptive threshold operation on a window of a W*W size corresponding to a pixel of the xth row and the yth column in the integration map, and obtain a pixel mean value of all pixels in the window corresponding to the W*W size.
- the pixel mean value is taken as the pixel value of the pixel point of the xth row and the yth column in the third threshold image P3.
- the processor may obtain the pixel value of the xth row and the yth column in the corresponding pixel average value in the window of the W*W size, multiply the mean magnification factor, and obtain the xth row in the third threshold image P3.
- the pixel value of the pixel of the y column may perform an adaptive threshold operation on a window of a W*W size corresponding to a pixel of the xth row and the yth column in the integration map, and obtain a pixel mean value of all pixels in the window corresponding to the W*W size.
- the pixel mean value is taken
- the processor may acquire the effective pixel point in the integral map corresponding to the preset window range of the pixel point, and according to the effective pixel The point calculates the pixel value of the corresponding pixel in the third threshold image P3.
- the pixel position of the pixel in the integration map is (i, j)
- the four vertex positions of the window corresponding to the pixel point are respectively (ia -1, ja-1), (i+a, ja-1), (ia-1, j+a), (i+a, j+a).
- the pixel values of the pixel points whose pixel position is (i, j) can be obtained according to the pixel values of all the pixels in the window.
- the actual effective area of the window may be an area where the shadow area coincides with the background grid, and the size of the actual effective area is smaller than the window size w.
- the processor can calculate the pixel value of the pixel point (i, j) in the third threshold image P3 according to the pixel value of the pixel included in the effective area of the window in the integration map, that is, according to the figure The pixel value of the pixel in the region where the shaded area coincides with the background grid in 16b is calculated.
- the size of the window in the above embodiment is not limited, and may be set according to actual needs. Further, the window size and the downsampling coefficient when obtaining the downsampled image may be determined according to the original image size, the maximum window size supported by the hardware, and the physical feature size of the image object in the target image of the historical frame to ensure the minimum image size.
- the image object can be embodied in the obtained third threshold image P3.
- the processor acquires the third threshold image P3 by the operation of the adaptive threshold, even the smallest image object in the image can be embodied in the adaptive threshold process.
- the preset window size may be different for different target images. Specifically, the corresponding window size may be set according to the size of the corresponding object in the target image. When the object is larger or close to the camera, a larger window may be set. window. Correspondingly, the set window size is different, and the pixel values of the respective pixel points in the third threshold image P3 obtained by the processor are also different.
- Step S221c when the second preset resolution is greater than the first preset resolution, the processor downsamples the third threshold image P3 until a second threshold image P2 whose resolution is less than or equal to the first preset resolution is obtained. .
- the processor may continue to downsample the obtained third threshold image P3 until the resolution is less than or equal to the first preset resolution.
- the second threshold image P2 is obtained to obtain a threshold image in which the storage space is as small as possible. For example, on the basis of the third threshold image P3 of the second preset resolution obtained by using N as the downsampling coefficient, sampling is continued with the downsampling coefficient M, and in the third threshold image P3, M*M is used.
- the pixel reduction is 1*1 pixels.
- the processor may continue to downsample the downsampled third threshold image P3 with the downsampling coefficient M until the resolution is less than Or a third threshold image P3 equal to the first preset resolution, and the third threshold image P3 as the second threshold image P2.
- the processor may not downsample the third threshold image P3, and the third threshold image P3 is the resolution A second threshold image P2 of the first preset resolution.
- the processor may store the obtained second threshold image P2 in a memory for later use.
- the second threshold image P2 only needs to store a very small image in the final program, which effectively saves memory space.
- the memory space occupied by the second threshold image is only 1/(N*N*M*) when the sample is not downsampled. M), for some hardware with strict memory requirements, such as FPGA, is crucial.
- the manner in which the second threshold image P2 is acquired in the foregoing embodiment may not be performed as a specific implementation of step S121, but may be performed independently, and the acquired second threshold image P2 is stored.
- the processor may directly acquire the pre-stored second threshold image P2 from the memory.
- Step S223 The processor upsamples the second threshold image P2 to obtain a first threshold image P1 having the same resolution as the current frame target image.
- the second threshold image P2 may be obtained according to the historical frame of the current frame, and the second threshold image P2 is upsampled.
- the second threshold image P2 having the resolution of the first preset resolution is subjected to upsampling, and the first threshold image P1 having the same resolution as the current frame target image is obtained.
- the second threshold image P2 is obtained by processing the historical frame target image according to the downsampling coefficient N. If the historical frame target image resolution is the same as the current frame, the second threshold image P2 is upsampled by the upsampling coefficient N.
- the specific implementation of the upsampling is not limited in the embodiment of the present application, and may be implemented by a bilinear interpolation algorithm or the like.
- bilinear interpolation also known as bilinear interpolation.
- the processor after the processor acquires the first threshold image P1, for each pixel of the current frame target image, the pixel value of the pixel corresponding to the position in the first threshold image P1 may be used as a binarization threshold.
- the current frame target image is binarized.
- the processor uses the pixel value of each pixel in the first threshold image P1 as the binarization threshold of the corresponding position pixel in the current frame target image.
- the corresponding position is a position where the coordinates are the same in the current frame target image and the first threshold image P1 having the same resolution, such as the pixel in the second row and the third column in the current frame target image, and the first The pixel points in the second row and the third column in the threshold image P1 are pixel points corresponding to the position.
- the processor may set a pixel value of the pixel in the current frame target image to a first pixel value; when a pixel value of a pixel point of the current frame target image is less than or equal to a pixel point of a corresponding position in the first threshold image P1
- the processor may set the pixel value of the pixel in the current frame target image to the second pixel value to obtain a binarized image of the current frame target image.
- the pixel value of the pixel whose position is (i, j) in the image is set to the first pixel value 1 in the binarized image; the pixel position of the position (I, J) in the current frame target image, the pixel value is 50 a pixel at a position (I, J) in the first threshold image P1, the pixel value is 200, and the pixel value of the pixel at the position (I, J) in the current frame target image is set to be in the binarized image.
- the second pixel value is 0.
- the processor may perform processing to obtain a second threshold image P2, which is used for upsampling the target image of the next frame to obtain a corresponding first threshold.
- the image P1 is binarized for the next frame of the target image.
- the corresponding first threshold image P1 may be acquired, and binarization processing is performed according to the corresponding first threshold image P1.
- the process of binarization of each target image is performed, and the process of obtaining the second threshold image P2 is performed, and the order is not limited.
- the process of binarizing each frame of the target image, and the process of obtaining the first threshold image P1 corresponding to the next frame target image after processing the frame target image, the order of processing may not be limited.
- the binarization thresholds of the respective pixels may not be the same, and the binarization threshold of each pixel depends on the first threshold image P1 corresponding to the target image, due to the history frame and There is continuity between the latter frames. Therefore, the binarization threshold of the target image is set to be most suitable for the current scene, and is updated in real time as the scene changes, which is more in line with the current binarization scene requirements.
- the processor binarizes each frame target image of the obtained continuous multi-frame target image, the first background, the second background, and the sub-markers included in the target image respectively correspond to the corresponding binarized pixel values.
- the processor may process the portion between the markers in the target image and the sub-marker into a first color, and the portion of the marker other than the sub-marker is the second colour.
- the processor processes the portions of the marker that are in a surrounding relationship in turn to have a color gradation such that the portions form a connected domain that is sequentially surrounded.
- the processor may process the portion corresponding to the first background 1810 in the target image as the first color, and the second background 1820 in the marker 210 as the second color.
- the marker 212 is treated to a first color and the hollow portion enclosed by the sub-marker is treated to a second color. If the hollow portion of the sub-marker also includes a solid pattern, as shown in sub-marker 212b in Figure 7, the solid pattern is processed into a second color.
- the first color and the second color may be colors having a large difference in pixel values, such as a first color being black and a second color being white.
- a first color being black
- a second color being white
- the binarized image, the first background, the second background, and the sub-markers and the feature points can be distinguished by other methods such as contrast.
- the embodiment of the present application mainly uses a color layer as an example for description.
- the processor may obtain the connected domain information in the target image, and acquire the enclosing relationship of all the connected domains based on the connected domain information, and then according to the enclosing relationship between the multiple connected domains in the target image and the characteristics of the pre-stored markers. Determining the identity information of the marker in the target image as the identity information of the corresponding pre-stored marker, wherein the connected domain refers to an image region composed of pixels having the same pixel value and adjacent positions in the image.
- the processor acquires connected domain information in the target image, and can calculate the connected component labeled as a Boolean image by using 4-way or 8-way connectivity, and output the number of connected domains, wherein each connectivity can be output according to the surrounding relationship.
- the type of the domain that is, the connected domain corresponding to each part of the first background, the second background, the sub-marker, the feature point, and the like of the target image.
- the first background 1810 is a connected domain
- the second background 1820 in the marker is a connected domain
- each of the sub-markers 212 not including the black dot is a connected domain
- the white point in the middle is a connected field, including sub-markers 212 of black points (i.e., feature points 214), each of which is a connected field.
- the sub-marker not including the black dot is a sub-marker of the hollow figure, wherein the white point is a feature point, the sub-marker including the black point, and the black point is a feature point.
- the processor may acquire an enclosing relationship between the connected domains based on the connected domain in the target image.
- the sub-marker 212a including three white points in FIG. 18 is a connected domain
- the connected domain includes three white points 214
- each white point 214 is a connected domain.
- the connected domain corresponding to the white point 214 is surrounded by the connected domain corresponding to the sub-marker 212a.
- a surrounding relationship is formed between the first background 1810, the second background 1820, and the sub-markers, and if the sub-marker is a hollow figure, the sub-marker and the hollow figure
- the hollow portion included in the corresponding portion also has a surrounding relationship, as shown in FIG. 18, including a sub-marker of a white point, and forms a surrounding relationship with the white point.
- the first background encloses the second background
- the second background encloses the sub-marker
- the sub-marker also surrounds the white point therein, that is, the hollow portion.
- the connected domains corresponding to the first background, the second background, and the sub-markers respectively have an enclosing relationship, and the connected domains corresponding to the sub-markers also have a surrounding relationship with the connected domains corresponding to the hollow portions.
- the connected domain corresponding to the first background may be defined as a fourth connected domain, and the processor may first determine the fourth connected domain.
- the first background encloses all the markers, and therefore, the connected domain surrounding all other connected domains in the target image can be regarded as the fourth connected domain.
- the determined fourth connected domain satisfies the following condition: the color is the first color, the connected domain surrounded by the second color, and is not Surrounded by connected domains of two colors.
- the first background of the target image encloses the marker, and the fourth connected domain surrounds the connected domain corresponding to the second background in the marker, and the connected domain corresponding to the second background may be defined as the first connected domain.
- the processor may be a connected domain surrounded by the fourth connected domain and adjacent to the fourth connected domain as the first connected domain, and each of the first connected domains surrounded by the fourth connected domain corresponds to a marker, and the marker surrounds the other
- the connected domain of the connected domain is the first connected domain. Taking the binarized target image including the first color and the second color as an example, the processor may determine that the connected domain surrounded by the fourth connected domain and adjacent to the fourth connected domain, and the connected color of the second color is the first Connected domain.
- each sub-marker includes a feature point in the marker, each sub-marker has a feature point, and the connected domain surrounded by the first connected domain and adjacent to the first connected domain may be defined as a second connected domain, that is, the sub-marker may be defined.
- the corresponding connected domain is the second connected domain.
- the connected domain surrounded by the second connected domain may be defined as a third connected domain, that is, if the child mark is a hollow figure surrounding the white point as shown in FIG. 18, the hollow part (ie, the surrounded white part, that is, the white feature point)
- the corresponding connected domain is defined as a third connected domain, and each third connected domain is a feature point.
- the enclosing relationship of the corresponding connected domains in FIG. 18 may be represented by a tree diagram as shown in FIG. 19, and in FIG. 19, B in the first level in the tree diagram may be Corresponding to the connected domain of the first background 1810 (the fourth connected domain); W in the second level may correspond to the connected domain of the second background 1820 (the first connected domain); B1, B3, B2, B5 in the third hierarchy Corresponding to the connected domains of the four sub-marks; w and b in the fourth level are respectively used to represent the connected points of the black points and white points included in the sub-marks, wherein W and B can be used to represent the connected domains, respectively.
- the color is white or black, and can also be used to indicate the code of the connected domain, which is not limited herein.
- Each of the first connected domains is surrounded by a fourth connected domain
- each of the second connected domains is surrounded by a corresponding first connected domain
- each of the third connected domains is surrounded by a corresponding second connected domain.
- the processor may obtain a second connected domain surrounded by each first connected domain, and a second connected domain surrounded by each first connected domain, and may also acquire a third connected domain surrounded by each second connected domain. And the number of third connected domains surrounded by each of the second connected domains.
- the processor can determine whether the marker in the target image contains an inclusion pattern of the pre-stored marker based on the enclosing relationship of the connected domain in the target image and the characteristics of the pre-stored marker.
- the processor may distinguish each of the markers included in the target image according to the enclosing relationship of the connected domain, wherein each of the first connected domains may correspond to one marker, or each first connected domain and the second connectivity thereof
- the domain and the third connected domain constitute a marker in the target image.
- the memory and the identity information of the marker may be pre-stored in the memory, and the processor compares the feature of the marker in the target image with the feature of the pre-stored marker according to the feature of the pre-stored marker, thereby determining the target image.
- the identity information of the marker may be stored in advance in the memory.
- the feature of the pre-stored tag may include corresponding connected domain information in the tag, and the connected domain includes a first connected domain, a second connected domain, and a third connected domain, respectively, wherein the connected domain information may include connectivity
- the connected domain information may include connectivity
- the processor may combine the number of the markers in the target image to obtain the pre-stored markers with the same number combination, and the number combination is the same.
- the identity information of the pre-stored marker is the identity information of the marker in the target image, wherein the combination of the number of markers may refer to the combination of the number of feature points of each of the sub-markers included in the marker.
- a pre-stored corresponding first connected domain may be determined according to a feature of the pre-stored marker, wherein the first connected domains corresponding to each other are surrounded by the same number of The number of the third connected domains surrounded by the two connected domains and surrounded by the second connected domains is in one-to-one correspondence.
- the first connected domain corresponding to the second background 1820 of the marker 210 in the target image includes eight second connected domains, wherein the five second connected domains are not included.
- the processor may search for four sub-markers in the features of the pre-stored markers, and the feature points of the four sub-marks are 1 white point, 3 white points, 2 white points, and 5 black points respectively.
- the marker, the identity information of the found marker is the identity information of the marker shown in FIG. 18.
- the feature of the pre-stored tag includes connectivity domain information
- the connectivity domain information may include a enclosing relationship between the connectivity domains
- the enclosing relationship between the connectivity domains may be represented by coding, where each The second connected domain corresponds to one code, and the third connected domain surrounded by the second connected domain is different in number, and the corresponding codes are different.
- the processor obtains an enclosing relationship between the multiple connected domains in the target image, and sets different corresponding codes for the second connected domain that surrounds the different number of third connected domains in the target image, where the number of the third connected domains and the coded
- the correspondence relationship may be the same as the correspondence between the number of the third connected domains of the pre-stored marker and the encoding.
- the second connected domain enclosing a third connected domain is coded as B1, and the second connected domain surrounded by the two third connected domains is coded as B2, surrounded by two third connected domains.
- the second connected domain is coded as B3, and so on.
- the second connected domain enclosing a third connected domain is coded as B1
- the second connected domain surrounded by the two connected domains is coded as B2, surrounded by The second connected domain of the two third connected domains is coded as B3, and so on.
- the fourth connected domain when encoding the connected domain, such as the code corresponding to the plurality of pre-stored tags, the fourth connected domain may be represented by the first code, and the first connected domain is represented by the second code.
- the processor acquires the inclusion relationship between the plurality of connected domains in the target image, the fourth connected domain may be represented by the first code, and the first connected domain may be represented by the second code.
- the identity information of the first connected domain is determined by the coding of each second connected domain in the first connected domain, so that the identity information of the tag corresponding to the first connected domain is determined.
- the processor can search for the same tag in the pre-stored tag according to the encoding of the tag in the target image, thereby determining the identity information of the tag in the target image.
- each of the first connected domains in the target image encloses one or more second connected domains, and each of the first connected domains corresponds to one marker, and the code corresponding to the marker in the target image may be the marker.
- the code corresponding to the pre-stored tag may be the code of each second connected domain in the pre-stored tag.
- the processor may acquire the same encoding as the encoding of the marker in the target image in the encoding of the pre-stored marker, and the identity information corresponding to the pre-stored marker having the same encoding is the identity information of the marker in the target image.
- the encoding order of each second connected domain in the marker is not limited. For example, for the marker encoding B0B1B2B3, the same encoding as B0B1B2B3 is obtained in the pre-stored marker, where the codes B0 and B1 of each second connected domain are obtained.
- the order of B2 and B3 is not limited.
- the acquired code is B1B2B0B3, which is also considered to be the same as B0B1B2B3.
- the difference between the markers may also be that the number of the sub-markers included in the plurality of markers is different. For example, among the plurality of preset pre-stored markers, only one pre-stored marker corresponds to The first number of sub-markers. In the target image, if a marker includes a first number of sub-markers, the marker corresponds to a pre-stored marker having a second number of sub-markers.
- the target image when only one first connected domain of the pre-stored mark encloses the first number of second connected domains, and a certain first connected domain in the target image surrounds the first number of the second connected domains, the target The marker corresponding to the first connected domain in the image corresponds to a pre-stored marker enclosing the first number of second connected domains.
- each black dot that does not include a white feature point may be used as one feature point, and all black dots that do not include white feature points are used as one sub-marker. That is to say, each second connected domain that does not surround the third connected domain can be regarded as a feature point, and all the second connected domains that do not surround the third connected domain are regarded as one sub-marker, and in the identification process, The number of statistics of each second connected domain that is surrounded by the third connected domain is 1, and the number of statistics of all the second connected domains that do not surround the third connected domain is 1.
- the marker in the target image is not necessarily a complete marker. If only a part of the marker is acquired, and the marker is different from other markers, it has no other markers.
- the number of feature points of at least one sub-marker in which one pre-stored marker exists is different from the number of feature points in the sub-marker in other markers, that is, a plurality of pre-stored markers Among them, only one first connected domain is surrounded by a specific second connected domain, and the specific second connected domain is surrounded by a second number of third connected domains.
- the target image when there is a first connected domain, the enclosed second connected domain is surrounded by a second number of third connected domains, and the first connected domain in the target image corresponds to the specific second Corresponding to the pre-stored markers corresponding to the connected domain.
- the pre-stored markers there is a first connected domain surrounded by a third number of second connected domains that do not surround the third connected domain.
- the marker corresponding to the first connected domain in the target image and the third number correspond to each other.
- the mark corresponding to the sub-marker corresponds to the pre-stored mark.
- the first connected domain includes a fourth surrounded by And determining, by the number of connected domains, a marker corresponding to the first connected domain in the target image and a pre-stored marker corresponding to the fourth number of connected domains.
- the processor determines a pre-stored tag corresponding to the tag in the target image, and obtains identity information of the corresponding pre-stored tag, and uses the identity information as identity information of the tag in the target image.
- the identity information of the pre-stored marker may include various information of the marker, such as physical coordinates of respective feature points in the marker, information of the device body set by the marker, and the like.
- the identity information of the corresponding first connected domain in the enclosing relationship of the pre-stored tag is used as the identity information, and the identity information of the tag corresponding to the first connected domain is obtained, so that The physical coordinates of the feature points in the respective markers in the target image, the information required by the corresponding interactive device, and the like.
- Step S124 The processor determines, according to the marker information of the target image and the identity information of the marker, a tracking method used by the interaction device corresponding to the marker.
- the processor may determine, according to the identity information of the marker, whether the markers in the target image are coplanar or non-coplanar, and when the markers are coplanar, a corresponding planar localization tracking method may be used; When not coplanar, the corresponding stereo positioning tracking method can be adopted.
- the identity information of the tag various required information for identifying and tracking the interactive device is included. Such as the physical coordinates of the marker; which interactive device is used to set the marker, whether the markers are coplanar, whether the feature points of the same marker are coplanar, and the like. In addition, whether the markers are coplanar may be judged based on the same interactive device.
- a planar positioning tracking method may be employed.
- a stereo tracking method may be employed.
- whether the coplanarity between the individual markers can be calculated by the physical coordinates of the respective markers or based on the coplanar information between the corresponding pre-stored markers.
- Step S126 the processor acquires position and posture information between the interaction device and the image acquisition device according to the corresponding tracking method.
- a planar positioning tracking method may be employed, wherein the marker coplanarity may refer to all feature points in the target image being coplanar, that is, all the feature points are located on the same plane.
- the target image that is coplanar with the feature points may be an image that includes the marker surface of the planar marker object in the above embodiment; when the interaction device in the acquired image includes the multi-faceted marker structure, the feature points are coplanar
- the target image may also be an image containing only one of the marking faces of the multi-faceted marking structure.
- the target image is an image with an interaction device collected by the image acquisition device, and the target image includes information of a plurality of feature points.
- the feature points in the target image may be all feature points in the interaction device, or may be part of feature points in all the feature points in the interaction device.
- the processor may arbitrarily select a specific number of feature points from all the feature points in the target image as the target feature points for determining the image capturing device (equivalent to the head mounted display device) and the planar marker object having the target feature point. Or the actual position and attitude information between the image acquisition device (equivalent to the head-mounted display device) and the multi-faceted marker structure having the target feature points.
- the processor after the processor acquires the target image, it may be determined whether there is a marker including the target feature point in the target image. Since each feature point is distributed within the marker, it is possible to determine whether or not a feature point exists in the acquired target image by detecting whether or not a marker exists in the target image.
- the processor may determine whether a marker exists in the target image by matching an image of the marker in the target image with an image of all markers on the pre-stored interaction device, and when matching can be similar Or the same marker, it can be determined that there is a marker in the target image. When it is not possible to match similar or identical markers, it can be determined that there is no marker in the target image, and the processor can reacquire the acquired target image until it is determined that the target image exists.
- the processor can determine the marker in the target image by searching for an area in the target image that matches the contour of the marker. Taking the marker as a rectangle as an example, all the regions in the target image with a rectangular shape are searched as the to-be-confirmed markers, and each of the to-be-confirmed markers is matched with the image of all the markers on the pre-stored interaction device, and can be matched. When a similar or identical marker is reached, it can be determined that there is a marker in the target image, and when it is not possible to match a similar or identical marker, it is determined that the marker is not present in the target image.
- the processor may determine whether the number of the target feature points is greater than or equal to a preset value, wherein the target feature point may be any feature point in the target image, because the subsequent step is to be based on the target
- the pixel coordinates and physical coordinates of the feature points acquire the six-degree-of-freedom information of the image acquisition device in the physical coordinate system.
- the number of the target feature points is greater than or equal to the preset value, wherein the preset value is a value set by the user.
- the preset value may be 4.
- the target feature points greater than or equal to the preset value may be distributed within one marker or may be distributed within the plurality of markers as long as the number of feature points in the target image is greater than or equal to a preset value. Just fine.
- the processor acquires the position and posture information between the interaction device and the image acquisition device by using the plane positioning and tracking method, and may include steps S261 to S263.
- Step S261 the processor acquires pixel coordinates of the target feature point in the target image in the image coordinate system corresponding to the target image.
- the pixel coordinates of the target feature points in the target image refer to the positions of the feature points in the target image, and the pixel coordinates of each target feature point in the target image can be directly obtained in the image correspondingly captured by the image capturing device. For example, as shown in FIG.
- I1 is the target image
- the image coordinate system is uov
- the direction of u may be the row direction of the pixel matrix in the target image
- the direction of v It may be the column direction of the pixel matrix in the target image
- the position of the origin o in the image coordinate system may select a corner point of the target image, such as the top left corner or the bottom left corner, whereby each feature point is
- the pixel coordinates within the image coordinate system can be determined.
- the pixel coordinates of the feature point 221a in Fig. 21a are (u a , v a ).
- the processor needs to perform dedistortion processing on the target image, wherein the image distortion refers to an image generated during the imaging process.
- the geometric position of the pixel relative to the reference system (the actual position or topographic map of the ground) is deformed by extrusion, stretching, offset and distortion, which changes the geometric position, size, shape and orientation of the image. Common distortions include radial distortion, eccentric distortion, and thin prism distortion.
- the target image is dedistorted according to the distortion parameters and the distortion model of the image acquisition device.
- the processor performs distortion processing on the target image to remove distortion points in the target image, and then uses the target image after the distortion processing as the target image acquired this time, and acquires image coordinates corresponding to the target image of each target feature point.
- the pixel coordinates within the system are based on the following criteria:
- Step S263 The processor acquires position and posture information between the image capturing device and the interaction device according to the pixel coordinates of the target feature point in the target image and the physical coordinates corresponding to the target feature point acquired in advance.
- the physical coordinate is the coordinate of the target feature point acquired in advance in the physical coordinate system corresponding to the interaction device, and the physical coordinate of the target feature point is the real position of the target feature point on the corresponding interaction device.
- the physical coordinates of each feature point may be acquired in advance.
- a plurality of feature points and a plurality of markers are disposed on a label surface of the interaction device, and a certain point on the label surface is selected as an origin to establish a physical coordinate system.
- the marked surface is taken as the XOY plane of the physical coordinate system, and the origin of the XOY coordinate system is located in the marked surface.
- the first marking plate as a rectangular plate as an example, one corner point of the marking surface of the marking plate is used as the origin O, the length direction of the marking surface is the X axis, and the width direction of the marking surface is the Y axis.
- the direction perpendicular to the marking surface is the Z axis, and a physical coordinate system is established.
- the distance between each feature point and the X axis and the Y axis can be obtained, thereby being able to determine the physical coordinates of each feature point in the physical coordinate system.
- the physical coordinates of the feature point 221a in Fig. 21b are (X a , Y a , Z a ). Wherein, Z a is equal to 0.
- the processor After the processor acquires the pixel coordinates and the physical coordinates of all the target feature points in the target image, the position between the image capturing device and the marker can be obtained according to the pixel coordinates and the physical coordinates of all the target feature points in each marker. And posture information.
- the processor may acquire mapping parameters between the image coordinate system and the physical coordinate system according to pixel coordinates of each target feature point, physical coordinates, and internal parameters of the image acquisition device acquired in advance.
- (c x , c y ) is the center point of the image
- (f x , f y ) is the focal length in pixels, which can be calibrated by the image acquisition device Get, is a known amount.
- the processor may obtain a rotation parameter and a translation parameter between the camera coordinate system and the physical coordinate system of the image acquisition device according to the mapping parameter.
- the rotation parameter between the camera coordinate system and the physical coordinate system may be acquired according to the SVD algorithm. And pan parameters.
- R and T can be solved, where R is the rotation parameter between the camera coordinate system and the physical coordinate system of the image acquisition device, and T is the translation parameter between the camera coordinate system and the physical coordinate system of the image acquisition device. .
- the rotation parameter and the translation parameter can be used as position and orientation information between the image acquisition device and the marker plate.
- the rotation parameter represents a rotation state between the camera coordinate system and the physical coordinate system, that is, the degree of freedom of rotation of the image acquisition device in the physical coordinate system and the coordinate axes of the physical coordinate system.
- the translation parameter represents a movement state between the camera coordinate system and the physical coordinate system, that is, the degree of freedom of movement of the image acquisition device in the physical coordinate system and the coordinate axes of the physical coordinate system.
- the rotation parameter and the translation parameter are the six free information of the image acquisition device in the physical coordinate system, which can represent the rotation and movement state of the image acquisition device in the physical coordinate system, that is, the visual field and the physical coordinate system of the image acquisition device can be obtained. The angle and distance between the coordinate axes inside.
- the method may further include acquiring physical coordinates of the target feature point.
- the processor acquires the physical coordinates of the target feature point, including steps S631 to S635.
- Step S631 The processor determines a model feature point corresponding to each feature point in the preset marker model.
- the processor may determine a correspondence between the target feature point and the model feature point in the preset marker model, wherein the preset marker model is a pre-stored standard image containing the marker information, and the marker information may include the marker The physical coordinates of each feature point.
- the processor can obtain the physical coordinates of the corresponding target feature points according to the physical coordinates of the model feature points in the preset marker model by determining the correspondence between the target feature points and the model feature points in the preset marker model.
- the processor may acquire a mapping parameter between the image coordinate system corresponding to the target image and the preset marker model, and determine, according to the mapping parameter, the correspondence between the target feature point and the model feature point in the preset marker model. relationship.
- the processor may first acquire pixel coordinates of feature points in the target image, and obtain a centroid of each sub-marker in the target image according to pixel coordinates of each feature point in the target image.
- each sub-marker includes one or more feature points, and a plurality of feature points of one sub-marker corresponding to one centroid, that is, the centroid of the sub-marker.
- the processor may calculate the coordinates of the centroid corresponding to each sub-marker according to the pixel coordinates of the feature points included in each sub-marker in the target image.
- the specific calculation method of the centroid is not limited in the embodiment of the present application, and may be calculated according to the weight calculation method.
- the processor may determine whether the centroid of the sub-marker in the target image satisfies a first preset condition, wherein the first preset condition may be determined according to actual needs.
- the first preset condition may be that the number of sub-markers or centroids in the target image reaches a preset number. Since at least 4 corresponding points are needed in calculating the mapping parameters, the preset number can be 4.
- the processor may re-acquire the target image.
- the processor may expand a preset number of new centroids in the sub-marker according to the feature points in the sub-marker in the target image, thereby Expand the number of centroids in the marker to get more accurate mapping parameters.
- the processor may establish a coordinate system by using a centroid of the sub-marker in the target image as a coordinate origin, and the sub-marker may be any one of the selected sub-markers for performing centroid expansion.
- the feature point satisfying the third preset condition is displaced to a position centered on the coordinate origin, and a new centroid is obtained according to each feature point corresponding to the post-displacement sub-marker, wherein the third The preset condition may include any one of the established coordinate system with the abscissa being less than zero, the abscissa being greater than zero, the ordinate being less than zero, and the ordinate being greater than zero, and different third preset conditions may correspondingly acquire a new centroid.
- the processor selects a centroid in the target image to establish a coordinate system as the coordinate origin.
- a coordinate system as shown in (a) of FIG. 23, the feature points a, b, c, and d in the target image are feature points included in the same sub-marker, and the feature points a, b, c, and d constitute one.
- the origin o of the coordinate system is the centroid o of the feature points a, b, c, d.
- the feature points a, b whose abscissa is less than zero in the coordinate system are displaced to the symmetrical position with the coordinate origin as the center of symmetry, that is, the horizontal and vertical of the feature points a, b
- the coordinates are multiplied by the position after -1, and the result is as shown in (b) of Fig. 23.
- each feature point corresponding to the centroid o corresponds to a new centroid, that is, a centroid o' is calculated together with the positions of a, b, and c, d after displacement, and the centroid o' is a new centroid.
- a new centroid can also be obtained with the abscissa being greater than zero as the third preset condition. That is, the feature points c and d in which the abscissa is greater than zero in the coordinate system are displaced to a position centered on the coordinate origin, that is, the horizontal and vertical coordinates of the feature points c and d are multiplied by the position of -1 to obtain the position. The result is shown in (c) of FIG. After the displacement, each feature point corresponding to the centroid o corresponds to a new centroid o", that is, a centroid o" is calculated together with the positions of a, b, and c, d after the displacement, and the centroid o" is a new one. Centroid. It can be understood that each displacement is used to calculate a new centroid and does not change the position of each feature point in the target image.
- the third sub-marker may be third, the abscissa is less than zero, the abscissa is greater than zero, the ordinate is less than zero, and the ordinate is greater than zero.
- Preset conditions under different third preset conditions, can respectively obtain a new centroid, for each sub-marker, can be extended to obtain 4 new centroids.
- 4*N new centroids can be obtained.
- the established coordinate system is not limited to the two-dimensional coordinate system shown in FIG. 23, and may also include a three-dimensional coordinate system or other coordinate system of more dimensions, or a coordinate system including more quadrants. If the established coordinate system is a multi-dimensional coordinate system, when the symmetry point of the feature point with the coordinate origin as the symmetry center is obtained, the coordinate value of the feature point corresponding to each coordinate is multiplied by -1 to obtain the symmetry point about the coordinate origin. As an implementation manner, a preset number of new centroids may be expanded according to requirements, and the preset number may not be limited.
- the processor may acquire mapping parameters between the image coordinate system corresponding to the target image and the preset marker model based on the pixel coordinates of the respective centroids, the physical coordinates, and the internal parameters of the image acquisition device acquired in advance.
- the processor calculates a mapping parameter between the image coordinate system and the preset marker model according to each centroid in the image, and the mapping parameter may be a parameter in which the points in the image coordinate system are mapped to the coordinate system in which the preset marker model is located, such as a plane.
- the homography matrix includes the original centroid before expansion and the new centroid obtained by the expansion.
- the physical coordinates of the centroid are pre-acquired coordinates of the centroid in the physical coordinate system corresponding to the marker, and the coordinate origin of the physical coordinate system may be set on the plane marker object or the multi-face marker structure where the marker is located.
- the preset marker model includes physical coordinates of each feature point in the marker, and the physical coordinates of the centroid of each of the child markers can be calculated by preset physical coordinates of each model feature point in the marker model.
- the processor may expand the new centroid in a preset marker model in a corresponding manner of the target image expansion centroid, and the expanded new centroid in the preset marker model corresponds to the expanded new centroid in the target image.
- the processor may pre-acquire a one-to-one correspondence between the sub-markers in the preset marker model and the sub-markers in the target image.
- a sub-marker corresponding to the sub-marker in the target image is included.
- the processor pre-acquires the corresponding relationship between the sub-marker in the preset marker model and the sub-marker in the target image, and the specific acquisition manner is not limited in the embodiment of the present application, for example, the corresponding feature of each sub-marker in the marker
- the shape of the dot is different, and the correspondence between the sub-marker in the preset marker model and the sub-marker in the target image is determined according to the shape; for example, the number of feature points included in each sub-marker in the marker is different, according to the number of feature points
- the correspondence between the sub-marker in the preset marker model and the sub-marker in the target image is determined.
- the centroid expansion of the preset marker model is the same as the centroid extension in the target image. That is to say, in the preset marker model, the coordinate system is established with the centroid corresponding to the centroid of the centroid expansion in the target image as the coordinate origin.
- the centroids corresponding to each other in the target image and the preset marker model are respectively the centroids of the sub-markers corresponding to the target image and the preset marker model.
- the model feature points corresponding to the centroid of the coordinate marker origin in the preset marker model the model feature points satisfying the third preset condition are displaced to the position where the coordinate origin is the symmetry center, and the respective model features corresponding to the centroid according to the displacement Click to get a new centroid.
- the third preset condition is the same as the third preset condition for performing centroid expansion in the target image, and the obtained new centroid corresponds to the extended new centroid in the target image.
- FIG. 24(a) is a sub-marker corresponding to the sub-marker shown in FIG. 23(a) in the preset marker model, wherein A, B, C, and D are the sub-markers.
- the coordinate system is established with the centroids m of A, B, C, and D as the coordinate origins.
- the abscissa is smaller than zero as the third preset condition
- the model feature points A and B whose abscissa is smaller than zero in the coordinate system are displaced to the position where the coordinate origin m is the center of symmetry, that is, the model feature is
- the horizontal and vertical coordinates of points A and B are multiplied by the position after -1, and the result is as shown in (b) of FIG.
- a centroid m' is calculated together with the positions of the displaced A, B, C, and D, and the centroid m' is the preset mark.
- centroid m is calculated together with the positions of the shifted C, D, and A, B, and the centroid m" is the preset marker A new centroid obtained in the model, the new centroid m" corresponds to the new centroid o obtained by the target image.
- the processor can obtain a new centroid in the preset marker model that respectively corresponds to the new centroid of the target image.
- the processor may calculate physical coordinates of each centroid in the preset marker model according to physical coordinates of each model feature point in the preset marker model.
- the physical coordinates of each model feature point of the preset marker model are stored in advance, and the physical coordinates of each centroid can be calculated according to the physical coordinates of each model feature point.
- the calculated centroid includes the original centroid before expansion and the new centroid after expansion.
- the centroid calculation method is not limited in the embodiment of the present application, and is calculated by using a weight calculation method.
- the processor may use the physical coordinates of the centroid in the preset marker model as the physical coordinates of the corresponding centroid in the target image according to the correspondence between the centroid in the target image and the centroid in the preset marker, thereby obtaining the physics of each centroid in the target image. coordinate.
- the physical coordinates of the centroid m in Fig. 24 are taken as the physical coordinates of the centroid o in Fig. 23 corresponding thereto.
- the processor may calculate a mapping parameter between the image coordinate system corresponding to the target image and the preset marker model according to the pixel coordinates of each centroid in the target image, the physical coordinates, and the internal parameters of the image acquisition device acquired in advance.
- the relationship between the image coordinates and the physical coordinate system may be as shown in the above formula (1), and after converting the formula (1) in the above embodiment into the formula (2), The obtained pixel coordinates and physical coordinates of the plurality of centroids, and the internal parameters of the image capturing device are brought into the equation (2) in the above embodiment, and H is calculated, that is, the mapping parameter between the image coordinate system and the physical coordinate system.
- the coordinate system of the preset marker model corresponds to the physical coordinate system corresponding to the marker, and each feature point is The coordinates in the coordinate system of the preset marker model are the same as the physical coordinates. Therefore, the image coordinate system corresponding to the target image can be obtained according to the pixel coordinates of each centroid, the physical coordinates, and the internal parameters of the image acquisition device acquired in advance. Preset mapping parameters between marker models.
- the processor may map each feature point in the target image to the coordinate system of the preset marker model according to the mapping parameter, thereby obtaining the target image.
- Corresponding relationship between each feature point and each model feature point in the preset marker model, that is, a corresponding model feature point of each feature point in the target marker image in the preset marker model can be obtained.
- the processor may determine whether the second preset condition is met.
- each feature point and the preset marker model in the target image may be acquired according to the mapping parameter.
- the centroid expansion of the target image may be continued, more centroids are acquired, and more accurate mapping parameters are calculated again by using more acquired centroids.
- the number of new centroids acquired each time is not limited in the embodiment of the present application.
- the second preset condition may be that a matching error between the feature point in the target image and the model feature point in the preset marker model satisfies a preset accuracy requirement.
- the processor may map each feature point in the target image to a coordinate system of the preset marker model according to the mapping parameter to obtain coordinates of each feature point in the target image in a coordinate system of the preset marker model.
- the matching error between the feature point of the target image and the model feature point of the preset marker model is less than the preset error threshold, it is determined that the second preset condition is satisfied.
- the method for determining by the processor may be: calculating a distance between each feature point of the target image and a model feature point of the preset marker model in a coordinate system of the preset marker model, and the feature points of the target image are The minimum distance corresponding to the feature points of the model is the matching error of the feature points in the target image.
- the processor may determine that the second preset condition is met when the matching error between each feature point and the model feature point in the target image is less than a preset error threshold; or when there is a preset number of feature points in the target image.
- the matching error is less than the preset error threshold, and the processor may determine that the second preset condition is met, wherein the preset number is not limited.
- the second preset condition may be that the matching error between the feature point in the target image and the model feature point of the preset marker model is no longer reduced.
- the processor may acquire mapping parameters according to the centroid calculation of the multiple extensions, map each feature point in the target image to the coordinate system of the preset marker model according to the mapping parameters acquired multiple times, and acquire the target image in each mapping. Matching error between feature points and model feature points.
- the processor may determine that the second preset condition is satisfied.
- the second preset condition may be that the number of times of expanding the new centroid in the target image reaches a preset number of times, and each time a new centroid is expanded in the target image, it may be extended once.
- the processor may determine that the second preset condition is satisfied.
- the second preset condition may be that the number of new centroids expanded in the target image reaches a preset number.
- the processor may determine that the second preset condition is met, and the specific value of the preset number is not limited in the embodiment of the present application.
- the second pre-conditions are not limited in the embodiment of the present application, and may be combined in the foregoing various embodiments, as in the above various embodiments, as the second preset condition.
- Step S633 The processor searches for the physical coordinates of each model feature point in the physical coordinate system corresponding to the interaction device in the preset marker model.
- Step S635 The processor uses the physical coordinate of the model feature point corresponding to each target feature point as the physical coordinate of the target feature point in the physical coordinate system corresponding to the interaction device.
- the processor may map each feature point in the target image to a coordinate system of the preset marker model according to the mapping parameter, to acquire coordinates of each feature point in the target image in a coordinate system of the preset marker model.
- the model feature point of the preset marker model closest to the coordinate distance of each feature point in the target image in the coordinate system of the preset marker model may be used as the feature point in the target image.
- the corresponding model feature points in the preset marker model may be used as the feature point in the target image.
- FIG. 25 is taken as an example. As shown in FIG. 25, FIG. 25a includes various feature points e, f, and g in the image coordinate system, and the processor can calculate each feature point in the target image according to the mappable parameter H. Set the coordinates in the coordinate system of the marker model, map the feature points e, f, g into the coordinate system of the preset marker model, and obtain the mapped target feature points e', f', g', as shown in Figure 25b. Shown. In Fig. 25b, E, F, and G are feature points in the marker corresponding to the sub-markers formed by e, f, and g in the preset marker model.
- the processor can separately calculate the distances of e' to E, F, G three model feature points, and the distance from e' to E is the smallest, then the feature points e' in the target image can be obtained in the preset marker model corresponding model feature points.
- E respectively calculate the distance from f' to E, F, G three model feature points, the distance from f' to F is the smallest, then the feature point f' in the target image can be obtained in the preset marker model corresponding model feature point F
- Calculating the distances of g' to E, F, and G model points respectively, and the distance from g' to G is the smallest, then the model feature point G corresponding to the feature point g' in the target image in the preset marker model can be obtained. .
- the processor determines a correspondence between each feature point in the target image and a model feature point in the preset marker model, and can search for physical coordinates of each model feature point in the physical coordinate system corresponding to the interaction device in the preset marker model. Obtaining physical coordinates of the corresponding feature points in the target image according to physical coordinates of the model feature points in the preset marker model. In one embodiment, the physical coordinates of the model feature points may be used as corresponding feature points in the target image. Physical coordinates.
- 26 is a flow chart of tracking and positioning an interactive device by a stereo tracking method in one embodiment.
- the processor acquires position and posture information between the interaction device and the image acquisition device by the stereo tracking method, and may include steps S2610 to S2620.
- Step S2610 The processor acquires pixel coordinates of the target feature point in the target image in an image coordinate system corresponding to the target image.
- the processor may acquire a target image with an interaction device collected by the image acquisition device, and the target image includes target feature points corresponding to at least two faces in the corresponding interaction device.
- the feature points within the target image are distributed in at least two planes, that is, the image acquisition device collects the interaction means of the markers on at least two planes.
- the target image may be an image of the feature points of the at least two faces of the multi-faceted mark structure acquired by the image capture device.
- I2 is a target image
- the image coordinate system is uov
- the direction of u may be the row direction of the pixel matrix in the target image
- the direction of v may be the column direction of the pixel matrix in the target image
- the position of the origin o in the image coordinate system can select a corner point of the target image, for example, the top left corner or the bottom left corner, whereby the pixel coordinates of each feature point in the image coordinate system can be determined.
- the pixel coordinates of the feature point 341a in Fig. 27 are (u a , v a ).
- Step S2620 The processor acquires position and posture information between the image capturing device and the interaction device according to the pixel coordinates of the target feature point in the target image and the physical coordinates corresponding to the target feature point acquired in advance.
- the physical coordinates of each target feature point may be acquired in advance, and multiple target feature points and a plurality of markers are set on different marking surfaces of the interaction device, and a certain point on one of the marking surfaces may be selected as an origin to establish a physical coordinate system. .
- a corner point of a rectangular sub-surface of the interaction device is used as the origin O, and the physical coordinate system XYZ is established, and each feature point is used.
- the distances to the X-axis, the Y-axis, and the Z-axis can be measured, whereby the physical coordinates of each feature point in the XOY coordinate system can be determined.
- the physical coordinates of the feature point 341a in Fig. 28 are (X). a , Y a , Z a ).
- the processor may acquire the physical coordinates of each target feature point in the physical coordinate system corresponding to the interaction device.
- the manner of obtaining the physical coordinates refer to the descriptions of steps S631 to S635 in the above embodiment. Let me repeat.
- the position between the image capturing device and the interactive device may be obtained according to the pixel coordinates and the physical coordinates of all the target feature points in each of the markers.
- the processor may first acquire mapping parameters between the image coordinate system and the physical coordinate system according to pixel coordinates of each target feature point, physical coordinates, and internal parameters of the image acquisition device acquired in advance.
- the relationship between the image coordinate system and the physical coordinate system may be as shown in the above formula (1).
- the equation (1) in the above embodiment can be converted into the equation (2) in the above embodiment, and the obtained pixel coordinates and physical coordinates of the plurality of target feature points, and the internal parameters of the image acquisition device are brought into the above.
- the equation (2) in the embodiment it is possible to acquire H, that is, a mapping parameter between the image coordinate system and the physical coordinate system. Then, the rotation parameter and the translation parameter between the camera coordinate system and the physical coordinate system of the image acquisition device are acquired according to the mapping parameters.
- the homography matrix H can be singularly decomposed according to the SVD algorithm, and the equation (3) in the above embodiment is obtained, and the equation (3) in the above embodiment is converted into the equation (4). And obtaining the equation (5) by the decomposition algorithm, and obtaining the rotation matrix R and the translation matrix T, wherein R is a rotation parameter between the camera coordinate system of the image acquisition device and the physical coordinate system, and T is an image acquisition device A translation parameter between the camera coordinate system and the physical coordinate system.
- the processor can use the rotation parameter and the translation parameter as the position and posture information between the image acquisition device and the interaction device.
- the rotation parameter and the translation parameter as the position and posture information between the image acquisition device and the interaction device.
- Step S130 The processor determines a virtual scene corresponding to the interaction device according to the position and posture information.
- the processor can determine the display content corresponding to the interaction device according to the position and posture information of the interaction device, and display the display content in the real scene through the display device and the optical component of the head display device, so that the user wears the wearing head The display device observes the virtual scene.
- an embodiment of the present application further provides an electronic device, including a memory and a processor, where the computer program is stored in the memory, and the computer program is executable by the processor to implement the method described in the foregoing embodiments. .
- the embodiment of the present application further provides a computer readable storage medium, where the computer readable storage medium stores a computer program executable by the processor to implement the method described in the foregoing embodiments.
- the computer readable storage medium may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), an EPROM, a hard disk, or a ROM.
- the computer readable storage medium comprises a non-transitory computer-readable storage medium.
- the computer readable storage medium has a storage space for a computer program that performs any of the method steps described above. These computer programs can be read from or written to one or more computer program products.
- the computer program can be compressed, for example, in a suitable form.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
一种图像处理方法,包括:获取图像采集装置采集的目标图像,目标图像包含设置在交互装置上的标记物,交互装置位于真实场景中;根据目标图像确定交互装置在真实场景内的位置及姿态信息;以及根据位置和姿态信息确定与交互装置对应的虚拟场景。
Description
相关申请的交叉引用
本申请要求于2018年02月06日提交中国专利局的申请号为CN201810119868.1名称为“图像处理方法、装置及识别跟踪系统”、于2018年02月06日提交中国专利局的申请号为CN201810119839.5名称为“图像处理方法、装置及计算机可读介质”、于2018年02月06日提交中国专利局的申请号为CN201810119854.X名称为“标记物的识别方法、装置及识别跟踪系统”、于2018年02月06日提交中国专利局的申请号为CN201810119776.3名称为“定位方法、装置、识别跟踪系统及计算机可读介质”、于2018年02月06日提交中国专利局的申请号为CN201810118639.8名称为“定位方法、装置、识别跟踪系统及计算机可读介质”、于2018年02月06日提交中国专利局的申请号为CN201810119387.0名称为“图像处理方法及装置”及于2018年02月06日提交中国专利局的申请号为CN201810119323.0名称为“虚拟场景处理方法、装置、交互系统、头戴显示装置、视觉交互装置及计算机可读介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及交互技术领域,具体涉及跟踪交互装置的方法、存储介质以及电子设备。
近年来,随着科技的进步,增强现实(AR,Augmented Reality)和虚拟现实(VR,Virtual Reality)等技术已逐渐成为国内外研究的热点。以增强现实为例,增强现实是通过计算机系统提供的信息增加用户对现实世界感知的技术,其将计算机生成的虚拟物体、场景或系统提示信息叠加到真实场景中,来增强或修改对现实世界环境或表示现实世界环境的数据的感知。
在增强现实及虚拟现实等系统中,常需要对目标物体进行识别跟踪。传统的识别跟踪方法,通常是采用磁传感器、光传感器、超声波、惯性传感器、目标物体图像处理等方式实现,但是这些识别跟踪的方法,通常,识别跟踪效果并不理想,如磁传感器、光传感器、超声波等通常受到环境较大的影响,惯性传感器对精度要求极高,市场上急需一种全新的识别方法,以实现低成本、高精度的交互,而对目标物体的图像的处理作为识别跟踪的重要技术也需要一套完美有效的解决方法。
发明内容
为使本申请的上述目的、特征和优点能更明显易懂,下文特举较佳实施例,并配合所附附图,作详细说明如下。
本申请实施例提供一种图像处理方法,包括:获取图像采集装置采集的目标图像,所述目标图像包含设置在交互装置上的标记物,所述交互装置位于真实场景中;根据所述目标图像确定所述交互装置在所述真实场景内的位置及姿态信息;根据所述位置及姿态信息确定与所述交互装置对应的虚拟场景。
本申请实施例提供一种图像处理方法,包括:获取连续多帧图像中除首帧图像外的当前帧图像对应的第一阈值图像,所述第一阈值图像为对历史帧图像进行处理后得到且与当前帧图像分辨率相同的灰度图像;对当前帧图像的每一个像素点,以所述第一阈值图像中对应位置的像素点作为二值化阈值,将当前帧图像二值化。
本申请实施例提供一种图像处理方法,包括:获取包括标记物的目标图像;对所述目标图像进行处理,并获取所述目标图像中多个连通域之间的包围关系;根据所述目标图像中多个连通域之间的包围关系,以及预存储的标记物的特征,确定所述目标图像中标记物的身份信息为对应的预存储标记物的身份信息。
本申请实施例提供一种图像处理方法,包括:获取具有交互装置的目标图像,以及所述目标图像中交互装置内的特征点的像素坐标,所述交互装置包括多个子标记物,每一子标记物包括一个或多个特征点;
获取所述目标图像内每个子标记物的质心;当所述目标图像中获得的子标记物的质心满足第一预设条件,根据所述目标图像内子标记物的特征点,在所述子标记物内扩展预设个数的新质心;基于扩展后各个质心的像素坐标、物理坐标以及预先获取的所述图像采集装置的内参数,获取所述目标图像与预设标记物模型之间的映射参数;基于所述映射参数获取所述目标图像中各个特征点与所述预设标记物模型中各个特征点的对应关系。
本申请实施例提供一种图像处理方法,包括:获取具有标记物的目标图像,所述标记物分布在所述交互装置的一个面上或多个面上;确认所述目标图像中标记物的身份信息;根据目标图像的标记物信息及所述标记物的身份信息,确定对所述标记物对应的交互装置采用的跟踪方法;根据相应的跟踪方法,获取所述交互装置与所述图像采集装置之间的位置及姿态信息。
本申请实施例提供一种图像处理方法,包括:获取图像采集装置采集的具有交互装置的目标图像,所述目标图像内包括所述交互装置内的多个共面的目标特征点;获取所述目标图像内的目标特征点在所述目标图像对应的图像坐标系内的像素坐标;根据所述目标特征点的像素坐标和预先获取的所述目标特征点对应的物理坐标,获取所述图像采集装置与所述交互装置之间的位置及姿态信息,其中,所述物理坐标为预先获取的目标特征点在所述交互装置对应的物理坐标系内的坐标。
本申请实施例提供一种图像处理方法,包括:获取图像采集装置采集的具有交互装置的目标图像,所述目标图像内包括所述交互装置内至少分布在两个面上的目标特征点;获取所述目标图像内的目标特征点在所述目标图像对应的图像坐标系内的像素坐标;根据所述目标特征点的像素坐标和预先获取的所述目标特征点的物理坐标,获取所述图像采集装置与所述交互装置之间的位置及姿态信息,其中,所述物理坐标为预先获取的所述目标特征点在所述交互装置对应的物理坐标系内的坐标。
本申请实施例提供一种计算机可读存储介质,存储有一个或多个计算机程序,所述一个或多个计算机程序被一个或多个处理器执行时,用于执行以下步骤:获取图像采集装置采集的目标图像,所述目标图像包含设置在交互装置上的标记物,所述交互装置位于真实场景中;根据所述目标图像确定所述交互装置在所述真实场景内的位置及姿态信息;根据所述位置及姿态信息确定与所述交互装置对应的虚拟场景。
本申请实施例提供一种电子设备,包括一个或多个处理器和存储器,所述存储器存储有一个或多个计算机程序,所述一个或多个计算机程序被所述一个或多个处理器执行时,用于执行以下步骤:获取图像采集装置采集的目标图像,所述目标图像包含设置在交互装置上的标记物,所述交互装置位于真实场景中;根据所述目标图像确定所述交互装置在所述真实场景内的位置及姿态信息;根据所述位置及姿态信息确定与所述交互装置对应的虚拟场景。
为了更清楚地说明本申请实施例的技术方案,下面将对实施例中所需要使用的附图作简单地介绍,应当理解,以下附图仅示出了本申请的某些实施例,因此不应被看作是对范围的限定,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他相关的附图。
图1为一个实施例中识别跟踪系统的架构图;
图2a、2b为本申请实施例中标记物的示意图;
图3a为一个实施例中交互装置的结构图;
图3b为另一个实施例中交互装置的结构图;
图3c为另一个实施例中交互装置的结构图;
图3d为另一个实施例中交互装置的结构图;
图3e为另一个实施例中交互装置的结构图;
图4为一个实施例中多面标记结构体的结构图;
图5为一个实施例中图4所示的所多面标记结构体在另一视角的结构图;
图6为一个实施例中平面标记物体的结构图;
图7为另一个实施例中标记物的示意图;
图8为一个实施例中图像处理方法的流程图;
图9为一个实施例中用户所观察到的第一标记板和二十六面标记结构体之间的位置及姿态示意图;
图10为一个实施例中显示的虚拟场景的效果图;
图11为一个实施例中基于交互装置与图像采集装置之间的不同位置及姿态信息显示的不同虚拟场景的示意图;
图12为一个实施例中基于多个交互装置之间的不同位置及姿态信息显示的不同虚拟场景的示意图;
图13为另一个实施例中图像处理方法的流程图;
图14为一个实施例中获取第一阈值图像P1的流程图;
图15为一个实施例中获取第二阈值图像P2的流程图;
图16a为一个实施例中计算像素值的示意图;
图16b为另一个实施例中计算像素值的示意图;
图17为一个实施例中双线性插值的示意图;
图18为又一个实施例中标记物的示意图;
图19为一个实施例中连通域的包围关系的树状图;
图20为一个实施例中通过平面定位跟踪方法对交互装置进行跟踪定位的流程图;
图21a为一个实施例中图像坐标系的示意图;
图21b为一个实施例中物理坐标系的示意图;
图22为一个实施例中获取目标特征点的物理坐标的流程图;
图23为一个实施例中在目标图像中扩展新的质心的示意图;
图24为一个实施例中在预设标记物模型中扩展新的质心的示意图;
图25为一个实施例中将目标图像的特征点映射到预设标记物模型的坐标系,并获取对应的模型特征点的示意图;
图26为一个实施例中通过立体跟踪方法对交互装置进行跟踪定位的流程图;
图27为另一个实施例中图像坐标系的示意图;
图28为另一个实施例中物理坐标系的示意图。
下面详细描述本申请的实施例,所述实施例的示例在附图中示出,其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的,仅用于解释本申请,而不能理解为对本申请的限制。
图1示出了本申请实施例提供的识别跟踪系统10,包括头戴显示装置100和交互装置200,其中交互装置200上具有至少一个标记物。
头戴显示装置100可采集包含交互装置200的标记物的图像,并根据采集的图像对交互装置200的标记物进行识别追踪,以获取交互装置200的位置和旋转信息,从而根据交互装置200的位置和旋转信息显示虚拟内容。
头戴显示装置100包括壳体(未标识)、图像采集装置110、显示装置120、光学组件130、处理器140和照明装置150。其中,显示装置120和图像采集装置110均与处理器140 电连接。
在一些实施方式中,照明装置150和图像采集装置110均通过滤光板(未标识)装设并覆盖在壳体内,该滤光板可过滤环境光等光线,例如照明装置150发射红外光,滤光板为过滤除红外光线外的光线的元件。
图像采集装置110用于采集物体的图像并发送至处理器140。具体地,图像采集装置110采集包含有上述平面标记板或多面标记结构体中至少一个的图像,并发送至处理器140。在本实施方式中,图像采集装置110为采用红外接收方式的单目相机,不仅成本低,无需双目相机之间的外参,而且功耗低,同等带宽下帧率更高。
处理器140用于根据图像输出对应的显示内容至显示装置120,还对交互装置200进行识别跟踪的运算。处理器140可以包括任何适当类型的通用或专用微处理器、数字信号处理器或微控制器。处理器140可以被配置为经由例如网络从系统的各种组件接收数据和/或信号。处理器140还可处理数据和/或信号以确定系统中的一个或多个操作条件。例如,当处理器140应用于头戴显示装置时,处理器根据图像采集装置采集的图像对交互装置200进行识别跟踪运算,生成对应虚拟的显示内容,将显示内容发送至显示装置120以进行显示,并通过光学组件130将显示内容投射至用户。需要说明的是,处理器140并不限定于装设在头戴显示装置100内。
在一些实施方式中,头戴显示装置100还包括设置在壳体上且与与处理器140连接的视觉里程相机160,该视觉里程相机160用于采集外界真实场景的场景图像,并将场景图像发送至处理器140。在用户穿戴头戴显示装置100时,处理器140根据视觉里程相机160采集的场景图像,利用视觉里程技术获取用户的头部在真实场景中的位置及姿态关系。具体地,处理器140通过视觉里程相机160获取的图像序列,经过特征提取、特征匹配与跟踪,以及运动估计等的处理,得出头戴显示装置100具体的位置和方向的变化,进而获得头戴显示装置100与真实场景的相对位置及姿态关系,以及头戴显示装置100在真实世界的位置,实现导航定位。处理器140根据交互装置200与头戴显示装置100之间的相对位置及姿态信息,可以推算出交互装置200与真实场景之间的相对位置及姿态关系,实现更深入的交互形式与体验。
显示装置120用于显示处理器140输出的显示内容。在一些实施例中,显示装置120可以是与头戴显示装置100连接的智能终端的一部分,即智能终端的显示屏,例如手机和平板电脑的显示屏。在一些实施例中,显示装置120可以是独立的显示器(例如,LED,OLED或LCD)等,此时显示装置固定安装在壳体上。需要说明的是,当显示装置120为智能终端的显示屏时,壳体上设置有用于安装该智能终端的安装结构,使用时将智能终端通过安装结构装设在壳体上,处理器140可以是智能终端内的处理器,也可以是独立设置在壳体内的处理器,并与智能终端通过数据线或通信接口连接;当显示装置120为与智能终端等终端分离的显示装置时,显示装置120可以固定在壳体上。
光学组件130用于将显示装置120发出的光线投射至预设位置处,该预设位置处可以是用户佩戴头戴显示装置100时,用户双眼的观察位置。
照明装置150用于为图像采集装置110采集物体的图像时提供光线。具体地,照明装置150的照明角度以及照明装置150的数量,可以根据实际使用而设定,以使所发射的照明光线能够覆盖目标物体。其中,照明装置150采用红外光照明装置,能够发出红外光线,此时图像采集装置110为近红外相机,可以接收红外光线。照明装置150的数量不限,可以是一个,也可以是多个。在一些实施方式中,照明装置150临近图像采集装置110设置,例如可以是多个照明装置150设置在图像采集装置110的附近。本申请通过主动照明的方式,可以提高图像采集装置110采集的目标图像的图像质量。
交互装置200可以是平面标记物体,也可以是多面标记结构体。如图1所示,平面标记物体包括第一标记板310和第二标记板320,多面标记结构体包括六面标记结构体410和二十六面标记结构体420,也可以是其他面数的标记结构体,在此不一一列举。
平面标记物体具有一个标记面,标记物设置在该标记面上,该平面标记物体可以是第一标记板310或第二标记板320。第一标记板310上设有多个标记物,每个标记物的内容均不相同,所有的标记物均设置在第一标记板310的标记面上,第一标记板310上各个标记物的特征点均在该标记面上。第二标记板320上设有一个标记物,第二标记板320上标记物的特征点也全部在该标记面上的。在识别跟踪系统10中,第二标记板320的数量可以是多个,且每个第二标记板320的标记物的内容互不相同,多个第二标记板320可以与该识别跟踪系统10在增强现实或虚拟现实等领域中组合使用。
多面标记结构体具有多个标记面,且至少两个不共面的标记面上设有标记物。在一个实施例中,该多面标记结构体可以是六面标记结构体410,也可以是二十六面标记结构体420等。六面标记结构体410包括6个标记面,每个标记面上均设有标记物,且每个面上的标记物的图案互不相同。二十六面标记结构体420包括二十六个面,其中,二十六个面中可设有17个标记面,每个标记面上均设有标记物,且每个面上的标记物的图案互不相同。当然,上述的多面标记结构体的总面数以及标记面的描述和标记物的设置,可以根据实际使用而决定,在此不做限定。
需要说明的是,交互装置并不限定于上述平面标记物体和多面标记结构体,交互装置可以是任何具有标记物的载体,载体可以根据实际场景设置,例如玩具枪、游戏枪等模型枪,在模型枪等交互装置上设置相应的标记物,通过识别追踪模型枪上的标记物,能够获取模型枪的位置和旋转信息,用户通过握持该模型枪在虚拟场景中进行游戏操作。
在一个实施例中,交互装置200包括第一背景和按照特定规则分布于第一背景的至少一个标记物。标记物包括第二背景以及按照特定规则分布于第二背景的若干子标记物,每个子标记物具有一个或多个特征点。其中,第一背景和第二背景有一定的区分度,例如,可以是第一背景为黑色,第二背景为白色。每个标记物内的子标记物的分布规则不同,因此,每个标记物所对应的图像互不相同。子标记物可以是具有一定形状的图案,且该子标记物的颜色与标记物内的第二背景有一定的区分度,例如,第二背景为白色,而子标记物的颜色为黑色。子标记物可以是由一个或多个特征点构成,该特征点的形状不做限定,可以是圆点、圆环,也可以是三角形等其他形状。
图2a、2b为本申请实施例中标记物的示意图,标记物210中的子标记物可以采用不同的形式。如图2a所示,标记物210内包括多个子标记物212,每个子标记物212由一个或多个特征点214构成,图2a中的每个白色圆形图案为一个特征点214。标记物210的轮廓为矩形,当然,标记物的形状也可以是其他形状,在此不做限定,矩形的白色区域(即第二背景)以及该白色区域内的多个子标记物212构成标记物210。如图2b所示,标记物210’内包括多个子标记物212’,每个子标记物212’由一个或多个特征点214’构成,该特征点214’可以是黑色圆点,也可以是白色圆点。其中,一个子标记物212’中可以包含一个或多个黑色圆点214’,一个子标记物212’中也可包含一个或多个白色圆点214’。
在实际应用时,用户在佩戴头戴显示装置100并进入预设场景时,当交互装置在图像采集装置110的视野范围内时,图像采集装置110采集到包含有交互装置的目标图像;处理器140获取到该目标图像及相关信息,运算识别出该交互装置,并获取到该目标图像内的标记物与图像采集装置之间的位置与旋转关系,进而得到交互装置相对于头戴显示装置100的位置及姿态关系,使得用户观看到的虚拟场景在相应的位置及姿态角度上。用户还可以通过多个交互装置的结合以在虚拟场景内进一步产生的新的虚拟图像,提高用户的体验感。用户还可以通过交互装置实现与虚拟场景的交互。此外,该识别追踪系统100还可以通过视觉里程相机160获取头戴显示装置100与真实场景的位置与旋转关系,进而可以获取交互装置与真实场景的位置和旋转关系,以及头戴显示装置100在真实世界的位置,当虚拟场景和真实场景有一定的对应关系时,可以构建出一个与真实场景类似的虚拟场景,进一步提高用户的体验感。
请参阅图3a~图3e,交互装置包括装置主体,以及设置于装置主体表面的一个或多个 标记物。当交互装置为平面标记物体时,标记物可设置在平面标记物体的一个表面上,如图3a所示,第一标记板310包括装置主体311,以及设置于装置主体311表面的一个或多个标记物210。当交互装置为多面标记结构体时,标记物可设置在多面标记结构体的一个或多个表面,,如图3b所示,六面标记结构体410包括装置主体411,以及设置在装置主体411的一个表面上的标记物210,如图3c所示,二十六面标记结构体420包括装置主体421,以及设置在装置主体421的不同表面的标记物210。在一些实施方式中,如图3d所示,六面标记结构体410的装置主体411包括多个表面,标记物210设置在装置主体411中两个相邻表面的交界处,也即是,一个标记物设置在相邻的多个平面的表面上。在一些实施方式中,标记物也可设置在装置主体具有不同平面的同一表面上,例如设置在球形表面、弧形表面等,如图3e所示,标记物210设置在装置主体431球形表面上。需要说明的是,交互装置中装置主体及在装置主体上设置的标记物的方式并不仅限于上述描述的几种,装置主体可以是其他形状,标记物也可按照其他方式进行设置,在此不作限定。
在一个实施例中,交互装置中的一个或多个标记物,可突出设置在装置主体的,即标记物为设置在装置主体表面的一个层结构。在一个实施例中,装置主体的表面可开设有对应标记物数量的凹槽,标记物对应设置在装置主体表面的凹槽内,凹槽的深度可以等于标记物的厚度,使标记物的外表面与凹槽顶部齐平,当然,凹槽的深度在本申请实施例中并不限定。
请同时参阅图4及图5,多面标记结构体400具有标记物210以允许被外部的图像采集装置110识别并追踪。在一个实施例中,多面标记结构体400包括装置主体401以及连接于装置主体401的手柄402。在一些实施方式中,手柄402设有连接部(图中未标出),装置主体401连接于连接部。
装置主体401上设有标记物210,图像采集装置110通过采集包含标记物210的图像,处理器根据该图像获取多面标记结构体400所搭载的信息,包括多面标记结构体400的身份信息,以及相对于头戴显示装置的位置和旋转信息,实现对多面标记结构体400的识别追踪,进而根据该位置旋转信息确定头戴显示装置的虚拟内容。装置主体401的具体形态结构不受限制,,例如本实施方式中,装置主体401为二十六面体,其包括十八个正方形面以及八个三角形面。
进一步地,装置主体401包括第一表面403以及第二表面404,第二表面404与第一表面403不共面。第一表面403上设置有第一标记物220,第二表面404上设置有区别于第一标记物220的第二标记物230。图像采集装置识别第一标记物220及第二标记物230中的任一个或两个,并获取多面标记结构体400的位置及姿态信息,以对多面标记结构体400进行识别跟踪。
需要说明的是,第一表面403及第二表面404之间的位置关系不受限制,例如,第一表面403与第二表面404可以相邻设置、相间隔设置,或者第一表面402及第二表面404可以为十八个正方形面以及八个三角形面中的任意两个,并不局限于本说明书所描述。
需要说明的是,装置主体401还包括第三表面、第四表面、第五表面……第二十六表面(未标识)中的任一个或多个,相应地,这些表面上可设置有对应的标记物210,每一表面上的标记物210的信息各不相同。
图6为本申请实施例中平面标记物体的示意图,平面标记物体300包括装置主体(未标出),装置主体上具有基层302,基层302上设置一个或多个标记物210。当标记物210为多个时,该多个标记物210分散设置于该基层302上。
具体地,基层302可以采用软质材料制成,如由布、塑料等;基层302也可以由硬质材料制成,如纸板、金属材料等。在一个实施例中,基层302可以设有折叠部,以使基层302具有折叠功能,以便于折叠收纳。作为一种实施方式,平面标记物体300设有相互垂直的两个折叠部,两个折叠部可将平面标记物体300均分为四个区域,通过两个折叠部将平面标记物体300的四个区域折叠后,可以将平面标记物体300堆叠为一个区域大小。基 层302的形状并不限制,如可以为圆形、三角形、正方形、长方形、不规则多边形等。
如图7所示,标记物210包括多个相互分离的子标记物212,每个子标记物212中的各个特征点214相互分离。每个子标记物212所包括的特征点214的数量并不限定,可以根据实际识别需求及标记物210所占区域大小确定。每个特征点214的形状并不限定,可以是三角形、四边形或圆形等。
在一个实施例中,子标记物212可以是空心图形,包括一个或多个空心部分,其中,每个空心部分可作为一个特征点214,如图7中包括三个白色圆点214的黑色子标记物212a所示。
在一个实施例中,在子标记物212的任意一个空心部分,还可以设置实心图形,并以该实心图形作为该子标记物212中该空心部分对应的特征点214,如图7中的子标记物212b所示。
在一个实施例中,在子标记物212的空心部分内设置的还可以是空心图形,如圆环,以空心部分的空心图形作为子标记物212中对应的一个特征点214。以此类推,在子标记物中设置层层嵌套的空心图形,如层层嵌套的圆环,以最后被嵌套的空心圆形作为特征点214。其中,子标记物212中空心图形的嵌套层数可以根据实际识别需求设定,或根据图像采集装置的分辨率确定。
在一个实施例中,在标记物210的子标记物212中,可以存在一个子标记物212由相互分离的实心图形构成,每个实心图形为一个特征点214。例如图7中,相互分离的各个黑色实心圆214构成一个子标记物212c,各个黑色实心圆为子标记物212c中的特征点214。
为便于对各个标记物210进行区分识别,确定各个标记物210的身份信息,在一个虚拟场景中每个标记物210的内容互不相同。
在一个实施例中,可以是标记物210包含的子标记物212的数量与其他标记物包含的子标记物的数量不同。例如,有3个标记物210,该3个标记物210的子标记物212的数量分别为x、y、z,其中,x、y、z可以是大于或等于1的整数,x、y、z不相等。
在一个实施例中,可以是标记物210中存在至少一个子标记物212的特征点214的类型与其他标记物210中子标记物212的特征点214的类型不同,例如标记物210中有一个子标记物212包括的特征点214为实心圆,其他任意一个标记物210中都不包括特征点214类型为实心圆的子标记物212。
在一个实施例中,可以是标记物210中存在至少一个子标记物212中空心图形的嵌套层数与其他其他标记物210的子标记物212的嵌套层数不同。例如,只有一个标记物210中的一个子标记物212的空心部分设置了一个实心圆点,该实心圆点作为该子标记物212的特征点214。当处理器识别到在空心部分设置有一个实心圆点的子标记物212时,可以确定该子标记物212对应的标记物210的身份,则该标记物210即为预设标记物模型中在空心部分设置有一个实心圆点的子标记物212对应的标记物210。
在一个实施例中,也可以是标记物210对应的数量组合与其他标记物210对应的数量组合不同。每个标记物210中各子标记物212的特征点214的数量构成该标记物210中的数量组合。以图7为例,该标记物210中包括四个子标记物212,其中,子标记物212a的特征点数量为3,子标记物212b的特征点数量为2,子标记物212c的特征点数量为5,子标记物212d的特征点数量为1,该四个子标记物的特征点214的数量形成了该标记物210中的数量组合。该数量组合可以是将子标记物按一定方向排列的数量组合。例如,将子标记物按顺序针方向排列的数量组合可以是3152,按逆时针方向排列的数量组合可以是3251等,其中,作为数量组合的起点的子标记物可以是任意选取的一个子标记物,也可以选取包含特征点数量最多或最少的子标记物等。标记物210对应的数量组合也可采用其他方式进行表示,并不限于上述描述的方式。
在上述识别跟踪系统中,交互装置可以为平面标记物体、曲面标记物体或立体标记结构体等,可以根据不同的虚拟场景进行设计。
图8示出了本申请的一种图像处理方法,应用于上述识别跟踪系统,以头戴显示装置的处理器作为执行主体。该识别跟踪系统包括图像采集装置和具有标记物的交互装置。该方法可包括步骤S110至步骤S130。
步骤S110:处理器获取图像采集装置采集的具有标记物的目标图像。
其中,交互装置位于真实场景中。目标图像为图像采集装置采集的具有交互装置的图像,该目标图像内包括交互装置的标记物,其中,交互装置可以是上述实施例中提及的任一交互装置,也可以是其他结构形态的交互装置。
步骤S120:处理器根据目标图像确定交互装置在真实场景内的位置及姿态信息。
其中,交互装置在真实场景内的位置及姿态信息,可包括交互装置在真实场景内的位置及旋转角度等信息。具体地,位置信息可指的是交互装置在真实场景内的空间位置信息,姿态信息可指的是交互装置的旋转信息,该位置及姿态信息可以是交互装置与图像采集装置之间的位置及姿态信息。所采集的目标图像内的交互装置可以是一个,也是可以多个。当所采集的目标图像内交互装置为多个时,处理器可获取在目标图像内的每个交互装置与图像采集装置之间的位置及姿态信息。
在一个实施例中,处理器获取目标图像,并识别目标图像中包含的标记物,确定目标图像中标记物的身份信息。处理器根据标记物的身份信息,可以确定标记物对应的交互装置,生成对应的虚拟对象;还可以确定交互装置是平面标记物体还是多面标记结构体,以采用对应的定位跟踪方法对该交互装置进行跟踪,进而获取交互装置与图像采集装置之间的位置及姿态等信息。
步骤S130:处理器根据位置及姿态信息确定与交互装置对应的虚拟场景。
处理器根据交互装置的位置及姿态信息可确定与交互装置对应的显示内容,通过头戴显示装置的显示装置及光学组件将显示内容呈现给用户,以产生虚拟场景叠加在真实场景中的效果。在一个实施例中,头戴显示装置中可预先存储有不同位置及姿态信息与显示内容的对应关系,处理器在获取到交互装置与图像采集装置之间的位置及姿态信息之后,根据该对应关系,查找当前的交互装置与图像采集装置之间的位置及姿态信息对应的显示内容。
处理器将显示内容发送至显示装置,指示显示装置显示与位置及姿态信息对应的显示内容,显示装置将显示内容进行显示,并经过光学元件投射至对应位置。当用户佩戴头戴显示装置时,该对应的位置可为用户的双眼位置,用户的双眼能够观察到该显示内容。当光学元件具有一定透光度时,真实环境也会被用户观察到,用户能够观察到显示内容与真实环境叠加的视觉效果。在一个实施例中,用户可以使用多个交互装置,以在虚拟场景内产生更多的显示内容,进一步提高用户的体验感;用户还可通过交互装置与显示的虚拟内容进行交互。
如图9所示,以图像采集装置的视觉范围内包括平面标记物体和多面标记结构体为例,示例性地,平面标记物体可以为第一标记板,多面标记结构体可以为二十六面标记结构体。图像采集装置采集到用户视野内的交互装置的图像,处理器分析该图像,确定第一标记板的身份信息,以及头戴显示装置与第一标记板之间的位置及姿态信息,渲染生成对应的显示内容,将该显示内容通过显示装置显示,并通过光学组件将该显示内容投射至用户的眼镜。在一个实施例中,用户可通过光学组件观看到真实场景,进而观察到显示内容与外界真实场景的叠加的视觉效果。
图10为一个实施例中显示的虚拟场景的效果图,虚拟物体w1表示的是水杯,虚拟物体w2表示的是汤勺,且该汤勺内装有食物,虚拟物体w3表示的是桌子。相对应的,虚拟物体w1的显示位置(即用户看到的位置)可对应图9中的标记物210A在真实场景中的位置,虚拟物体w2的显示位置可对应图9中的第一标记板在真实场景中的位置,虚拟物体w3的显示位置可对应图9中的二十六面标记结构体在真实场景中的位置。用户可手持二十六面标记结构体,向第一标记板上的标记物210A的位置处移动,以到达图9中的二十六 面标记结构体所示的位置处时,用户可通过头戴显示装置看到如图10中所示的虚拟图像。合理设置显示装置所显示的显示内容,使得用户在观察到图10所示的虚拟图像时,所看到的虚拟图像能够准确叠加在第一标记板和二十六面标记结构体上,视觉效果更佳。
在一个实施例中,随着交互装置与图像采集装置之间的位置及姿态信息的变化,头戴显示装置中显示的增强现实的虚拟场景也会相应发生变化。具体地,处理器获取交互装置与图像采集装置之间的姿态信息的变化量,根据变化量调整所显示的显示内容,以使增强现实场景跟随该姿态信息的变化量而相应变化。
图11为一个实施例中基于交互装置与图像采集装置之间的不同位置及姿态信息显示的不同虚拟场景的示意图。当头戴显示装置显示如图11(a)所示的虚拟场景时,交互装置与图像采集装置之间的姿态信息为S1,随着用户走动或者转动,交互装置与图像采集装置之间的姿态信息发生改变,头戴显示装置显示的虚拟场景可随着该姿态信息的变化而变化。当交互装置与图像采集装置之间的姿态信息变为S2时,头戴显示装置显示的虚拟场景可变成如图11(b)所示的虚拟场景。其中,图11(a)所示的虚拟物体w1可为正面,而图11(b)所示的虚拟物体w1可为背面,由此可以看出,当交互装置与图像采集装置之间的姿态信息发生改变时,用户可以观察到不同视觉角度下的虚拟物体,例如,可以观察到从虚拟物体w1的正面到虚拟物体w1的背面的变化过程。可以理解地,图11中的矩形框仅是用于示意图像的大小,用户在观察的时候可不看到该矩形框。
在一个实施例中,当多个交互装置之间的位置及姿态信息发生变化时,虚拟场景也会相应变化,处理器根据每个交互装置的位置及姿态信息确定各个交互装置之间的位置及姿态信息,根据每个交互装置与图像采集装置之间的位置及姿态信息,以及各个交互装置之间的位置及姿态信息确定显示的虚拟场景,确定每个交互装置对应的虚拟图像,多个虚拟图像用于构成虚拟场景。作为一种实施方式,处理器可判断至少两个交互装置之间的位置及姿态信息是否满足预设标准,当满足预设标准时,处理器可修改该至少两个交互装置对应的虚拟图像,以使显示的虚拟场景发生变化。其中,预设标准为根据需要而设定的标准,例如,可以是一个预设角度或预设距离值。
图12为一个实施例中基于多个交互装置之间的不同位置及姿态信息显示的不同虚拟场景的示意图,当图像采集装置采集到第一标记板的图像时,头戴显示装置显示如图12(a)中所示的虚拟场景,该虚拟场景为桌子上叠加显示一根蜡烛,其中,蜡烛的位置可以是第一标记板的某一个标记物的位置,当用户手持二十六面标记结构体靠近第一标记板的标记物210A时,图像采集装置同时采集到第一标记板和二十六面标记结构体的图像,头戴显示装置显示如图12(b)中所示的虚拟场景,燃烧的火柴棒可以是对应二十六面标记结构体的虚拟图像,当第一标记板和二十六面标记结构体之间的位置及姿态发生变化,例如二十六面标记结构体逐渐靠近第一标记板的标记物210A,显示的虚拟场景可以是燃烧的火柴棒逐渐靠近蜡烛。当二十六面标记结构体移动到某一个预定位置时,头戴显示装置显示的虚拟场景可以是燃烧的火柴棒将蜡烛点燃,当二十六面标记结构体消失在图像采集装置的视野范围内时,头戴显示装置显示的虚拟场景可如图12(c)所示,虚拟场景变为桌子上设置了一个被点燃的蜡烛。
图13示出了本申请一个实施例中的图像处理方法,该方法可应用于图1所示的识别跟踪系统,以头戴显示装置的处理器作为执行主体。该方法可包括步骤S110至S130,其中,步骤S120包括步骤S122至S126。
步骤S110:处理器获取图像采集装置采集的具有标记物的目标图像。
步骤S120:处理器根据目标图像确定交互装置在真实场景内的位置及姿态信息。
步骤S120包括步骤S122至S126。
步骤S122,处理器确认目标图像中标记物的身份信息。
处理器获取图像采集装置采集的具有标记物的目标图像,目标图像中包括至少一个具有多个子标记物的标记物。在一个实施例中,标记物包括的子标记物的数量可以大于或等 于4。该目标图像中,还可包括标记物之间的部分,即部分第一背景。
处理器可根据目标图像中标记物的特征获取标记物的身份信息。在一个实施例中,处理器可先对目标图像进行预处理,以获得能体现目标图像中各种特征信息的处理后的目标图像。处理器对目标图像进行预处理,从目标图像中分辨出第一背景、第二背景,以及子标记物、特征点各自对应的连通域。作为一种具体的实施方式,对目标图像进行预处理可以是对目标图像进行二值化处理,使目标图像中的第一背景与子标记物之间有区分,子标记物与第二背景之间有区分。处理器可采用固定阈值法、适应阈值法等对目标图像进行二值化处理,也可采用其他方法进行二值化,在此不作限定。
在一个实施例中,对于采集的连续多帧目标图像,其中的首帧目标图像可以通过全局固定阈值法、帧间固定阈值、自适应阈值法等方法进行二值化,获得该首帧目标图像的二值化图像。其中,处理器以图像采集装置开启后拍摄的第一帧目标图像作为首帧目标图像,也可以是以图像采集装置拍摄过程中的任意一帧目标图像作为首帧目标图像。连续多帧目标图像即是从该首帧目标图像起,后续拍摄的多帧目标图像,该连续多帧目标图像可以是图像采集装置拍摄的时间上依次邻接的目标图像,也可以是在时间上具有先后顺序的、彼此具有帧间隔的目标图像,在本申请实施例中并不限定,可根据实际需要确定。对于连续多帧目标图像中除首帧目标图像以外的其他帧目标图像,可以通过以下实施例中描述的方式进行二值化处理。
处理器可获取连续多帧目标图像中除首帧目标图像外的当前帧目标图像对应的第一阈值图像P1,并根据第一阈值图像P1对当前帧目标图像进行二值化。当处理器对连续多帧目标图像中除首帧目标图像外的任意一帧目标图像进行二值化,进行二值化的该帧目标图像可作为当前帧目标图像。处理器可获取当前帧目标图像对应的第一阈值图像P1,其中,当前帧目标图像对应的第一阈值图像P1,为对历史帧目标图像进行图像处理后获得的且与该当前帧目标图像分辨率相同的灰度图像,该历史帧目标图像指的是连续多帧目标图像在当前帧之前的目标图像,历史帧目标图像可以是一帧或多帧。例如,当前帧目标图像的分辨率为m*n,该第一阈值图像P1的分辨率也为m*n。在一个实施例中,当前帧目标图像的分辨率可为相机等图像采集装置获取到该当前帧目标图像的分辨率。
在一个实施例中,处理器对历史帧目标图像进行处理后获得第一阈值图像P1,第一阈值图像P1中每个像素点的像素值,可以是处理器通过对历史帧目标图像中每一个像素点的周围其他像素点进行处理后获得的像素值,并将该获得的像素值作为第一阈值图像P1中对应像素点的像素值。第一阈值图像P1中每一个像素点的像素值,由在历史帧目标图像中对应像素点的周围多个像素点共同决定。
图14为一个实施例中获取第一阈值图像P1的流程图,在一个实施例中,处理器获取连续多帧目标图像中除首帧目标图像外的当前帧目标图像对应的第一阈值图像P1的方法可包括步骤S221至S223。
步骤S221:处理器获取对历史帧目标图像进行处理后的第二阈值图像P2,第二阈值图像P2的分辨率为第一预设分辨率,第一预设分辨率低于当前帧目标图像的分辨率。
在一个实施例中,该第二阈值图像P2的第一预设分辨率,可以是硬件等其他外在组件的要求范围内的分辨率,例如,可由硬件端存储器支持的用于存储第二阈值图像P2的内存空间而定,一般来说,内存空间越小,第一预设分辨率可越小。
图15为一个实施例中获取第二阈值图像P2的流程图,在一个实施例中,处理器对历史帧目标图像进行处理,并获取第二阈值图像P2的方法可包括步骤S221a至S221c。
步骤S221a:处理器对历史帧目标图像进行降采样,获得具有第二预设分辨率的降采样图像。
作为一种实施方式,第二预设分辨率的大小并不限定,即降采样的系数并不限定。例如,以N作为降采样系数,按照行列均为1/N的比例采样,将历史帧目标图像中每N*N个像素点降维成1*1个像素点,其中N的大小并不限定;或者是,将N1*N2个像素点降维为 1*1个像素点,N1和N2的值可以不同,且N1、N2的具体值也不作为限定。可以根据实际处理需要设置,使历史帧目标图像降采样为第二预设分辨率的图像。
处理器对历史帧目标图像进行降采样的具体实现方法也不限定。例如,处理器可对历史帧目标图像中,行列间隔为N个像素点的N*N的区域求像素均值,并将该像素均值作为降采样图像中该N*N个像素点降采样后对应的像素点的像素值,作为一种实施方式,当像素均值小于预设最小像素值t时,可以将该像素均值设置为该预设最小像素值t;或者是,处理器可在历史帧目标图像中,行列每隔N个像素点取一个像素点,作为降采样图像中对应的像素点;或者是,处理器可对历史帧目标图像中,行间隔为N1个像素点,列间隔为N2个像素点的区域求像素值,作为降采样后该N1*N2个像素点采样后对应的像素点的像素值。
步骤S221b:处理器根据降采样图像计算并获取具有第二预设分辨率的第三阈值图像P3。
处理器可根据降采样图像中每个像素点在预设窗口范围内的各个像素点的像素值,确定第三阈值图像P3中每个像素点的像素值,以获取具有第二预设分辨率的第三阈值图像P3。
在一个实施例中,第三阈值图像P3中的每个像素点的像素值,可以是根据降采样图像中对应像素点在预设窗口范围内所有像素点的像素值得到。例如,处理器可确定降采样图像中第x行第y列的像素点对应的预设窗口,并根据该对应的预设窗口范围内所有像素点的像素值,得到第三阈值图像P3中第x行第y列的像素点的像素值。作为一种实施方式,处理器可以在降采样图像上采用窗口大小为W*W(W的值一般比较小)的自适应阈值操作,得到第三阈值图像P3。例如,处理器可对降采样图像中第x行第y列的像素点在W*W大小的窗口上采用自适应阈值操作,得到第三阈值图像P3中第x行第y列的像素点的像素值。
在一个实施例中,为了提高处理器获取第三阈值图像P3的速度,可以采用降采样图像的积分图信息。处理器可获取降采样图像的积分图,在一个实施例中,积分图中的任意一像素点(x,y)的像素值,可以是从降采样图像的左上角到该像素点(x,y)所构成的矩形区域内所有的像素点的灰度值之和。处理器可根据积分图计算并获取具有第二预设分辨率的第三阈值图像P3,可根据积分图中每个像素点在预设窗口范围内各个像素点的像素值,确定第三阈值图像P3中每个像素点的像素值,以获取具有第二预设分辨率的第三阈值图像P3。
处理器获得的第三阈值图像P3中的每个像素点,可以是根据积分图中对应像素点在预设窗口范围内所有像素点的像素值得到。例如,处理器可确定积分图中第x行第y列的像素点对应的预设窗口,并根据该对应的预设窗口范围内所有像素点的像素值,得到第三阈值图像P3中第x行第y列的像素点的像素值。作为一种实施方式,处理器可以在积分图上采用窗口大小为W*W(W的值一般比较小)的自适应阈值操作,得到第三阈值图像P3。
例如,处理器可对积分图中第x行第y列的像素点对应的W*W大小的窗口采用自适应阈值操作,获取该对应的W*W大小的窗口中所有像素点的像素均值,并将该像素均值作为第三阈值图像P3中第x行第y列的像素点的像素值。在一个实施例中,处理器可将第x行第y列的像素点在W*W大小的窗口中对应的像素均值,乘以均值放大系数后,获得第三阈值图像P3中第x行第y列的像素点的像素值。
在一些实施方式中,当积分图中某一像素点的预设窗口范围超出积分图边缘,则处理器可获取该像素点对应预设窗口范围内在积分图中的有效像素点,并根据有效像素点计算在第三阈值图像P3中对应像素点的像素值。例如,预设窗口大小为w*w,w=2*a+1,对于积分图中的像素位置为(i,j)的像素点,该像素点对应的窗口的四顶点位置分别为(i-a-1,j-a-1),(i+a,j-a-1),(i-a-1,j+a),(i+a,j+a)。如图16a所示,当窗口的四顶点位置全部处于积分图的有效区域范围内,根据该窗口中所有像素点的像素值,可获取像素位置为(i,j)的像 素点的像素值。如图16b所示,当窗口存在至少一个顶点的顶点位置处于积分图的有效范围外时,窗口的实际有效区域可以是阴影区域与背景网格重合的区域,实际有效区域的大小小于窗口大小w*w,此时处理器可根据该窗口在积分图中的有效区域内包含的像素点的像素值,计算在第三阈值图像P3中像素点(i,j)的像素值,即根据如图16b中阴影区域与背景网格重合的区域内的像素点的像素值计算。
上述实施例中的窗口大小并不进行限定,可以根据实际需要设定。进一步地,窗口大小与获取降采样图像时的降采样系数均可根据原始图像大小、硬件所支持的最大窗口大小以及该历史帧目标图像中图像对象的物理特征大小等确定,以保证图像中最小的图像对象能够在获得的第三阈值图像P3中得以体现。当处理器通过自适应阈值的操作获取第三阈值图像P3时,即使是图像中最小的图像对象也能够在自适应阈值过程中得以体现。作为一种实施方式,对于不同的目标图像,预设窗口大小可以不同,具体可以根据目标图像中对应的物体的大小设置相应的窗口大小,当物体较大或者靠近相机时,可以设置较大的窗口。对应的,设置的窗口大小不同,处理器计算获得的第三阈值图像P3中各个像素点的像素值也不同。
步骤S221c:当第二预设分辨率大于第一预设分辨率时,处理器对第三阈值图像P3进行降采样,直到获得分辨率小于或等于第一预设分辨率的第二阈值图像P2。
当第三阈值图像P3的第二预设分辨率大于第一预设分辨率,处理器可继续对获得的第三阈值图像P3进行降采样,直到获得分辨率小于或等于第一预设分辨率的第二阈值图像P2,以获得存储空间尽可能小的阈值图像。例如,在上述以N作为降采样系数获得的第二预设分辨率的第三阈值图像P3的基础上,继续以降采样系数为M进行采样,在该第三阈值图像P3中,将M*M个像素点降维为1*1个像素点。当降采样后的第三阈值图像P3的分辨率依然大于第一预设分辨率,处理器可继续以降采样系数M对降采样后的第三阈值图像P3进行降采样处理,直至获得分辨率小于或等于第一预设分辨率的第三阈值图像P3,并将该第三阈值图像P3作为第二阈值图像P2。
当第三阈值图像P3的第二预设分辨率小于或等于第一预设分辨率,则处理器可以不再对第三阈值图像P3进行降采样,该第三阈值图像P3即为分辨率为第一预设分辨率的第二阈值图像P2。
在一个实施例中,处理器可以将获得的第二阈值图像P2存储于存储器中,以备后续使用。第二阈值图像P2在最终程序里面只需要存储非常小的图像即可,有效地节省了内存空间。例如,在对历史帧目标图像进行两次降采样系数分别为N和M的降采样后,得到的第二阈值图像所占内存空间仅是未降采样时的1/(N*N*M*M),对于一些对内存有严格要求的硬件,例如FPGA,是至关重要的。比如,对于该历史帧目标图像为1280x800的图像,取N=4,M=8,最终存储的第二阈值图像P2为40x25,内存空间为之前的1/4。
在一个实施例中,上述实施方式中获取第二阈值图像P2的方式可以不作为步骤S121的具体实施方式,而是独立进行,并存储获取的第二阈值图像P2。步骤S121中获取具有第一预设分辨率的第二阈值图像P2时,处理器可直接从存储器获取的预先存储的第二阈值图像P2。
步骤S223:处理器对第二阈值图像P2进行升采样,获得与当前帧目标图像分辨率相同的第一阈值图像P1。
当处理器需要根据当前帧对应的第一阈值图像P1对当前帧进行二值化时,可根据该当前帧的历史帧获得第二阈值图像P2,并对第二阈值图像P2进行升采样,将分辨率为第一预设分辨率的第二阈值图像P2进行升采样的插值,获得分辨率与当前帧目标图像相同的第一阈值图像P1。例如,第二阈值图像P2为根据降采样系数N对历史帧目标图像进行处理得到,历史帧目标图像分辨率和当前帧相同,则将第二阈值图像P2进行升采样系数为N的升采样得到第一阈值图像P1。
在该升采样的具体实现方式在本申请实施例中并不限定,如可以通过双线性插值算法 等进行插值实现。其中,双线性插值,又称为双线性内插。如图17所示,已知Q
12,Q
22,Q
11,Q
21,但是要插值的点为P点,需要得到点P=(x,y)的值。假设我们已知函数f在Q
11=(x
1,y
1),Q
12=(x
1,y
2),Q
21=(x
2,y
1),及Q
22=(x
2,y
2)四个点的值。
首先在x方向进行线性插值,得到:
然后在y方向进行线性插值,得到
这样就得到所要的结果f=(x,y),
在一个实施例中,处理器获取第一阈值图像P1后,对于当前帧目标图像的每一个像素点,可将第一阈值图像P1中对应位置的像素点的像素值作为二值化阈值,将当前帧目标图像二值化。处理器以第一阈值图像P1中每一个像素点的像素值作为当前帧目标图像中对应位置像素点的二值化阈值。其中,对应位置即为,在分辨率相同的当前帧目标图像及第一阈值图像P1中,坐标相同的位置,如在当前帧目标图像中,第2行第3列的像素点,与第一阈值图像P1中第2行第3列的像素点为对应位置的像素点。
作为一种实施方式,在二值化过程中,对于当前帧目标图像的每一个像素点,当当前帧目标图像的像素点的像素值大于第一阈值图像P1中对应位置的像素点的像素值时,处理器可将当前帧目标图像中该像素点的像素值设置为第一像素值;当当前帧目标图像的像素点的像素值小于或等于第一阈值图像P1中对应位置的像素点的像素值,处理器可将当前帧目标图像中该像素点的像素值设置为第二像素值,得到当前帧目标图像的二值化图像。例如,在当前帧目标图像中位置为(i,j)的像素点,像素值为232,第一阈值图像P1中位置为(i,j)的像素点,像素值为100,将当前帧目标图像中位置为(i,j)的像素点的像素值设置为二值化图像中的第一像素值1;在当前帧目标图像中位置为(I,J)的像素点,像素值为50,第一阈值图像P1中位置为(I,J)的像素点,像素值为200,将当前帧目标图像中位置为(I,J)的像素点的像素值设置为二值化图像中的第二像素值0。
对于连续多帧目标图像中除最后一帧目标图像的任意一帧目标图像,处理器都可以进行处理获得第二阈值图像P2,用于后一帧目标图像进行升采样后获得对应的第一阈值图像P1,从而对该后一帧目标图像进行二值化。对于连续多帧目标图像中除首帧以外的任意一帧目标图像,都可以获取其对应的第一阈值图像P1,根据对应的第一阈值图像P1进行二值化处理。每帧目标图像进行二值化的过程,进行处理获得第二阈值图像P2的过程, 先后顺序并不限定。每帧目标图像进行二值化的过程,以及对该帧目标图像处理后获得对应下一帧目标图像的第一阈值图像P1的过程,处理的先后顺序可以没有限定。
在对目标图像进行二值化过程中,各个像素点的二值化阈值可能并不相同,每个像素点的二值化阈值都取决于目标图像对应的第一阈值图像P1,由于历史帧与后一帧之间具有连续性,因此,目标图像的二值化阈值被设置成最适合当前场景,并随场景变化实时更新,更符合当前的二值化场景需求。
处理器将获得的连续多帧目标图像的每一帧目标图像进行二值化后,目标图像中包含的第一背景、第二背景以及子标记物分别对应相应的二值化像素值。作为一种实施方式,目标图像进行二值化后,处理器可将目标图像中标记物之间的部分以及子标记物处理为第一颜色,标记物中除子标记物以外的部分为第二颜色。
在一个实施例中,处理器将标记物中依次呈包围关系的各个部分,处理成具有颜色层次,使各部分之间形成依次包围的连通域。以图18中所示的标记物为例,处理器可将目标图像中第一背景1810对应的部分处理为第一颜色,将标记物210中的第二背景1820处理为第二颜色,将子标记物212处理为第一颜色,将子标记物围成的空心部分处理为第二颜色。若子标记物的空心部分中还包括实心图形,如图7中子标记物212b所示,将该实心图形处理为第二颜色。其中,第一颜色与第二颜色可以是像素值差别较大的颜色,如第一颜色为黑色,第二颜色为白色。当然,二值化后的图像,第一背景、第二背景以及子标记物、特征点之间的区分也可以通过对比度等其他方式,本申请实施例主要以颜色层次为例进行说明。
处理器可获取目标图像中的连通域信息,并基于该连通域信息获取所有连通域的包围关系,再根据目标图像中多个连通域之间的包围关系,以及预先存储的标记物的特征,确定目标图像中标记物的身份信息为对应的预存储标记物的身份信息,其中,连通域是指图像中具有相同像素值且位置相邻的像素点组成的图像区域。在一个实施例中,处理器获取目标图像中的连通域信息,可以用4路或8路连通性计算标记为布尔图像的连通分量,输出连通域的数量,其中,可以根据包围关系输出各个连通域的类型,即输出对应目标图像第一背景、第二背景、子标记物、特征点等各个部分的连通域。
如图18所示的目标图像中,第一背景1810为一个连通域,标记物中第二背景1820为一个连通域,每一不包含黑点的子标记物212是一个连通域,子标记物中的白点(即特征点214)是一个连通域,包括黑点(即特征点214)的子标记物212中,每一个黑点是一个连通域。其中,不包含黑点的子标记物为空心图形的子标记物,其中的白点为特征点,包括黑点的子标记物,黑点为特征点。
处理器可基于目标图像中的连通域获取各个连通域之间的包围关系。以图18中包含有3个白点的子标记物212a为例,该子标记物212a为一个连通域,该连通域内包括三个白点214,每个白点214分别为一个连通域,各个白点214对应的连通域被该子标记物212a对应的连通域包围。
在一个实施例中,如图18所示,在目标图像中,第一背景1810、第二背景1820以及子标记物之间形成了包围关系,若子标记物为空心图形,子标记物与空心图形中包含的空心部分对应还具有包围关系,如图18中包括白点的子标记物,与白点之间形成包围关系。其中,第一背景包围第二背景,第二背景包围子标记物,子标记物还包围其中的白点,即空心部分。也就是说,第一背景、第二背景以及子标记物分别对应的连通域具有包围关系,子标记物对应的连通域与其空心部分对应的连通域之间也有包围关系。
在一个实施例中,可以将第一背景对应的连通域定义为第四连通域,处理器可先确定第四连通域。在目标图像中,第一背景包围了所有标记物,因此,可以将目标图像中包围其他所有连通域的连通域作为第四连通域。以二值化后的目标图像包括第一颜色和第二颜色为例,其中确定的第四连通域可满足如下条件:颜色为第一颜色、包围有第二颜色的连通域,且未被第二颜色的连通域包围。
目标图像的第一背景包围标记物,则第四连通域包围标记物中第二背景对应的连通域,可定义该第二背景对应的连通域为第一连通域。处理器可以将被第四连通域包围、且与第四连通域相邻的连通域作为第一连通域,第四连通域包围的每个第一连通域对应一个标记物,标记物中包围其他连通域的连通域为第一连通域。以二值化后的目标图像包括第一颜色和第二颜色为例,处理器可确定被第四连通域包围且与第四连通域相邻,同时颜色为第二颜色的连通域为第一连通域。
由于标记物中包括各个子标记物,每个子标记物具有特征点,则可以定义被第一连通域包围且与第一连通域相邻的连通域为第二连通域,即可定义子标记物对应的连通域为第二连通域。可以定义第二连通域包围的连通域为第三连通域,即若子标记物为如图18中所示的包围白点的空心图形,空心部分(即包围的白色部分,也就是白色特征点)对应的连通域定义为第三连通域,每个第三连通域是一个特征点。当第二连通域不包围第三连通域,则可确定每个不包围第三连通域的第二连通域为一个特征点。
在一个实施例中,如图18中对应的各个连通域的包围关系可以由如图19所示的树状图表示,在图19中,在树状图中的第一层级中的B,可对应第一背景1810的连通域(第四连通域);第二层级中的W,可对应第二背景1820的连通域(第一连通域);第三层级中的B1、B3、B2、B5分别对应四个子标记物的连通域;第四层级中的w和b分别用于表示子标记物中包含的黑点及白点对应的连通域,其中的W和B可分别用于表示连通域的颜色为白色或黑色,也可以用于表示连通域的编码,在此不作限定。各个第一连通域被第四连通域包围,各个第二连通域被相应的第一连通域包围,各个第三连通域被相应的第二连通域包围。处理器可以获取到每个第一连通域包围的第二连通域,以及每个第一连通域中包围的第二连通域的数量,还可获取每个第二连通域包围的第三连通域,以及每个第二连通域包围的第三连通域的数量。
在一个实施例中,处理器可根据目标图像中连通域的包围关系,以及预存储的标记物的特征,判断目标图像中的标记物是否包含预存储标记物的包含样式。处理器可根据连通域的包围关系区分出目标图像中包含的各个标记物,其中,每个第一连通域可对应一个标记物,或者说,每个第一连通域及其包围的第二连通域、第三连通域构成目标图像中的一个标记物。存储器中可预先存储有标记物的特征和身份信息,处理器根据预存储的标记物的特征,将目标图像中标记物的特征与预存储的标记物的特征进行比对,从而确定目标图像中标记物的身份信息。其中,可将标记物的身份信息及特征对应预先存储在存储器中。
在一个实施例中,预存储标记物的特征可包括标记物中对应的连通域信息,连通域分别包括第一连通域、第二连通域以及第三连通域,其中,连通域信息可包括连通域之间的包围关系,例如第一连通域包围的第二连通域以及包围的各第二连通域的数量,每个第二连通域包围的第三连通域以及包围的第三连通域的数量等。
作为一种实施方式,当多个预存储标记物对应的数量组合不同时,则处理器可根据目标图像中的标记物的数量组合,获取数量组合相同的预存储标记物,则该数量组合相同的预存储标记物的身份信息为该目标图像中标记物的身份信息,其中,标记物的数量组合可指的是标记物包含的各个子标记物的特征点的数量组合。在一个实施例中,对于目标图像中每个第一连通域,可根据预先存储标记物的特征确定预先存储的对应第一连通域,其中,相互对应的第一连通域包围有相同数量的第二连通域、且包围的各个第二连通域所包围的第三连通域的数量一一对应。例如,以图18中标记物210为例,在目标图像中的该标记物210的第二背景1820对应的第一连通域,包括8个第二连通域,其中5个第二连通域不包括有第三连通域,该5个第二连通域对应5个特征点,构成一个子标记物212c;其中3个第二连通域包括有第三连通域,该3个第二连通域各对应一个子标记物,分别包围1个第三连通域、3个第三连通域、两个第三连通域,即是3个子标记物分别具有1个特征点,3个特征点,2个特征点,且各个特征点为白点。处理器可在预存储的标记物的特征中,查找包括4个子标记物,该4个子标记物的特征点分别为1个白点、3个白点、2个 白点、5个黑点的标记物,查找到的标记物的身份信息即为图18所示的标记物的身份信息。
在一个实施例中,预先存储的标记物的特征中包括连通域信息,连通域信息可包括连通域之间的包围关系,该连通域之间的包围关系可通过编码进行表示,其中,每个第二连通域对应一个编码,第二连通域包围的第三连通域数量不同,对应的编码不同。处理器获取目标图像中多个连通域之间的包围关系,可对目标图像中,包围不同数量第三连通域的第二连通域,设置不同的对应编码,其中第三连通域数量与编码的对应关系可与预存储标记物的第三连通域数量与编码的对应关系相同。例如,若预存储标记物中,包围有一个第三连通域的第二连通域编码为B1,包围有两个第三连通域的第二连通域编码为B2,包围有两个第三连通域的第二连通域编码为B3,以此类推。则在对目标图像中的第二连通域进行编码时,包围有一个第三连通域的第二连通域编码为B1,包围有两个第三连通域的第二连通域编码为B2,包围有两个第三连通域的第二连通域编码为B3,以此类推。
在一个实施例中,在对连通域进行编码时,如预存储的多个标记物对应的编码中,第四连通域可通过第一编码表示,第一连通域通过第二编码表示。处理器获取目标图像中多个连通域之间的包含关系时,可以通过第一编码表示第四连通域,通过第二编码表示第一连通域。其中,仍然主要通过第一连通域中各个第二连通域的编码确定第一连通域的身份信息,从而确定第一连通域对应的标记物的身份信息。
处理器可根据目标图像中标记物的编码,在预存储的标记物中查找编码相同的标记物,从而确定目标图像中标记物的身份信息。作为一种实施方式,目标图像中每个第一连通域包围一个或多个第二连通域,每个第一连通域对应一个标记物,则目标图像中标记物对应的编码,可以是该标记物包含的各个第二连通域对应的各个编码。同样的,预存储标记物对应的编码可以是预先存储的标记物中各个第二连通域的编码。处理器可在预存储标记物的编码中,获取与目标图像中标记物的编码相同的编码,该具备相同编码的预存储标记物对应的身份信息即为目标图像中标记物的身份信息。其中,标记物中各个第二连通域的编码顺序并不限定,例如,对于编码B0B1B2B3的标记物,在预存储标记物中获取与B0B1B2B3相同的编码,其中各个第二连通域的编码B0、B1、B2、B3的顺序不限定,例如获取到的编码为B1B2B0B3,也认为与B0B1B2B3相同。
在一个实施例中,标记物之间的不同,也可以是该多个标记物包括的子标记物的数量不同,例如,多个预设预存储标记物中,只有一个预存储标记物对应有第一数量的子标记物。在目标图像中,若某个标记物中包括第一数量的子标记物,该标记物与有第二数量的子标记物的预存储标记物对应。进一步地,当预存储标记物中只有一个第一连通域包围有第一数量的第二连通域,且在目标图像中某个第一连通域包围第一数量的第二连通域,则该目标图像中的第一连通域对应的标记物与包围有第一数量的第二连通域的预存储标记物对应。
作为一种实施方式,可以将每个不包括白色特征点的黑色点作为一个特征点,所有不包括白色特征点的黑色点作为一个子标记物。也就是说,可将每个不包围第三连通域的第二连通域作为一个特征点,所有不包围第三连通域的第二连通域作为一个子标记物,则在该识别过程中,将包围有第三连通域的每个第二连通域统计数量为1,不包围第三连通域的所有第二连通域统计数量为1。
在一个实施例中,目标图像中的标记物并不一定为完整的标记物,若只获取到标记物的一部分,且该标记物与其他标记物差异化较大,具有其他标记物所没有的特征,可以根据该标记物的特征判定该标记物的身份。
例如,在多个预存储标记物中,存在一个预存储标记物的至少一个子标记物的特征点数量与其他标记物中子标记物的特征点数量不同,也就是说,多个预存储标记物中,只有一个第一连通域包围有一个特定的第二连通域,该特定的第二连通域包围有第二数量的第三连通域。在目标图像中,当存在一个第一连通域中,包围的第二连通域包围有第二数量的第三连通域,该目标图像中的第一连通域对应的标记物与该特定的第二连通域所对应的 预存储标记物对应。
或者是,多个预存储标记物中,存在一个第一连通域包围有第三数量个不包围第三连通域的第二连通域。在目标图像中,当存在一个第一连通域,包围有第三数量个不包围第三连通域的第二连通域,该目标图像中的第一连通域对应的标记物与该第三数量的第二连通域所对应的预存储标记物对应。
在一个实施例中,若多个预存储标记物的特征中,某个预存储标记物的至少一个子标记物中空心图形的嵌套层数与其他子标记物的嵌套层数不同,则在目标图像中,若某个子标记物的图形嵌套层数与该预存储标记物中该子标记物的嵌套层数相同,该子标记物对应的标记物与该预存储标记物对应。也就是说,预存储的多个预存储标记物中,若只有一个第一连通域包含依次包围的第四数量的连通域,则当目标图像中,存在第一连通域包含依次包围的第四数量的连通域,确定该目标图像中的第一连通域对应的标记物与该第四数量的连通域对应的预存储标记物对应。
处理器确定与目标图像中标记物对应的预存储标记物,可获取该对应的预存储标记物的身份信息,并将该身份信息作为目标图像中该标记物的身份信息。预先存储的标记物的身份信息可包括标记物的各种信息,如标记物中各个特征点的物理坐标、标记物所设置的装置主体的信息等。对于目标图像中的第一连通域,以预存储标记物的包围关系中对应的第一连通域的身份信息作为其身份信息,获得该第一连通域对应的标记物的身份信息,从而可以获得目标图像中各个标记物中特征点的物理坐标、对应的交互装置等所需要的信息。
步骤S124,处理器根据目标图像的标记物信息及标记物的身份信息,确定对标记物对应的交互装置采用的跟踪方法。
在一个实施例中,处理器可根据标记物的身份信息,判断目标图像中的标记物是共面还是不共面,当标记物共面时,可采用相应的平面定位跟踪方法;当标记物不共面时,可采用相应的立体定位跟踪方法。
在标记物的身份信息中,包括各种对交互装置进行识别跟踪的需要信息。如标记物的物理坐标;设置标记物的交互装置为哪一个交互装置,标记物之间是否共面,同一个标记物的各个特征点之间是否共面等。另外标记物是否共面,可以是基于同一个交互装置进行判断。当目标图像中的标记物共面时,则可以采用平面定位跟踪方法,当目标图像中的标记物不共面,则可以采用立体跟踪方法。在一个实施例中,各个标记物之间是否共面可以通过各个标记物的物理坐标进行计算,或者根据对应的预存储标记物之间的共面信息判断。
步骤S126,处理器根据相应的跟踪方法,获取交互装置与图像采集装置之间的位置及姿态信息。
当目标图像中的标记物共面时,则可以采用平面定位跟踪方法,其中,标记物共面可指的是目标图像内的所有特征点共面,即所有特征点位于同一个平面上。在一些实施方式中,特征点共面的目标图像可以是包含有上述实施例中平面标记物体的标记面的图像;当采集的图像中的交互装置包括多面标记结构体时,特征点共面的目标图像还可以是包含有仅采集到多面标记结构体的某一个标记面的图像。目标图像为图像采集装置采集的具有交互装置的图像,该目标图像内包括多个特征点的信息。其中,目标图像内的特征点可以是交互装置内的所有特征点,也可以是交互装置内的所有特征点中的部分特征点。
进一步地,处理器可以从目标图像内的所有特征点中任意选取特定数量的特征点作为目标特征点,用于确定图像采集装置(相当于头戴显示装置)与具有目标特征点的平面标记物体,或图像采集装置(相当于头戴显示装置)与具有目标特征点的多面标记结构体之间的真实的位置及姿态信息。
在一个实施例中,处理器获取目标图像后,可判断目标图像内是否存在包括有目标特征点的标记物。由于每个特征点是分布在标记物内的,通过检测目标图像内是否存在标记物,从而能够判断所采集的目标图像内是否存在特征点。
作为一种实施方式,处理器判断目标图像内是否存在标记物的方式可以是,将目标图 像内的标记物的图像与预先存储的交互装置上的所有标记物的图像匹配,当能够匹配到相似或相同的标记物时,可判定目标图像内存在标记物。当不能够匹配到相似或相同的标记物时,可判定目标图像内不存在标记物,处理器可重新获取采集的目标图像,直至判定目标图像存在标记物。
其中,处理器可以通过查找目标图像内轮廓与标记物的轮廓一致的区域确定目标图像内的标记物。以标记物为矩形为例,查找目标图像内所有轮廓为矩形的区域,作为待确认标记物,再将每个待确认标记物与预先存储的交互装置上所有标记物的图像匹配,当能够匹配到相似或相同的标记物时,可判定目标图像内存在标记物,当不能够匹配到相似或相同的标记物时,判定目标图像内不存在标记物。
当目标图像内存在标记物时,处理器可判断目标特征点的数量是否大于或等于预设值,其中,目标特征点可以是目标图像内的任意的特征点,由于,后续步骤中要根据目标特征点的像素坐标和物理坐标获取图像采集设备在物理坐标系内的六自由度信息,而在求解的过程中,需要一定数量的目标特征点来组建多个方程组,因此,需要目标图像内的目标特征点的数量大于或等于预设值,其中,预设值为用户设定的数值,于本申请实施例中,该预设值可为4。在一些实施方式中,大于或等于预设值的目标特征点可以是分布在一个标记物内,也可以分布在多个标记物内,只要目标图像内的特征点的数量大于或等于预设值即可。
图20为一个实施例中通过平面定位跟踪方法对交互装置进行跟踪定位的流程图。在一个实施例中,处理器通过平面定位跟踪方法获取交互装置与图像采集装置之间的位置及姿态信息,可以包括步骤S261至S263。
步骤S261,处理器获取目标图像内的目标特征点在该目标图像对应的图像坐标系内的像素坐标。
目标图像内的目标特征点的像素坐标是指该特征点在目标图像中的位置,每个目标特征点在目标图像中的像素坐标可以直接在图像采集设备对应拍摄的图像中获得。例如,如图21a所示,以交互装置为第一标记板为例,I1为目标图像,图像坐标系为uov,其中,u的方向可以是目标图像中的像素矩阵的行方向,v的方向可以是目标图像中的像素矩阵的列方向,而图像坐标系中的原点o的位置可以选择目标图像的一个角点,比如最左上角或最左下角的点,由此,每个特征点在图像坐标系内的像素坐标就能够确定。例如,图21a中的特征点221a的像素坐标为(u
a,v
a)。
在一些实施方式中,当图像采集装置不能够满足使用标准,即所拍摄的图像存在畸变时,处理器需要对该目标图像做去畸变处理,其中,图像畸变是指成像过程中所产生的图像像元的几何位置相对于参照系统(地面实际位置或地形图)发生的挤压、伸展、偏移和扭曲等变形,使图像的几何位置、尺寸、形状、方位等发生改变。常见的畸变包括径向畸变、偏心畸变和薄棱镜畸变。根据图像采集装置的畸变参数和畸变模型对目标图像去畸变处理。处理器通过对目标图像进行去畸变处理,以去除目标图像中的畸变点,再将经畸变处理之后的目标图像作为本次获取的目标图像,获取每个目标特征点在目标图像对应的图像坐标系内的像素坐标。
步骤S263,处理器根据目标图像内的目标特征点的像素坐标和预先获取的目标特征点对应的物理坐标,获取图像采集装置与交互装置之间的位置及姿态信息。
物理坐标为预先获取的目标特征点在交互装置对应的物理坐标系内的坐标,目标特征点的物理坐标即为该目标特征点在对应交互装置上的真实位置。各个特征点的物理坐标可以预先获取,作为一种实施方式,多个特征点和多个标记物设置在交互装置的标记面上,选择标记面上的某一个点作为原点,建立物理坐标系。将标记面作为物理坐标系的XOY平面,XOY坐标系的原点位于标记面内。
如图21b所示,以第一标记板为矩形板为例,以标记板的标记面的一个角点作为原点O,以标记面的长度方向为X轴,以标记面的宽度方向为Y轴,垂直于标记面的方向为Z 轴,建立物理坐标系,每个特征点到X轴和Y轴的距离都可以获得,由此,就能够确定每个特征点在物理坐标系内的物理坐标,例如,图21b中的特征点221a的物理坐标为(X
a,Y
a,Z
a)。其中,Z
a等于0。
处理器获取到目标图像中所有目标特征点的像素坐标和物理坐标之后,可根据每个标记物内的所有目标特征点的像素坐标和物理坐标,获取图像采集装置与该标记物之间的位置及姿态信息。在一个实施例中,处理器可根据每个目标特征点的像素坐标、物理坐标和预先获取的图像采集装置的内参数,获取图像坐标系与物理坐标系之间的映射参数。
图像坐标系与物理坐标系之间的关系为:
其中,(u,v)为特征点在目标图像的图像坐标系中的像素坐标,(X,Y,Z)为特征点在物理坐标系的物理坐标,则将Z设为0,物理坐标系下的物理坐标为(X,Y,0)。
因此,将所获取的多个目标特征点的像素坐标和物理坐标,以及图像采集装置的内参数,带入上式(2),就能够获取H,即图像坐标系与物理坐标系之间的映射参数。
处理器可根据映射参数获取图像采集装置的相机坐标系与物理坐标系之间的旋转参数和平移参数,作为一种实施方式,可以根据SVD算法获取相机坐标系与物理坐标系之间的旋转参数和平移参数。
将上述单应性矩阵H做奇异值分解,得到下式:
H=UΛV
T (3),
则可以得到两个正交矩阵U和V,以及一个对角矩阵Λ。其中,对角矩阵Λ包含单应性矩阵H的奇异值。因此,也可以将这个对角矩阵当作单应性矩阵H,则可以将上式(3)写成:
当矩阵H被分解成对角矩阵时,就能够计算出旋转矩阵R和平移矩阵T。t
Λ可以在由上述公式(4)分离出来的三个向量方程中被消除,由于R
Λ是一个正交矩阵,则可以通过一个新的方程组线性求解法向量n内各参数,其中,该方程组将该法向量n内各参数与单 应性矩阵H的奇异值相关联。
通过上述分解算法,可以得到上述三个未知量的8个不同的解式,其中,该三个未知量为:{R
Λ,t
Λ,n
Λ}。然后,假设矩阵Λ的分解完成,则为了获取最终的分解元素,我们只需要使用下面的表达式:
由此,就可以求解处R和T,其中,R为图像采集装置的相机坐标系与物理坐标系之间的旋转参数,T为图像采集装置的相机坐标系与物理坐标系之间的平移参数。
可将旋转参数和平移参数作为图像采集装置与标记板之间的位置及姿态信息。其中,旋转参数表示相机坐标系与物理坐标系之间的旋转状态,也即图像采集装置在物理坐标系内,与物理坐标系的各坐标轴的转动自由度。其中,平移参数表示相机坐标系与物理坐标系之间的移动状态,也即图像采集装置在物理坐标系内,与物理坐标系的各坐标轴的移动自由度。则旋转参数和平移参数即为图像采集设备在物理坐标系内的六自由信息,能够表示图像采集设备在物理坐标系内的转动和移动状态,也即能够得到图像采集设备的视野与物理坐标系内的各坐标轴之间的角度和距离等。
在一个实施例中,在步骤S263之前,上述方法还可包括获取目标特征点的物理坐标。如图22所示,处理器获取目标特征点的物理坐标,包括步骤S631至S635。
步骤S631:处理器确定每个特征点在预设标记物模型中所对应的模型特征点。
处理器可确定目标特征点与预设标记物模型中的模型特征点的对应关系,其中,预设标记物模型为预先存储的包含有标记物信息的标准图像,标记物信息可包括标记物中各个特征点的物理坐标。处理器通过确定目标特征点与预设标记物模型中的模型特征点的对应关系,可根据预设标记物模型中模型特征点的物理坐标获取对应目标特征点的物理坐标。
在一个实施例中,处理器可获取目标图像对应的图像坐标系与预设标记物模型之间的映射参数,并根据映射参数确定目标特征点与预设标记物模型中的模型特征点的对应关系。处理器可先获取目标图像中标记物内的特征点的像素坐标,并根据目标图像内各个特征点的像素坐标,获取目标图像内每个子标记物的质心。在目标图像中,每个子标记物包括一个或多个特征点,一个子标记物的多个特征点对应存在一个质心,即为该子标记物的质心。处理器可以根据每个子标记物包括的特征点在目标图像中的像素坐标,计算得到每个子标记物对应的质心的坐标。质心的具体计算方式在本申请实施例中并不限定,如可以是根据权重计算方法进行计算。
在一个实施例中,处理器可判断目标图像中子标记物的质心是否满足第一预设条件,其中,该第一预设条件可以根据实际需求确定。作为一种实施方式,第一预设条件可以是,目标图像中子标记物或质心的数量达到预设数量。由于在计算映射参数时最少需要4个对应点,因此,该预设数量可以是4。当目标图像中子标记物的质心不满足第一预设条件时,处理器可重新获取目标图像。
当目标图像中子标记物的质心满足第一预设条件时,处理器可根据目标图像中子标记物内的特征点,在该子标记物内扩展预设个数的新质心,从而可通过扩展标记物中质心的数量,以获得更准确的映射参数。作为一种实施方式,处理器可以目标图像中子标记物的质心作为坐标原点建立坐标系,该子标记物可以是选择的任意一个进行质心扩展的子标记物。将该质心对应的子标记物中,满足第三预设条件的特征点位移到以坐标原点为对称中心的位置,根据位移后子标记物对应的各个特征点获取新的质心,其中,第三预设条件可包括在建立的坐标系中横坐标小于零、横坐标大于零、纵坐标小于零以及纵坐标大于零中的任意一个,不同的第三预设条件可对应获取一个新的质心。
处理器在目标图像中选取一个质心,作为坐标原点建立坐标系。以图23为例,如图23的(a)所示,目标图像中的特征点a、b、c、d为同一个子标记物包括的特征点,特征点a、b、c、d构成一个子标记物,该坐标系原点o为特征点a、b、c、d的质心o。以横坐标小于零作为第三预设条件,将该坐标系中横坐标小于零的特征点a、b位移到以坐标原点为对称中心的对称位置,即,将特征点a、b的横纵坐标均乘以-1后所在位置,得到结果如图23的(b)所示。在位移后,质心o对应的各个特征点对应存在一个新的质心,即以位移后的a、b以及c、d所在位置共同计算一个质心o’,该质心o’即为一个新的质心。以横坐标大于零作为第三预设条件,也可以获得一个新的质心。即,将该坐标系中横坐标大于零的特征点c、d位移到以坐标原点为对称中心的位置,即,将特征点c、d的横纵坐标均乘以-1后所在位置,得到结果如图23的(c)所示。在位移后,质心o对应的各个特征点对应存在一个新的质心o”,即以位移后的a、b以及c、d所在位置共同计算一个质心o”,该质心o”即为一个新的质心。可以理解的,每次位移用于计算新的质心,并不改变目标图像中各个特征点的位置。
在一个实施例中,对于一个子标记物,例如图23(a)所示的子标记物,可分别以横坐标小于零、横坐标大于零、纵坐标小于零以及纵坐标大于零作为第三预设条件,在不同第三预设条件下,可以分别获得一个新的质心,对于每一个子标记物,可以扩展获得4个新的质心。当目标图像中包括标记物的N个子标记物时,则可以获得4*N个新的质心。
在一个实施例中,建立的坐标系并不限定为图23所示的二维坐标系,也可以包括三维坐标系或者其他更多维数的坐标系,或者是包括更多象限的坐标系。若建立的坐标系为多维的坐标系,则在获取特征点以坐标原点作为对称中心的对称点时,特征点对应各个坐标的坐标值均乘以-1可以得到其关于坐标原点的对称点。作为一种实施方式,可以根据需求扩展的预设个数的新的质心,该预设个数可以并不限定。
处理器可基于扩展后各个质心的像素坐标、物理坐标以及预先获取的图像采集装置的内参数,获取目标图像对应的图像坐标系与预设标记物模型之间的映射参数。处理器根据图像中各个质心计算图像坐标系与预设标记物模型之间的映射参数,该映射参数可以是图像坐标系中各个点映射到预设标记物模型所在坐标系内的参数,如平面单应性矩阵。其中,用于计算的各个质心包括扩展前的原始质心以及扩展获得的新的质心。质心的物理坐标为预先获取的、质心在标记物对应的物理坐标系内的坐标,该物理坐标系的坐标原点可以是设置于标记物所在平面标记物体或多面标记结构体上。
在一个实施例中,预设标记物模型包括标记物中各个特征点的物理坐标,通过预设标记物模型中各个模型特征点的物理坐标可以计算获得各个子标记物的质心的物理坐标。处理器可在预设标记物模型中,以目标图像扩展质心的对应方式扩展新的质心,预设标记物模型中扩展的新的质心与目标图像中扩展的新的质心一一对应。
处理器可预先获取预设标记物模型中的子标记物与目标图像中的子标记物的一一对应关系。在预设标记物模型中,包括与目标图像中子标记物对应的子标记物。处理器预先获取预设标记物模型中子标记物与目标图像中子标记物的对应关系,其具体获取方式在本申请实施例中并不限定,如,标记物中各个子标记物对应的特征点的形状不同,根据形状确定预设标记物模型中子标记物与目标图像中子标记物的对应关系;又如,标记物中各个子标记物包括的特征点数量不同,根据特征点的数量确定预设标记物模型中子标记物与目标图像中子标记物的对应关系等。
对预设标记物模型进行质心扩展,扩展方式与目标图像中质心扩展相同。也就是说,在预设标记物模型中,以与目标图像中进行质心扩展的质心对应的质心作为坐标原点建立坐标系。其中,目标图像与预设标记物模型中相互对应的质心,分别为目标图像与预设标记物模型相互对应的子标记物的质心。将预设标记物模型中作为坐标原点的质心对应的模型特征点中,满足第三预设条件的模型特征点位移到以坐标原点为对称中心的位置,根据位移后该质心对应的各个模型特征点获取新的质心。其中,该第三预设条件与目标图像中 进行质心扩展的第三预设条件相同,获得的新的质心与目标图像中扩展的新的质心对应。
以图24为例,其中,图24(a)为预设标记物模型中与图23(a)所示的子标记物对应的子标记物,其中A、B、C、D为该子标记物中的模型特征点,以A、B、C、D的质心m作为坐标原点建立坐标系。以在该坐标系内横坐标小于零作为第三预设条件,将该坐标系中横坐标小于零的模型特征点A、B位移到以坐标原点m为对称中心的位置,即,将模型特征点A、B的横纵坐标均乘以-1后所在位置,得到结果如图24中的(b)所示。在位移后,质心m对应的各个模型特征点对应存在一个新的质心,即以位移后的A、B以及C、D所在位置共同计算一个质心m’,该质心m’即为该预设标记物模型中获得的一个新的质心,该新的质心m’与目标图像获得的新的质心o’对应。以横坐标大于零作为第三预设条件,将该坐标系中横坐标大于零的模型特征点C、D位移到以坐标原点m为对称中心的位置,即,将模型特征点C、D的横纵坐标均乘以-1后所在位置,得到结果如图24中的(c)所示。位移后,质心m对应的各个模型特征点对应存在一个新的质心,即以位移后的C、D及A、B所在位置共同计算一个质心m”,该质心m”即为该预设标记物模型中获得的一个新的质心,该新的质心m”与目标图像获得的新的质心o”对应。处理器可以获得预设标记物模型中分别与目标图像的新的质心一一对应的新的质心。
处理器可根据预设标记物模型中各个模型特征点的物理坐标计算预设标记物模型中每个质心的物理坐标。预先存储有预设标记物模型的各个模型特征点的物理坐标,根据各个模型特征点的物理坐标,可以计算各个质心的物理坐标。其中,计算的质心包括扩展前的原始质心以及扩展后的新的质心。质心计算方法在本申请实施例中并不限定,如采用权重的计算方式进行计算。处理器可根据目标图像中质心与预设标记物中质心的对应关系,将预设标记物模型中质心的物理坐标作为目标图像中对应质心的物理坐标,从而,获得目标图像中各个质心的物理坐标。例如,将图24中质心m的物理坐标作为与其对应的图23中质心o的物理坐标。
处理器根据目标图像中各个质心的像素坐标、物理坐标以及预先获取的所述图像采集装置的内参数,可以计算获得目标图像对应的图像坐标系与预设标记物模型之间的映射参数。在一个实施例中,图像坐标与物理坐标系之间的关系可如上述实施例中的式(1)所示,将上述实施例中的式(1)转化为式(2)后,可将获取的多个质心的像素坐标和物理坐标,以及图像采集装置的内参数,带入上述实施例中的式(2),计算得到H,即图像坐标系与物理坐标系之间的映射参数。
由于预设标记物模型根据实际标记物建立,或者是根据标记物所在平面标记物体或多面标记结构体建立,预设标记物模型的坐标系与标记物对应的物理坐标系对应,各个特征点在预设标记物模型的坐标系内的坐标与物理坐标相同,因此,可以根据各个质心的像素坐标、物理坐标以及预先获取的所述图像采集装置的内参数,获得目标图像对应的图像坐标系与预设标记物模型之间的映射参数。
处理器获取目标图像对应的图像坐标系与预设标记物模型之间的映射参数后,可根据映射参数将目标图像中各个特征点映射到预设标记物模型所在坐标系,从而可以获得目标图像中各个特征点与预设标记物模型中各个模型特征点的对应关系,也就是说,可以获得目标图像中各个特征点在预设标记物模型中对应的模型特征点。
在一个实施例中,处理器获取映射参数后,可判断是否满足第二预设条件,当满足第二预设条件时,则可根据映射参数获取目标图像中各个特征点与预设标记物模型中各个模型特征点的对应关系,当不满足第二预设条件时,可继续对目标图像进行质心扩展,获取更多的质心,用获取到的更多的质心再次计算更精确的映射参数,每次获取的新的质心的个数在本申请实施例中并不限定。
作为一种实施方式,该第二预设条件可以是目标图像中特征点与预设标记物模型中模型特征点之间的匹配误差满足预设的精度要求。处理器可根据映射参数将目标图像中每个特征点映射到预设标记物模型的坐标系内,以获取目标图像中每个特征点在预设标记物模 型的坐标系内的坐标。在预设标记物模型的坐标系中,当目标图像的特征点与预设标记物模型的模型特征点之间的匹配误差小于预设误差阈值时,判定满足第二预设条件。处理器进行判定的方式,可以是在预设标记物模型的坐标系中,计算目标图像的每个特征点与预设标记物模型的模型特征点之间的距离,目标图像的特征点与多个模型特征点对应的最小距离为目标图像中该特征点的匹配误差。当目标图像中各个特征点与模型特征点之间的匹配误差是否小于预设误差阈值,处理器可判定满足第二预设条件;或是当目标图像中有预设个数的特征点对应的匹配误差小于预设误差阈值,处理器可判定满足第二预设条件,其中,该预设个数并不限定。
作为一种实施方式,该第二预设条件可以是目标图像中特征点与预设标记物模型的模型特征点之间的匹配误差不再减小。处理器可根据多次扩展的质心计算获取映射参数,根据多次获取的映射参数将目标图像中每个特征点映射到预设标记物模型的坐标系内,并获取每次映射中目标图像的特征点与模型特征点之间的匹配误差。当目标图像中特征点与模型特征点之间的匹配误差不再减小,则处理器可判定满足第二预设条件。
作为一种实施方式,该第二预设条件可以是目标图像内扩展新的质心的次数达到预设次数,目标图像中每扩展一个新的质心,则可作为一次扩展。当目标图像内扩展新的质心的次数达到预设次数,处理器可判定满足第二预设条件。
作为一种实施方式,该第二预设条件可以是目标图像内扩展的新的质心个数达到预设的个数。当目标图像内扩展的新的质心个数达到预设的个数时,则处理器可判定满足第二预设条件,该预设的个数的具体值在本申请实施例中并不限定。
第二预设条件具体为何种条件在本申请实施例中并不限定,也可以以上述多种实施方式相结合,如上述多种实施方式中,同时作为第二预设条件。
步骤S633:处理器查找预设标记物模型中,每个模型特征点在交互装置对应的物理坐标系内的物理坐标。
步骤S635:处理器将每个目标特征点对应的模型特征点的物理坐标作为该目标特征点在交互装置对应的物理坐标系内的物理坐标。
处理器可根据映射参数将目标图像中每个特征点映射到预设标记物模型的坐标系内,以获取目标图像中每个特征点在预设标记物模型的坐标系内的坐标。在一个实施例中,可将在预设标记物模型的坐标系中,与目标图像中每个特征点的坐标距离最近的预设标记物模型的模型特征点,作为目标图像中该特征点在预设标记物模型中对应的模型特征点。
以图25为例进行说明,如图25所示,图25a中包括在图像坐标系内各个特征点e、f、g,处理器可将根据可映射参数H计算目标图像中各个特征点在预设标记物模型的坐标系内的坐标,将特征点e、f、g映射到预设标记物模型的坐标系内,得到映射后的目标特征点e’、f’、g’,如图25b所示。在图25b中,E、F、G为在预设标记物模型中与e、f、g形成的子标记物对应的标记物中的特征点。处理器可分别计算e’到E、F、G三个模型特征点的距离,e’到E的距离最小,则可以获得目标图像中特征点e’在预设标记物模型中对应模型特征点E;分别计算f’到E、F、G三个模型特征点的距离,f’到F的距离最小,则可以获得目标图像中特征点f’在预设标记物模型中对应模型特征点F;分别计算g’到E、F、G三个模型特征点的距离,g’到G的距离最小,则可以获得目标图像中特征点g’在预设标记物模型中对应的模型特征点G。
处理器确定目标图像中各个特征点与预设标记物模型中的模型特征点的对应关系,可查找预设标记物模型中,每个模型特征点在交互装置对应的物理坐标系内的物理坐标,并根据预设标记物模型中模型特征点的物理坐标获取目标图像中对应的特征点的物理坐标,在一个实施例中,可将模型特征点的物理坐标作为目标图像中对应的特征点的物理坐标。
图26为一个实施例中通过立体跟踪方法对交互装置进行跟踪定位的流程图。在一个实施例中,如图26所示,处理器通过立体跟踪方法获取交互装置与图像采集装置之间的位置及姿态信息,可以包括步骤S2610至S2620。
步骤S2610,处理器获取目标图像内的目标特征点在该目标图像对应的图像坐标系内的像素坐标。
处理器可获取图像采集装置采集的具有交互装置的目标图像,目标图像内包括对应交互装置内至少分布在两个面上的目标特征点。目标图像内的特征点分布在至少两个平面上,即图像采集装置采集的是标记物在至少两个平面上的交互装置。作为一种实施方式,目标图像可以是图像采集装置采集的包含有多面标记结构体的至少两个面的特征点的图像。
如图27所示,I2为目标图像,图像坐标系为uov,其中,u的方向可以是目标图像中的像素矩阵的行方向,v的方向可以是目标图像中的像素矩阵的列方向,而图像坐标系中的原点o的位置可以选择目标图像的一个角点,例如,最左上角或最左下角的点,由此,每个特征点在图像坐标系内的像素坐标就能够确定。例如,图27中的特征点341a的像素坐标为(u
a,v
a)。
步骤S2620,处理器根据目标图像内的目标特征点的像素坐标和预先获取的目标特征点对应的物理坐标,获取图像采集装置与交互装置之间的位置及姿态信息。
各个目标特征点的物理坐标可以预先获取,多个目标特征点和多个标记物设置在交互装置的不同的标记面上,可选择其中一个标记面上的某一个点作为原点,建立物理坐标系。作为一种实施方式,如图28所示,以二十六面标记结构体为例,以交互装置的一个矩形子表面的一个角点作为原点O,建立物理坐标系XYZ,则每个特征点到X轴、Y轴和Z轴的距离都可以测得,由此,就能够确定每个特征点在XOY坐标系内的物理坐标,例如,图28中的特征点341a的物理坐标为(X
a,Y
a,Z
a)。
在一个实施例中,处理器可获取每个目标特征点在交互装置对应的物理坐标系内的物理坐标,获取物理坐标的方式可参照上述实施例中的步骤S631至S635的描述,在此不再赘述。
在获取到目标图像中所有目标特征点的像素坐标和物理坐标之后,可根据每个标记物内的所有目标特征点的像素坐标和物理坐标,获取图像采集装置与该交互装置之间的位置及姿态信息。处理器可先根据每个目标特征点的像素坐标、物理坐标和预先获取的图像采集装置的内参数,获取图像坐标系与物理坐标系之间的映射参数。
在一个实施例中,图像坐标系与物理坐标系之间的关系可如上述实施例中的式(1)所示。可将上述实施例中的式(1)转换为上述实施例中的式(2),将所获取的多个目标特征点的像素坐标和物理坐标,以及图像采集装置的内参数,带入上述实施例中的式(2),就能够获取H,即图像坐标系与物理坐标系之间的映射参数。再根据映射参数获取图像采集装置的相机坐标系与物理坐标系之间的旋转参数和平移参数。作为一种实施方式,可以根据SVD算法,将单应性矩阵H做奇异值分解,得到上述实施例中的式(3),并将上述实施例中的式(3)转化为式(4),并通过分解算法得到式(5),求解得到旋转矩阵R和平移矩阵T,其中,R为图像采集装置的相机坐标系与所述物理坐标系之间的旋转参数,T为图像采集装置的相机坐标系与所述物理坐标系之间的平移参数。
处理器可将旋转参数和平移参数作为图像采集装置与交互装置之间的位置及姿态信息,此部分可参考上述实施例中平面定位跟踪方法的描述,在此不再赘述。
步骤S130:处理器根据位置及姿态信息确定与交互装置对应的虚拟场景。
处理器根据交互装置的位置及姿态信息可确定与交互装置对应的显示内容,可通过头戴显示装置的显示装置及光学组件将显示内容叠加显示在真实场景中,以使用户通过佩戴的头戴显示装置观察到虚拟场景。
在一个实施例中,本申请实施例还提供一种电子设备,包括存储器及处理器,存储器中存储有计算机程序,该计算机程序可被该处理器执行,以实现上述实施例中所描述的方法。
在一个实施例中,本申请实施例还提供一种计算机可读存储介质,计算机可读存储介质存储有计算机程序,该计算机程序可被处理器执行以实现上述实施例中所描述的方法。 计算机可读存储介质可以是诸如闪存、EEPROM(电可擦除可编程只读存储器)、EPROM、硬盘或者ROM之类的电子存储器。可选地,计算机可读存储介质包括非易失性计算机可读介质(non-transitory computer-readable storage medium)。计算机可读存储介质具有执行上述方法中的任何方法步骤的计算机程序的存储空间。这些计算机程序可以从一个或者多个计算机程序产品中读出或者写入到这一个或者多个计算机程序产品中。计算机程序可以例如以适当形式进行压缩。
以上所述仅为本申请的优选实施例而已,并不用于限制本申请,对于本领域的技术人员来说,本申请可以有各种更改和变化。凡在本申请的精神和原则之内,所作的任何修改、等同替换、改进等,均应包含在本申请的保护范围之内。应注意到:相似的标号和字母在下面的附图中表示类似项,因此,一旦某一项在一个附图中被定义,则在随后的附图中不需要对其进行进一步定义和解释。
尽管上面已经示出和描述了本申请的实施例,可以理解的是,上述实施例是示例性的,不能理解为对本申请的限制,本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。
Claims (50)
- 一种图像处理方法,包括:获取图像采集装置采集的目标图像,所述目标图像包含设置在交互装置上的标记物,所述交互装置位于真实场景中;根据所述目标图像确定所述交互装置在所述真实场景内的位置及姿态信息;根据所述位置及姿态信息确定与所述交互装置对应的虚拟场景。
- 根据权利要求1所述的方法,其特征在于,所述方法还包括:获取所述位置及姿态信息的变化;根据所述位置及姿态信息的变化更新所述虚拟场景,以使所述虚拟场景跟随所述位置及姿态信息的变化而相应变化。
- 根据权利要求1所述的方法,其特征在于,所述交互装置为多个;所述根据所述位置及姿态信息确定与所述交互装置对应的虚拟场景,包括:根据每个所述交互装置的位置及姿态信息,确定各个交互装置之间的位置及姿态信息;根据所述交互装置的姿态信息,以及各个交互装置之间的姿态信息确定所述交互装置对应的虚拟场景。
- 根据权利要求1所述的方法,其特征在于,所述位置及姿态信息为所述交互装置与所述图像采集装置之间的位置及姿态信息;所述根据所述目标图像确定所述交互装置在所述真实场景内的位置及姿态信息,包括:确认所述目标图像中标记物的身份信息;根据所述目标图像的标记物信息及所述标记物的身份信息,确定与所述标记物对应的交互装置采用的跟踪方法;根据相应的跟踪方法,获取所述交互装置与所述图像采集装置之间的位置及姿态信息。
- 根据权利要求4所述的方法,其特征在于,在所述确认所述目标图像中标记物的身份信息之前,所述方法还包括:获取连续多帧目标图像中除首帧目标图像外的当前帧目标图像对应的第一阈值图像,所述第一阈值图像为对历史帧目标图像进行处理后得到且与当前帧目标图像分辨率相同的灰度图像;对所述当前帧目标图像的每一个像素点,以所述第一阈值图像中对应位置的像素点作为二值化阈值,将所述当前帧目标图像二值化。
- 根据权利要求4所述的方法,其特征在于,所述确认所述目标图像中标记物的身份信息,包括:获取所述目标图像中多个连通域之间的包围关系;根据所述包围关系,以及预先存储的标记物的特征,确定所述目标图像中标记物的身份信息为对应的预存储标记物的身份信息。
- 根据权利要求4所述的方法,其特征在于,所述确定与所述标记物对应的交互装置采用的跟踪方法,包括:当所述标记物为平面标记物时,采用相应的平面定位跟踪方法;当所述标记物为立体标记物,且立体标记物属于同一平面,采用相应的平面定位跟踪方法;当所述立体标记物不属于同一平面,采用相应的立体定位跟踪方法。
- 根据权利要求7所述的方法,其特征在于,所述标记物包括特征点,所述目标图像内包括对应交互装置内的多个共面的目标特征点;根据所述平面定位跟踪方法,获取所述交互装置与所述图像采集装置之间的位置及姿态信息,包括:获取所述目标图像内的目标特征点在所述目标图像对应的图像坐标系内的像素坐标;根据所述目标特征点的像素坐标和预先获取的所述目标特征点对应的物理坐标,获取所述图像采集装置与所述交互装置之间的位置及姿态信息,其中,所述物理坐标为预先获取的目标特征点在所述交互装置对应的物理坐标系内的坐标。
- 根据权利要求8所述的方法,其特征在于,所述方法还包括:获取所述目标特征点对应的物理坐标;所述获取所述目标特征点对应的物理坐标,包括:确定所述目标特征点在预设标记物模型中所对应的模型特征点;查找所述预设标记物模型中的模型特征点在所述交互装置对应的物理坐标系内的物理坐标;将所述目标特征点所对应的模型特征点的物理坐标作为所述目标特征点在所述物理坐标系内的物理坐标。
- 根据权利要求9所述的方法,其特征在于,所述确定所述目标特征点在预先获取的预设标记物模型中所对应的模型特征点,包括:根据目标图像内各个特征点的像素坐标,获取所述目标图像内每个子标记物的质心;当获得的子标记物的质心满足第一预设条件时,根据所述子标记物内的特征点,在所述子标记物内扩展预设个数的新的质心;根据所述目标图像中各个质心的像素坐标、物理坐标以及预先获取的所述图像采集装置的内参数,获取所述目标图像对应的图像坐标系与预设标记物模型之间的映射参数;根据所述映射参数获取所述目标图像中各个特征点与所述预设标记物模型中各个模型特征点的对应关系。
- 根据权利要求8所述的方法,其特征在于,所述根据所述目标特征点的像素坐标和预先获取的所述目标特征点对应的物理坐标,获取所述图像采集装置与所述交互装置之间的位置及姿态信息,包括:根据所述目标特征点的像素坐标、物理坐标和预先获取的所述图像采集装置的内参数,获取所述图像坐标系与所述物理坐标系之间的映射参数;根据所述映射参数获取所述图像采集装置的相机坐标系与所述物理坐标系之间的旋转参数和平移参数;根据所述旋转参数和平移参数获取所述图像采集装置与所述交互装置之间的位置及姿态信息。
- 根据权利要求7所述的方法,其特征在于,所述标记物包括特征点,所述目标图像内包括对应交互装置内至少分布在两个面上的目标特征点;根据立体定位跟踪方法,获取所述交互装置与所述图像采集装置之间的位置及姿态信息,包括:获取目标图像内的目标特征点在该目标图像对应的图像坐标系内的像素坐标;根据所述目标特征点的像素坐标和预先获取的所述目标特征点的物理坐标,获取所述图像采集装置与所述交互装置之间的位置及姿态信息,其中,所述物理坐标为预先获取的所述目标特征点在所述交互装置对应的物理坐标系内的坐标。
- 一种图像处理方法,包括:获取连续多帧图像中除首帧图像外的当前帧图像对应的第一阈值图像,所述第一阈值图像为对历史帧图像进行处理后得到且与当前帧图像分辨率相同的灰度图像;对当前帧图像的每一个像素点,以所述第一阈值图像中对应位置的像素点作为二值化阈值,将当前帧图像二值化。
- 根据权利要求13所述的方法,其特征在于,所述获取连续多帧图像中除首帧图像外的当前帧图像对应的第一阈值图像,包括:获取对历史帧图像进行处理后的具有第一预设分辨率的第二阈值图像,所述第一预设分辨率低于当前帧图像的分辨率;对所述第二阈值图像进行升采样,获得与当前帧图像分辨率相同的第一阈值图像。
- 根据权利要求14所述的方法,其特征在于,所述获取对历史帧图像进行处理后的具有第一预设分辨率的第二阈值图像包括:对所述历史帧图像进行降采样,获得具有第二预设分辨率的降采样图像;根据所述降采样图像计算获取具有第二预设分辨率的第三阈值图像,若所述第二预设分辨率小于或等于所述第一预设分辨率,获得所述第二阈值图像,其中,根据所述降采样图像中每个像素点在预设窗口范围内的各个像素点的像素值,确定所述第三阈值图像中每个像素点的像素值。
- 根据权利要求14所述的方法,其特征在于,所述获取对历史帧图像进行处理后的具有第一预设分辨率的第二阈值图像包括:对所述历史帧图像进行降采样,获得具有第二预设分辨率的降采样图像;获取所述降采样图像的积分图;根据所述积分图计算获取具有第二预设分辨率的第三阈值图像,若所述第二预设分辨率小于等于所述第一预设分辨率,获得所述第二阈值图像,其中,根据所述积分图中每个像素点在预设窗口范围内的各个像素点的像素值,确定所述第三阈值图像中每个像素点的像素值。
- 根据权利要求15所述的方法,其特征在于,若所述第二预设分辨率大于所述第一预设分辨率,所述根据所述降采样图像计算获得具有第二预设分辨率的第三阈值图像之后,还包括:对所述第三阈值图像继续进行降采样,直到获得分辨率小于或等于所述第一预设分辨率的所述第二阈值图像。
- 一种图像处理方法,包括:获取包括标记物的目标图像;对所述目标图像进行处理,并获取所述目标图像中多个连通域之间的包围关系;根据所述目标图像中多个连通域之间的包围关系,以及预存储的标记物的特征,确定所述目标图像中标记物的身份信息为对应的预存储标记物的身份信息。
- 根据权利要求18所述的方法,其特征在于,所述对所述目标图像进行处理包括:将所述目标图像处理为二值化图像,使标记物中子标记物与除子标记物以外的部分具有区分度。
- 根据权利要求19所述的方法,其特征在于,所述预存储的标记物的特征包括:第一连通域包围的第二连通域的数量,每个第二连通域包围的第三连通域的数量;所述获取所述目标图像中多个连通域之间的包围关系,包括:确定标记物中包围其他连通域的连通域为第一连通域,确定第一连通域包围的连通域为第二连通域,确定第二连通域包围的连通域为第三连通域;获取每个第一连通域中包围的第二连通域的数量以及每个第二连通域包围的第三连通域的数量;所述确定所述目标图像中标记物的身份信息为对应的预存储标记物的身份信息包括:对于目标图像中每个第一连通域,在预先存储的标记物的特征信息中确定对应的第一连通域,其中,相互对应的第一连通域包围有相同数量的第二连通域、且包围的各个第二连通域所包围的第三连通域的数量一一对应。
- 根据权利要求20所述的方法,其特征在于,当多个预存储标记物中,只有一个第一连通域包围的第二连通域包围有第一数量的第三连通域,所述确定所述目标图像中标记物的身份信息为对应的预存储标记物的身份信息包括:当所述目标图像中存在一个第一连通域,该第一连通域包围的第二连通域包围有第一数量的第三连通域时,确定所述第一连通域对应的标记物的身份信息为所述第一数量的第三连通域对应的预存储标记物的身份信息。
- 根据权利要求20所述的方法,其特征在于,当多个预存储标记物中,只有一个第一连通域包围有第二数量的第二连通域,所述确定所述目标图像中标记物的身份信息为对应的预存储标记物的身份信息包括:当所述目标图像中存在一个第一连通域,该第一连通域包围有第二数量的第二连通域,确定该第一连通域对应的标记物的身份信息为所述第二数量的第二连通域对应的预存储标记物的身份信息。
- 根据权利要求20所述的方法,其特征在于,当多个预存储标记物中,只有一个预存储标记物包括依次包围的第三数量的连通域,所述确定所述目标图像中标记物的身份信息为对应的预存储标记物的身份信息包括:当所述目标图像中包括依次包围的第三数量的连通域时,确定该第三数量的连通域对应的标记物的身份信息为预存储的所述第三数量的连通域对应的标记物的身份信息。
- 根据权利要求20所述的方法,其特征在于,所述预存储的标记物的特征还包括第四连通域,所述第四连通域包围所述第一连通域;所述获取所述目标图像中多个连通域之间的包围关系,还包括:确定第四连通域包围的每个第一连通域对应一个标记物。
- 一种图像处理方法,包括:获取具有交互装置的目标图像,以及所述目标图像中交互装置内的特征点的像素坐标,所述交互装置包括多个子标记物,每一子标记物包括一个或多个特征点;获取所述目标图像内每个子标记物的质心;当所述目标图像中获得的子标记物的质心满足第一预设条件,根据所述目标图像内子标记物的特征点,在所述子标记物内扩展预设个数的新质心;基于扩展后各个质心的像素坐标、物理坐标以及预先获取的所述图像采集装置的内参数,获取所述目标图像与预设标记物模型之间的映射参数;基于所述映射参数获取所述目标图像中各个特征点与所述预设标记物模型中各个特征点的对应关系。
- 根据权利要求25所述的方法,其特征在于,所述第一预设条件为获得的质心的数量达到预设数量。
- 根据权利要求25所述的方法,其特征在于,所述根据所述目标图像内子标记物的特征点,在所述子标记物内扩展预设个数的新质心,包括:以目标图像中子标记物的质心作为坐标原点建立坐标系;以坐标原点为对称中心,将所述质心对应的子标记物中满足第三预设条件的特征点位移到相应位置,根据位移后该质心对应的各个目标特征点获取新质心,其中,所述第三预设条件包括在建立的坐标系中横坐标小于零、横坐标大于零、纵坐标小于零以及纵坐标大于零中的任意一个,每个不同的第三预设条件用于扩展一个新质心。
- 根据权利要求27所述的方法,其特征在于,所述方法还包括:获取所述质心的物理坐标;所述获取所述质心的物理坐标,包括:在所述预设标记物模型中以所述目标图像中扩展质心的对应方式扩展新质心,预设标记物模型中扩展的新的质心与目标图像中扩展的新的质心一一对应,其中,预先获取有所述预设标记物模型中的子标记物与所述目标图像中的子标记物的一一对应关系;根据所述预设标记物模型中各个特征点的物理坐标计算所述预设标记物模型中的每个质心的物理坐标;根据所述对应关系,将所述预设标记物模型中质心的物理坐标,作为所述目标图像中对应质心的物理坐标。
- 根据权利要求25所述的方法,其特征在于,在所述基于所述映射参数获取所述目标图像中各个特征点与所述预设标记物模型中各个特征点的对应关系之前,还包括:基于所述映射参数将目标图像中每个特征点映射到所述预设标记物模型的坐标系内,以获取目标图像中每个特征点在所述预设标记物模型的坐标系内的坐标;在所述预设标记物模型的坐标系中,当所述目标图像的特征点与预设标记物模型中的特征点满足第二预设条件时,执行所述根据所述映射参数获取所述目标图像中各个特征点与所述预设标记物模型中各个特征点的对应关系的步骤;当所述目标图像的特征点与预设标记物模型中的特征点不满足第二预设条件时,则再次执行在所述目标图像内扩展预设个数的新的质心的步骤。
- 根据权利要求29所述的方法,其特征在于,所述满足第二预设条件包括:在所述预设标记物模型的坐标系中,目标图像的特征点与预设标记物模型中的特征点之间的匹配误差小于预设误差阈值。
- 根据权利要求29所述的方法,其特征在于,所述满足第二预设条件包括:在所述目标图像内扩展新的质心的次数达到预设次数;或者是在所述目标图像内扩展的质心个数是否达到预设的个数。
- 根据权利要求25所述的方法,其特征在于,所述根据所述映射参数获取所述目标图像中各个特征点与所述预设标记物模型中各个特征点的对应关系,包括:根据所述映射参数将目标图像中每个特征点映射到所述预设标记物模型的坐标系内,以获取目标图像中每个特征点在所述预设标记物模型的坐标系内的坐标;将在所述预设标记物模型的坐标系中,与目标图像中每个特征点的坐标距离最近的特征点,作为目标图像中该特征点在预设标记物模型中对应的特征点。
- 根据权利要求25所述的方法,其特征在于,所述获取所述目标图像内各个目标标签的质心之前,还包括:对所述目标图像做去畸变处理,以去除所述目标图像中的畸变点;将经畸变处理之后的目标图像作为本次获取的目标图像。
- 一种图像处理方法,包括:获取具有标记物的目标图像,所述标记物分布在所述交互装置的一个面上或多个面上;确认所述目标图像中标记物的身份信息;根据目标图像的标记物信息及所述标记物的身份信息,确定对所述标记物对应的交互装置采用的跟踪方法;根据相应的跟踪方法,获取所述交互装置与所述图像采集装置之间的位置及姿态信息。
- 根据权利要求34所述的方法,其特征在于,所述确定对所述标记物对应的交互装置采用的跟踪方法,包括:当所述目标图像中的标记物共面时,采用相应的平面定位跟踪方法;当所述目标图像中的标记物不共面时,采用相应的立体定位跟踪方法。
- 一种图像处理方法,包括:获取图像采集装置采集的具有交互装置的目标图像,所述目标图像内包括所述交互装置内的多个共面的目标特征点;获取所述目标图像内的目标特征点在所述目标图像对应的图像坐标系内的像素坐标;根据所述目标特征点的像素坐标和预先获取的所述目标特征点对应的物理坐标,获取所述图像采集装置与所述交互装置之间的位置及姿态信息,其中,所述物理坐标为预先获取的目标特征点在所述交互装置对应的物理坐标系内的坐标。
- 根据权利要求36所述的方法,其特征在于,所述方法还包括:获取所述目标特征点对应的物理坐标;所述获取所述目标特征点对应的物理坐标,包括:确定所述目标特征点在预设标记物模型中所对应的模型特征点;查找所述预设标记物模型中的模型特征点在所述交互装置对应的物理坐标系内的物理坐标;将所述目标特征点所对应的模型特征点的物理坐标作为该目标特征点在所述交互装置对应的物理坐标系内的物理坐标。
- 根据权利要求37所述的方法,其特征在于,所述确定每个所述目标特征点在预先获取的预设标记物模型中所对应的模型特征点,包括:将所述目标特征点映射到所述预设标记物模型对应的坐标系内,以获取所述目标特征点在所述预设标记物模型对应的坐标系内的坐标;将在所述预设标记物模型对应的坐标系中,与所述目标特征点的坐标距离最近的模型特征点作为该目标特征点对应的模型特征点。
- 根据权利要求36所述的方法,其特征在于,所述根据所述目标图像内的目标特征点的像素坐标和预先获取的所述目标特征点对应的物理坐标,获取所述图像采集装置与所述交互装置之间的位置及姿态信息,包括:根据所述目标特征点的像素坐标、物理坐标和预先获取的所述图像采集装置的内参数,获取所述图像坐标系与所述物理坐标系之间的映射参数;根据所述映射参数获取所述图像采集装置的相机坐标系与所述物理坐标系之间的旋转参数和平移参数;根据所述旋转参数和平移参数获取所述图像采集装置与所述交互装置之间的位置及姿态信息。
- 根据权利要求36所述的方法,其特征在于,所述获取所述目标特征点在所述目标图像对应的图像坐标系内的像素坐标,包括:当所述目标特征点的数量大于预设值时,获取所述目标特征点在所述目标图像对应的图像坐标系内的像素坐标。
- 一种图像处理方法,包括:获取图像采集装置采集的具有交互装置的目标图像,所述目标图像内包括所述交互装置内至少分布在两个面上的目标特征点;获取所述目标图像内的目标特征点在所述目标图像对应的图像坐标系内的像素坐标;根据所述目标特征点的像素坐标和预先获取的所述目标特征点的物理坐标,获取所述图像采集装置与所述交互装置之间的位置及姿态信息,其中,所述物理坐标为预先获取的所述目标特征点在所述交互装置对应的物理坐标系内的坐标。
- 根据权利要求41任一所述的方法,其特征在于,根据所述目标特征点的像素坐标和预先获取的所述目标特征点的物理坐标,获取所述图像采集装置与所述交互装置之间的位置及姿态信息,包括:根据所述目标特征点的像素坐标、物理坐标和预先获取的所述图像采集装置的内参数,获取所述图像坐标系与所述物理坐标系之间的映射参数;根据所述映射参数获取所述图像采集装置的相机坐标系与所述物理坐标系之间的旋转参数和平移参数;根据所述旋转参数和平移参数获取所述交互装置与所述图像采集装置之间的位置及姿态信息。
- 一种计算机可读存储介质,存储有一个或多个计算机程序,所述一个或多个计算机程序被一个或多个处理器执行时,用于执行以下步骤:获取图像采集装置采集的目标图像,所述目标图像包含设置在交互装置上的标记物,所述交互装置位于真实场景中;根据所述目标图像确定所述交互装置在所述真实场景内的位置及姿态信息;根据所述位置及姿态信息确定与所述交互装置对应的虚拟场景。
- 一种计算机可读存储介质,存储有一个或多个计算机程序,所述一个或多个计算机程序被一个或多个处理器执行时,用于执行以下步骤:获取连续多帧图像中除首帧图像外的当前帧图像对应的第一阈值图像,所述第一阈值 图像为对历史帧图像进行处理后得到且与当前帧图像分辨率相同的灰度图像;对当前帧图像的每一个像素点,以所述第一阈值图像中对应位置的像素点作为二值化阈值,将当前帧图像二值化。
- 一种计算机可读存储介质,存储有一个或多个计算机程序,所述一个或多个计算机程序被一个或多个处理器执行时,用于执行以下步骤:获取包括标记物的目标图像;对所述目标图像进行处理,并获取所述目标图像中多个连通域之间的包围关系;根据所述目标图像中多个连通域之间的包围关系,以及预存储的标记物的特征,确定所述目标图像中标记物的身份信息为对应的预存储标记物的身份信息。
- 一种计算机可读存储介质,存储有一个或多个计算机程序,所述一个或多个计算机程序被一个或多个处理器执行时,用于执行以下步骤:获取具有交互装置的目标图像,以及所述目标图像中交互装置内的特征点的像素坐标,所述交互装置包括多个子标记物,每一子标记物包括一个或多个特征点;获取所述目标图像内每个子标记物的质心;当所述目标图像中获得的子标记物的质心满足第一预设条件,根据所述目标图像内子标记物的特征点,在所述子标记物内扩展预设个数的新质心;基于扩展后各个质心的像素坐标、物理坐标以及预先获取的所述图像采集装置的内参数,获取所述目标图像与预设标记物模型之间的映射参数;基于所述映射参数获取所述目标图像中各个特征点与所述预设标记物模型中各个特征点的对应关系。
- 一种电子设备,包括一个或多个处理器和存储器,所述存储器存储有一个或多个计算机程序,所述一个或多个计算机程序被所述一个或多个处理器执行时,用于执行以下步骤:获取图像采集装置采集的目标图像,所述目标图像包含设置在交互装置上的标记物,所述交互装置位于真实场景中;根据所述目标图像确定所述交互装置在所述真实场景内的位置及姿态信息;根据所述位置及姿态信息确定与所述交互装置对应的虚拟场景。
- 一种电子设备,包括一个或多个处理器和存储器,所述存储器存储有一个或多个计算机程序,所述一个或多个计算机程序被所述一个或多个处理器执行时,用于执行以下步骤:获取连续多帧图像中除首帧图像外的当前帧图像对应的第一阈值图像,所述第一阈值图像为对历史帧图像进行处理后得到且与当前帧图像分辨率相同的灰度图像;对当前帧图像的每一个像素点,以所述第一阈值图像中对应位置的像素点作为二值化阈值,将当前帧图像二值化。
- 一种电子设备,包括一个或多个处理器和存储器,所述存储器存储有一个或多个计算机程序,所述一个或多个计算机程序被所述一个或多个处理器执行时,用于执行以下步骤:获取包括标记物的目标图像;对所述目标图像进行处理,并获取所述目标图像中多个连通域之间的包围关系;根据所述目标图像中多个连通域之间的包围关系,以及预存储的标记物的特征,确定所述目标图像中标记物的身份信息为对应的预存储标记物的身份信息。
- 一种电子设备,包括一个或多个处理器和存储器,所述存储器存储有一个或多个计算机程序,所述一个或多个计算机程序被所述一个或多个处理器执行时,用于执行以下步骤:获取具有交互装置的目标图像,以及所述目标图像中交互装置内的特征点的像素坐标,所述交互装置包括多个子标记物,每一子标记物包括一个或多个特征点;获取所述目标图像内每个子标记物的质心;当所述目标图像中获得的子标记物的质心满足第一预设条件,根据所述目标图像内子标记物的特征点,在所述子标记物内扩展预设个数的新质心;基于扩展后各个质心的像素坐标、物理坐标以及预先获取的所述图像采集装置的内参数,获取所述目标图像与预设标记物模型之间的映射参数;基于所述映射参数获取所述目标图像中各个特征点与所述预设标记物模型中各个特征点的对应关系。
Applications Claiming Priority (14)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810119776.3 | 2018-02-06 | ||
CN201810119387.0 | 2018-02-06 | ||
CN201810119839.5A CN110119653A (zh) | 2018-02-06 | 2018-02-06 | 图像处理方法、装置及计算机可读介质 |
CN201810119323.0A CN110119194A (zh) | 2018-02-06 | 2018-02-06 | 虚拟场景处理方法、装置、交互系统、头戴显示装置、视觉交互装置及计算机可读介质 |
CN201810119839.5 | 2018-02-06 | ||
CN201810119323.0 | 2018-02-06 | ||
CN201810119868.1 | 2018-02-06 | ||
CN201810118639.8A CN110119190A (zh) | 2018-02-06 | 2018-02-06 | 定位方法、装置、识别跟踪系统及计算机可读介质 |
CN201810118639.8 | 2018-02-06 | ||
CN201810119776.3A CN110120099A (zh) | 2018-02-06 | 2018-02-06 | 定位方法、装置、识别跟踪系统及计算机可读介质 |
CN201810119854.X | 2018-02-06 | ||
CN201810119868.1A CN110120100B (zh) | 2018-02-06 | 2018-02-06 | 图像处理方法、装置及识别跟踪系统 |
CN201810119387.0A CN110120062B (zh) | 2018-02-06 | 2018-02-06 | 图像处理方法及装置 |
CN201810119854.XA CN110120060B (zh) | 2018-02-06 | 2018-02-06 | 标记物的识别方法、装置及识别跟踪系统 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019154169A1 true WO2019154169A1 (zh) | 2019-08-15 |
Family
ID=67549297
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/073578 WO2019154169A1 (zh) | 2018-02-06 | 2019-01-29 | 跟踪交互装置的方法、存储介质以及电子设备 |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2019154169A1 (zh) |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107194968A (zh) * | 2017-05-18 | 2017-09-22 | 腾讯科技(上海)有限公司 | 图像的识别跟踪方法、装置、智能终端和可读存储介质 |
CN107250891A (zh) * | 2015-02-13 | 2017-10-13 | Otoy公司 | 头戴式显示器与真实世界对象之间的相互通信 |
CN107450714A (zh) * | 2016-05-31 | 2017-12-08 | 大唐电信科技股份有限公司 | 基于增强现实和图像识别的人机交互支撑测试系统 |
-
2019
- 2019-01-29 WO PCT/CN2019/073578 patent/WO2019154169A1/zh active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107250891A (zh) * | 2015-02-13 | 2017-10-13 | Otoy公司 | 头戴式显示器与真实世界对象之间的相互通信 |
CN107450714A (zh) * | 2016-05-31 | 2017-12-08 | 大唐电信科技股份有限公司 | 基于增强现实和图像识别的人机交互支撑测试系统 |
CN107194968A (zh) * | 2017-05-18 | 2017-09-22 | 腾讯科技(上海)有限公司 | 图像的识别跟踪方法、装置、智能终端和可读存储介质 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110568447B (zh) | 视觉定位的方法、装置及计算机可读介质 | |
KR101666959B1 (ko) | 카메라로부터 획득한 영상에 대한 자동보정기능을 구비한 영상처리장치 및 그 방법 | |
EP3422955B1 (en) | System and method for assisted 3d scanning | |
TWI729995B (zh) | 基於場景之拍攝圖像而產生合併、融合三維點雲 | |
US7079679B2 (en) | Image processing apparatus | |
US8422777B2 (en) | Target and method of detecting, identifying, and determining 3-D pose of the target | |
CN109801374B (zh) | 一种通过多角度图像集重构三维模型的方法、介质及系统 | |
US10846844B1 (en) | Collaborative disparity decomposition | |
JP5075182B2 (ja) | 画像処理装置、画像処理方法、および、画像処理プログラム | |
KR102354299B1 (ko) | 단일 영상을 이용한 카메라 캘리브레이션 방법 및 이를 위한 장치 | |
CN110120099A (zh) | 定位方法、装置、识别跟踪系统及计算机可读介质 | |
EP3069100B1 (en) | 3d mapping device | |
CN113223135B (zh) | 一种基于特殊复合平面镜虚像成像的三维重建装置和方法 | |
CN110119190A (zh) | 定位方法、装置、识别跟踪系统及计算机可读介质 | |
CN112184793B (zh) | 深度数据的处理方法、装置及可读存储介质 | |
WO2015023483A1 (en) | 3d mapping device for modeling of imaged objects using camera position and pose to obtain accuracy with reduced processing requirements | |
CN108022265A (zh) | 红外相机位姿确定方法、设备及系统 | |
Wenzel et al. | High-resolution surface reconstruction from imagery for close range cultural Heritage applications | |
CN113505626A (zh) | 一种快速三维指纹采集方法与系统 | |
CN110120100B (zh) | 图像处理方法、装置及识别跟踪系统 | |
JP7432793B1 (ja) | 三次元点群に基づくマッピング方法、装置、チップ及びモジュール機器 | |
KR101781515B1 (ko) | 다면체를 이용한 카메라 칼리브레이션 시스템 및 방법 | |
McIlroy et al. | Kinectrack: 3d pose estimation using a projected dense dot pattern | |
CN112433641B (zh) | 一种多rgbd深度传感器的自动校准桌面道具互动系统的实现方法 | |
CN112070844A (zh) | 结构光系统的校准方法及装置、校准工具图、设备及介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19751942 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 17/12/2020) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19751942 Country of ref document: EP Kind code of ref document: A1 |