WO2017167282A1 - 一种目标跟踪方法及电子设备、计算机存储介质 - Google Patents
一种目标跟踪方法及电子设备、计算机存储介质 Download PDFInfo
- Publication number
- WO2017167282A1 WO2017167282A1 PCT/CN2017/079030 CN2017079030W WO2017167282A1 WO 2017167282 A1 WO2017167282 A1 WO 2017167282A1 CN 2017079030 W CN2017079030 W CN 2017079030W WO 2017167282 A1 WO2017167282 A1 WO 2017167282A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- tracking
- target
- electronic device
- target point
- camera
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 81
- 230000000007 visual effect Effects 0.000 claims abstract description 90
- 230000003287 optical effect Effects 0.000 claims description 18
- 238000004458 analytical method Methods 0.000 claims description 7
- 230000003993 interaction Effects 0.000 abstract description 19
- 230000008569 process Effects 0.000 description 41
- 230000008901 benefit Effects 0.000 description 11
- 230000000694 effects Effects 0.000 description 8
- 230000007774 longterm Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000013459 approach Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 230000007423 decrease Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 239000003086 colorant Substances 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 206010034719 Personality change Diseases 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20101—Interactive definition of point of interest, landmark or seed
Definitions
- the present invention relates to the field of visual tracking technologies, and in particular, to a target tracking method, an electronic device, and a computer storage medium.
- the visual tracking technology refers to detecting, extracting, recognizing and tracking the tracking targets in the image sequence, and obtaining the tracking target motion parameters, such as position, velocity, acceleration and motion trajectory, so as to carry out the next processing and analysis, and realize the pair. Track the behavioral understanding of the target to complete a higher level of inspection tasks.
- the main purpose of visual tracking technology is to imitate the motion perception function of the physiological vision system, and to give the machine the ability to recognize the motion of objects in the image sequence and their relationship, providing an important way for image sequence understanding.
- Visual tracking technology has broad application prospects, such as video surveillance, video analysis, video retrieval, video-based motion analysis and synthesis, and motion information-based identification.
- the interactive mode of the existing online learning visual tracking algorithm requires the user to specify the area where the tracking target is located in the video picture (the area is usually a rectangular area, and the user needs to draw the area through the frame operation), thereby Define a tracking template.
- the visual tracking algorithm of online learning has high requirements on the accuracy of the tracking template.
- the too large tracking template tends to contain more interference background, and the too small tracking template does not have sufficient discrimination.
- This interactive mode of defining a tracking template based on a user-specified area requires that the tracking target must be in a stationary state during the user-specified area to give the user enough time to accurately specify the area. However, if the tracking target is in motion, the user does not have enough time to accurately define the area in which the tracking target is located, and thus the tracking template that satisfies the requirements cannot be obtained, and then the tracking target in the motion cannot be visually tracked.
- the visual tracking algorithm for online learning in the prior art has a technical problem that it is impossible to visually track an object in motion because a user-specified area is used to define an interaction mode of the tracking template.
- Embodiments of the present invention provide a target tracking method, an electronic device, and a computer storage medium.
- the visual tracking algorithm for online learning in the prior art is solved. Since the user-specified area is used to define the interaction mode of the tracking template, there is a technical problem that the object in motion cannot be visually tracked.
- a target tracking method which is applied to an electronic device, the electronic device having a camera, and the target tracking method includes:
- Target Point Receiving location information of a target point sent by the control device, where the target point is a point selected by the user on the screen when the control device outputs a screen of the image data through a display screen, the target Point to track any point on the target;
- the tracking target is visually tracked by using a visual tracking algorithm based on the tracking template corresponding to the tracking target.
- the determining, according to the location information of the target point, the tracking template corresponding to the tracking target including:
- the optimal candidate tracking template is A tracking template corresponding to the tracking target.
- the method further includes:
- controlling the electronic device to advance to the target point comprises:
- the pitch angle of the camera is adjusted according to the vertical off angle such that the target point is in the vertical direction near the center of the picture.
- controlling the electronic device to advance to the target point comprises:
- the electronic device is controlled to rotate based on the rotational angular velocity such that the target point is horizontally close to the center of the picture.
- controlling the electronic device to advance to the target point comprises:
- the electronic device is controlled to advance toward the tracking target based on the forward speed.
- an electronic device having a camera, and the electronic device further includes:
- a sending unit configured to send image data collected by the camera to a control device
- a receiving unit configured to receive location information of a target point sent by the control device, where the target point is selected by the user on the screen when the control device outputs a screen of the image data through a display screen Point, the target point is any point on the tracking target;
- a determining unit configured to determine a tracking template corresponding to the tracking target based on location information of the target point
- the tracking unit is configured to perform visual tracking on the tracking target by using a visual tracking algorithm based on the tracking template corresponding to the tracking target.
- the determining unit includes:
- a first determining module configured to determine the target point in the screen based on location information of the target point
- Generating a module configured to generate a plurality of candidate tracking templates around the target point
- An analysis module configured to analyze the plurality of candidate tracking templates separately based on the saliency algorithm, and select a best candidate tracking template from the plurality of candidate tracking templates, wherein the best The candidate tracking template is the tracking template corresponding to the tracking target.
- the electronic device further includes:
- control unit configured to control the electronic device to be aligned with the target point during the visual tracking of the tracking target by using a visual tracking algorithm based on the tracking template, wherein the electronic device is Mobile electronic devices.
- control unit includes:
- an adjustment module configured to adjust a pitch angle of the camera according to the vertical deviation angle such that the target point is in a vertical direction near a center of the picture.
- control unit includes:
- a fourth determining module configured to determine a rotational angular velocity of the electronic device according to the horizontal offset angle
- the first control module is configured to control the electronic device to rotate based on the rotational angular velocity such that the target point is horizontally close to a center of the picture.
- control unit includes:
- a fifth determining module configured to determine a forward speed of the electronic device according to a size of the tracking template corresponding to the tracking target in the screen
- the second control module is configured to control the electronic device to advance to the tracking target based on the forward speed.
- a computer storage medium stores computer executable instructions configured to perform target tracking according to an embodiment of the present invention. method.
- the interaction mode of the user defining the tracking template by clicking operation is integrated with the visual tracking algorithm of the online learning, and only the user needs to click on a target point on the tracking target to obtain the tracking target correspondingly.
- Tracking template to visually track the tracking target using a visual tracking algorithm for online learning Since the user does not need to spend too much time, the user can complete the click operation in an instant, thereby completing the definition of the tracking template, and has the advantages of convenient operation and quick operation. Therefore, the visual tracking algorithm for online learning in the prior art is effectively solved. Since the user-specified region is used to define the interaction mode of the tracking template, it takes a long time, and there is a technical problem that the object in motion cannot be visually tracked. The technical effect of visually tracking the moving object by integrating the interaction mode of the user to define the tracking template by clicking operation with the visual tracking algorithm of online learning is realized.
- the user can monitor whether the electronic device has a target and a lost situation through the control device.
- the target and the lost situation occur, the target point can be corrected in time by clicking the operation, thereby making up for some vision.
- Tracking algorithms eg long-term tracking algorithms
- the image data collected by the camera is sent to the control device, so that the field of view of the electronic device is shared with the user, so that the user can conveniently determine the target point in the screen output by the control device, thereby making
- the electronic device can determine a tracking template of the tracking target based on the target point, and visually track the tracking target by using a visual tracking algorithm.
- the electronic device can also automatically align with the target point, so that the user can click on any target point on the screen output by the control device to control the driving direction of the electronic device.
- the user can watch the road condition information of the electronic device during driving in real time on the control device, and The driving direction of the sub-device is adjusted, which has the advantage of convenient handling.
- FIG. 1 is a flowchart of a target tracking method according to an embodiment of the present invention.
- FIG. 2 is a structural diagram of an electronic device according to an embodiment of the present invention.
- FIG. 3 is a flowchart of a target tracking method according to an embodiment of the present invention.
- the embodiment of the present invention solves the prior art visual tracking algorithm for online learning by providing a target tracking method and an electronic device. Since the user-specified area is used to define the interaction mode of the tracking template, there is a possibility that the moving object cannot be performed. Technical issues of visual tracking.
- a target tracking method is applied to an electronic device, the electronic device having a camera, the method comprising: transmitting image data collected by the camera to the control device; receiving location information of the target point sent by the control device, wherein the target The point is a point selected by the user on the screen when the control device outputs the image data through the display screen, and the target point is any point on the tracking target; based on the position information of the target point, the tracking template corresponding to the tracking target is determined; based on the tracking target The corresponding tracking template uses a visual tracking algorithm to visually track the tracking target.
- the embodiment provides a target tracking method, which is applied to an electronic device.
- the electronic device has a camera.
- the target tracking method includes:
- Step S101 Send image data collected by the camera to the control device.
- the electronic device may be a mobile electronic device, including a power system capable of driving the electronic device to travel (eg, straight, turn, reverse, curve, and many more).
- a power system capable of driving the electronic device to travel (eg, straight, turn, reverse, curve, and many more).
- a camera is disposed on the electronic device for acquiring image information.
- the camera In order to obtain a broader view of image acquisition, for ground mobile electronic devices, the camera should be placed close to the top of the electronic device; for aerial mobile electronic devices (such as drones), the camera should be placed as close as possible. The location of the bottom of the electronic device.
- control device can be connected to the electronic device by wire or wirelessly, so that the control device can communicate with the electronic device. After the electronic device collects the image data through the camera, the collected image data is sent to the control device in real time.
- control device may be a device independent of the electronic device (for example: a smart phone, or a tablet computer, or a laptop computer, or a desktop computer, or a smart TV, or a smart key, etc.); Alternatively, the control device can also be placed directly on the electronic device as part of the electronic device.
- Step S102 Receive location information of a target point sent by the control device, where the target point is a point selected by the user on the screen when the control device outputs a screen of image data through the display screen.
- the control device may output a screen corresponding to the image data (ie, a bitmap image) through a touch screen (eg, a capacitive touch screen or a resistive touch screen).
- a touch screen eg, a capacitive touch screen or a resistive touch screen.
- the user can search for a tracking target on the screen output by the control device (the tracking target can be a stationary object or a moving object), and click and click on any pixel on the tracking target by clicking the operation.
- the target point After the control device obtains the click operation of the user, the location information of the target point that the user clicks (for example, the two-dimensional coordinates of the target point on the screen) can be sent to the electronic device.
- the electronic device receives the location information of the target point sent by the control device.
- Step S103 Determine a tracking template corresponding to the tracking target based on the location information of the target point.
- step S103 includes:
- the target point and the color information of other pixels near the target point can be obtained, and the color information of these pixels is analyzed, which constitutes a tracking target.
- the pixels may have the same or similar colors, so an area composed of pixels of the same or similar color may be defined as an alternate tracking template, thereby generating a plurality of candidate tracking templates.
- noise and random deformation may be added to each candidate tracking template to generate a positive sample of each candidate tracking template; the background area around each candidate tracking template is randomly selected. As such, a negative sample of each candidate tracking template is generated; a check set for each candidate tracking template is generated based on a positive sample of each candidate tracking template and a negative sample of each candidate tracking template.
- training a tracking model of each candidate tracking template generating a check set of each candidate tracking template; and testing a response of each candidate tracking template tracking model on a test set corresponding to the candidate tracking template;
- the saliency of the tracking model of each candidate tracking template is determined based on the ratio of the average response on the positive sample and the negative sample set; finally, the candidate tracking template corresponding to the most significant tracking model is determined as the best candidate tracking template.
- each candidate tracking template is a rectangular area, so that the best candidate tracking template is also a rectangular area, which satisfies the requirements of the online learning visual tracking algorithm for the tracking template, and thus realizes
- This interaction pattern that defines the tracking template by the user through a click operation is integrated with the visual tracking algorithm of the online learning.
- Step S104 visually track the tracking target by using a visual tracking algorithm based on the tracking template corresponding to the tracking target.
- the visual tracking algorithm has high robustness in a certain period of time (specifically limited by the complexity of the application environment, which is between tens of seconds and ten minutes), so the visual tracking algorithm It provides a high degree of automation, allowing accurate tracking of tracking targets without the need for user control during accurate tracking.
- the visual tracking algorithm refers to a visual tracking algorithm for online learning.
- the specific visual tracking algorithm, any short-term tracking algorithm and long-term tracking algorithm are not limited herein. Can be used. However, in view of the better robustness of the short-term tracking algorithm and the more mature algorithm, the short-term tracking algorithm should be selected as much as possible. For example, a tracking algorithm using a Correlation Filter can be selected to obtain better robustness. And reduce the complexity of the algorithm.
- the tracking target when the electronic device performs visual tracking on the tracking target, the tracking target may be marked in the image data collected by the camera, and the image data carrying the mark is sent to the control device.
- the control device When the control device outputs the screen corresponding to the image data, the user can determine whether the electronic device has a target or not according to the positional relationship between the mark in the screen and the tracking target. For example, if the mark in the picture coincides with the position of the tracking target, it means that there is no missing; if the mark in the picture does not coincide with the position of the tracking target, or even far apart, it means that it is lost.
- the electronic device When the user finds that the electronic device has a target and loses, it can click on any point of the tracking target on the display screen of the control device to determine a new target point, so that the electronic device can re-determine the tracking based on the new target point.
- the tracking template corresponding to the target, and visual tracking of the tracking target by using a visual tracking algorithm.
- the user can monitor whether the electronic device has a target and a lost situation through the control device, and when the target is lost, the target point can be repaired in time by clicking the operation. Positive, thus making up for the lack of robustness of some visual tracking algorithms (eg long-term tracking algorithms).
- the interaction mode of the user defining the tracking template by clicking operation is integrated with the visual tracking algorithm of the online learning, and only the user needs to click on a target point on the tracking target to obtain the tracking corresponding to the tracking target.
- a template for visually tracking the tracking target using a visual tracking algorithm for online learning Since the user does not need to spend too much time, the user can complete the click operation in an instant, thereby completing the definition of the tracking template, and has the advantages of convenient operation and quick operation. Therefore, the visual tracking algorithm for online learning is effectively solved. Since the user-specified area is used to define the interaction mode of the tracking template, it takes a long time, and there is a technical problem that the object in motion cannot be visually tracked. The technical effect of visually tracking objects in motion using a visual tracking algorithm of online learning is realized.
- step S105 may also be performed simultaneously.
- Step S105 The control electronic device advances toward the target point.
- the pitch angle of the camera and the turning of the electronic device can be adjusted according to the position of the target point in the screen, so that the target point gradually approaches the center of the screen, thereby controlling the electronic device to be aligned with the target point, and then controlling the electronic device. go ahead.
- the vertical deviation angle of the target point from the center of the screen in the vertical direction may be determined based on the equation (1):
- Picth is the vertical deviation angle
- f is the focal length of the camera
- dy is the vertical offset of the target point to the optical axis of the camera
- the pitch angle of the camera is adjusted so that the vertical deviation angle Picth gradually decreases toward 0, and finally the target point is brought to the center of the screen in the vertical direction.
- the horizontal offset angle of the target point and the center of the screen in the horizontal direction may be determined based on the equation (2):
- yaw is the horizontal offset angle
- f is the focal length of the camera
- dx is the horizontal offset of the target point to the optical axis of the camera
- the rotational angular velocity of the electronic device is determined; based on the rotational angular velocity, the electronic device is controlled to rotate, so that the horizontal offset angle yaw gradually decreases toward zero, and finally the target point is horizontally close to the center of the screen.
- the rotational angular velocity is proportional to the horizontal offset angle, so that the greater the horizontal offset angle, the greater the rotational angular velocity, and the target point can be quickly approached to the center of the screen in the horizontal direction.
- the target point base This is located at the center of the screen.
- the electronic device can be controlled to advance and gradually approach the target point, thereby achieving tracking of the target point.
- the advancement speed of the electronic device may be determined according to the size of the tracking template corresponding to the tracking target in the screen; and based on the forward speed, the control electronic device advances to the tracking target.
- the sigmoid function can be used to formulate the velocity decay model.
- the advancement speed of the electronic device is inversely proportional to the size of the tracking template corresponding to the tracking target in the screen, so that the tracking target and the electronic device are indicated when the size of the tracking template corresponding to the tracking target is smaller in the screen.
- the advancement speed of the control electronic device is slowed, thereby preventing the electronic device from colliding with the tracking target.
- the image data collected by the camera is sent to the control device, so that the field of view of the electronic device is shared with the user, so that the user can conveniently determine the target point in the screen output by the control device, thereby making the electronic device
- the tracking template of the tracking target can be determined based on the target point, and the tracking target is visually tracked by using a visual tracking algorithm.
- the electronic device can also automatically align with the target point, so that the user can click on any target point on the screen output by the control device to control the driving direction of the electronic device, and the operation is convenient. And the user can watch the road condition information of the electronic device during driving in real time on the control device, and adjust the driving direction of the electronic device.
- the user can continuously click on different target points in the screen on the touch screen of the control device, so that the remote control electronic device can freely travel.
- the embodiment provides an electronic device, and the electronic device has a camera. As shown in FIG. 2, the electronic device further includes:
- the sending unit 201 is configured to send image data collected by the camera to the control device;
- the receiving unit 202 is configured to receive location information of a target point sent by the control device, where the target point is a point selected by the user on the screen when the control device outputs a screen of image data through the display screen, and the target point is on the tracking target. Any point;
- a determining unit 203 configured to determine, according to location information of the target point, a tracking template corresponding to the tracking target;
- the tracking unit 204 is configured to perform visual tracking on the tracking target by using a visual tracking algorithm based on the tracking template corresponding to the tracking target.
- the determining unit 203 includes:
- a first determining module configured to determine a target point in the screen based on location information of the target point
- An analysis module for analyzing a plurality of candidate tracking templates based on the saliency algorithm, And selecting a best candidate tracking template from the plurality of candidate tracking templates, wherein the best candidate tracking template is the tracking template corresponding to the tracking target.
- the electronic device further includes:
- the control unit 205 is configured to control the electronic device to advance to the target point during the visual tracking of the tracking target by using the visual tracking algorithm based on the tracking template, wherein the electronic device is a movable electronic device.
- control unit 205 includes:
- the adjustment module is configured to adjust the pitch angle of the camera according to the vertical deviation angle so that the target point is vertically close to the center of the screen.
- control unit 205 includes:
- a fourth determining module configured to determine a rotational angular velocity of the electronic device according to the horizontal offset angle
- the first control module is configured to control the rotation of the electronic device based on the rotational angular velocity such that the target point is horizontally close to the center of the screen.
- control unit 205 includes:
- a fifth determining module configured to determine a forward speed of the electronic device according to a size of the tracking template corresponding to the tracking target in the screen
- a second control module configured to control the electronic device to advance to the tracking target based on the forward speed.
- the electronic device introduced in this embodiment is an electronic device used in the implementation of the target tracking method in the embodiment of the present invention. Therefore, those skilled in the art can understand the target tracking method according to the embodiment of the present invention.
- the specific implementation of the electronic device and various changes thereof are not described in detail herein for how the electronic device implements the method in the embodiment of the present invention.
- the electronic device used by the person skilled in the art to implement the target tracking method in the embodiment of the present invention is within the scope of the present invention.
- the interaction mode of the user defining the tracking template by clicking operation is integrated with the visual tracking algorithm of the online learning, and only the user needs to click on a target point on the tracking target to obtain the tracking target.
- Corresponding tracking templates are used to visually track the tracking target using a visual tracking algorithm of online learning. Since the user does not need to spend too much time, the user can complete the click operation in an instant, thereby completing the definition of the tracking template, and has the advantages of convenient operation and quick operation. Therefore, the visual tracking algorithm for online learning is effectively solved. Since the user-specified area is used to define the interaction mode of the tracking template, it takes a long time, and there is a technical problem that the object in motion cannot be visually tracked. The technical effect of visually tracking the moving object by integrating the interaction mode of the user to define the tracking template by clicking operation with the visual tracking algorithm of online learning is realized.
- the user can monitor whether the electronic device has a target and a lost situation through the control device, and when the target is lost, the target point can be corrected in time by clicking the operation, thereby making up for some Visual tracking algorithms (eg, long-term tracking algorithms) lack robustness.
- some Visual tracking algorithms eg, long-term tracking algorithms
- the image data collected by the camera is sent to the control device, so that the field of view of the electronic device is shared with the user, so that the user can conveniently determine the target point in the screen output by the control device, and then
- the electronic device is enabled to determine a tracking template of the tracking target based on the target point, and visually track the tracking target by using a visual tracking algorithm.
- the electronic device can also automatically align with the target point, so that the user can click on any target point on the screen output by the control device to control the driving direction of the electronic device.
- the user can watch the road condition information of the electronic device during driving in real time on the control device, and adjust the traveling direction of the electronic device, which has the advantages of convenient control.
- the embodiment provides a target tracking method, which is applied to a self-balancing vehicle.
- the self-balancing vehicle has a camera.
- the target tracking method includes:
- Step S301 Send image data collected by the camera to the control device.
- the self-balancing vehicle mainly has two types of single wheel and two wheels, and its operation principle is mainly based on a basic principle called “Dynamic Stabilization", which utilizes the interior of the vehicle body.
- the gyroscope and accelerometer are used to detect changes in the attitude of the car body, and the servo control system is used to accurately drive the motor to adjust accordingly to maintain the balance of the system.
- the self-balancing vehicle includes a power system that can drive a self-balancing vehicle on the road (eg straight, turn, reverse, curve, etc.).
- a camera is disposed on the self-balancing vehicle for collecting image information.
- the camera In order to obtain a wider view of the image acquisition, the camera should be placed as close as possible to the top of the self-balancing car.
- control device can be connected to the self-balancing vehicle by wire or wirelessly, so that the control device can communicate with the self-balancing vehicle. After the self-balancing vehicle collects image data through the camera, the collected image data is sent to the control device in real time.
- the control device may be a device independent of the self-balancing vehicle (eg, Such as: smart phone, or tablet, or laptop, or desktop computer, or smart TV, or smart key, etc.; or, the control device can also be directly set on the self-balancing car as part of the self-balancing car .
- the control device is a smart key, it can control the self-balancing car to lock the car, unlock the car, or turn on the anti-theft alarm.
- Step S302 Receive location information of a target point sent by the control device, where the target point is a point selected by the user on the screen when the control device outputs a screen of image data through the display screen.
- the control device may output a picture corresponding to the image data through a touch screen (for example, a capacitive touch screen or a resistive touch screen) (ie, a picture transfer screen).
- a touch screen for example, a capacitive touch screen or a resistive touch screen
- the vision of the self-balancing vehicle is shared with the user.
- the user can search for a tracking target on the screen output by the control device (the tracking target can be a stationary object or a moving object), and click and click on any pixel on the tracking target by clicking the operation.
- the target point After the control device obtains the click operation of the user, the location information (for example, coordinate information) of the target point clicked by the user can be sent to the self-balancing vehicle.
- the self-balancing vehicle receives the position information of the target point sent from the control device.
- Step S303 Determine a tracking template corresponding to the tracking target based on the location information of the target point.
- step S303 includes:
- the target point and the color information of other pixels near the target point can be obtained, and the color information of these pixels is analyzed, which constitutes a tracking target.
- the pixels may have the same or similar colors, so an area composed of pixels of the same or similar color may be defined as an alternate tracking template, thereby generating a plurality of candidate tracking templates.
- noise and random deformation may be added to each candidate tracking template to generate a positive sample of each candidate tracking template; randomly sample the background area around each candidate tracking template to generate each candidate A negative sample of the tracking template is generated; a check set for each candidate tracking template is generated based on a positive sample of each candidate tracking template and a negative sample of each candidate tracking template.
- training a tracking model of each candidate tracking template generating a check set of each candidate tracking template; and testing a response of each candidate tracking template tracking model on a test set corresponding to the candidate tracking template;
- the saliency of the tracking model of each candidate tracking template is determined based on the ratio of the average response on the positive sample and the negative sample set; finally, the candidate tracking template corresponding to the most significant tracking model is determined as the best candidate tracking template.
- each candidate tracking template is a rectangular area, so that the final result
- the best candidate tracking template is also a rectangular area, which satisfies the requirements of the online learning visual tracking algorithm for the tracking template, and thus realizes the interaction mode and the online learning visual tracking algorithm for the user to define the tracking template by clicking operation. Convergence.
- Step S304 The tracking target is visually tracked by using a visual tracking algorithm based on the tracking template corresponding to the tracking target.
- the visual tracking is highly robust in a certain period of time (specifically limited by the complexity of the application environment, which is between tens of seconds and ten minutes), so the visual tracking algorithm It provides a high degree of automation, allowing accurate tracking of tracking targets without the need for user control during accurate tracking.
- the visual tracking algorithm refers to a visual tracking algorithm for online learning.
- the specific visual tracking algorithm, any short-term tracking algorithm and long-term tracking algorithm are not limited herein. Can be used. However, in view of the better robustness of the short-term tracking algorithm and the more mature algorithm, the short-term tracking algorithm should be selected as much as possible. For example, a visual tracking algorithm using a Correlation Filter can be selected for better robustness. And reduce the complexity of the algorithm.
- the tracking target when the self-balancing vehicle visually tracks the tracking target, the tracking target may be marked in the image data collected by the camera, and the image data carrying the marking is sent to the control device.
- the control device When the control device outputs the screen corresponding to the image data, the user can judge whether the self-balancing vehicle has a target or a loss according to the positional relationship between the mark in the screen and the tracking target. For example, if the mark in the picture coincides with the position of the tracking target, it means that there is no missing; if the mark in the picture does not coincide with the position of the tracking target, or even far apart, it means that it is lost.
- the user finds that the self-balancing vehicle has a target and a lost situation, he can click on any point of the tracking target on the display screen of the control device to determine a new target point, so that the self-balancing vehicle is re-based on the new target point.
- the tracking template corresponding to the tracking target is determined, and the tracking target is visually tracked by using a visual tracking algorithm.
- the user can monitor whether the self-balancing vehicle has a target and a lost situation through the control device.
- the target point can be corrected in time by clicking the operation, thereby making up for some visual tracking.
- Algorithms eg, long-term tracking algorithms
- the interaction mode of the user defining the tracking template by clicking operation is integrated with the visual tracking algorithm of the online learning, and only the user needs to click on a target point on the tracking target to obtain the tracking corresponding to the tracking target.
- a template for visually tracking the tracking target using a visual tracking algorithm for online learning Since the user does not need to spend too much time, the user can complete the click operation in an instant, thereby completing the definition of the tracking template, and has the advantages of convenient operation and quick operation. Therefore, the visual tracking algorithm for online learning is effectively solved. Since the user-specified area is used to define the interaction mode of the tracking template, it takes a long time, and there is a technical problem that the object in motion cannot be visually tracked. question. The technical effect of visually tracking objects in motion using a visual tracking algorithm of online learning is realized.
- step S305 may also be performed simultaneously.
- Step S305 Control the self-balancing vehicle to advance toward the target point.
- the pitch angle of the camera can be adjusted, and the self-balancing vehicle can be turned, so that the target point gradually approaches the center of the screen, thereby controlling the self-balancing vehicle to align with the target point, and then controlling.
- Self-balancing car advances.
- the vertical deviation angle of the target point from the center of the screen in the vertical direction may be determined based on the equation (1):
- Picth is the vertical deviation angle
- f is the focal length of the camera
- dy is the vertical offset of the target point to the optical axis of the camera
- the pitch angle of the camera is adjusted so that the vertical deviation angle Picth gradually decreases toward 0, and finally the target point is brought to the center of the screen in the vertical direction.
- the horizontal offset angle of the target point and the center of the screen in the horizontal direction may be determined based on the equation (2):
- yaw is the horizontal offset angle
- f is the focal length of the camera
- dx is the horizontal offset of the target point to the optical axis of the camera
- the rotational angular velocity of the self-balancing vehicle is determined; based on the rotational angular velocity, the self-balancing vehicle is controlled to rotate, so that the horizontal offset angle yaw gradually decreases toward zero, and finally the target point is horizontally close to the center of the screen.
- the rotational angular velocity is proportional to the horizontal offset angle, so that the greater the horizontal offset angle, the greater the rotational angular velocity, and the target point can be quickly approached to the center of the screen in the horizontal direction.
- the target point is basically at the center point of the screen, and at this time, the self-balancing vehicle can be controlled to advance and gradually approach the target point. Thereby achieving tracking of the target point.
- the forward speed of the self-balancing vehicle may be determined according to the size of the tracking template corresponding to the tracking target in the screen; and the self-balancing vehicle is controlled to advance to the tracking target based on the forward speed.
- the sigmoid function can be used to formulate the velocity decay model.
- the forward speed of the self-balancing vehicle is inversely proportional to the size of the tracking template corresponding to the tracking target in the screen, so that the smaller the size of the tracking template corresponding to the tracking target in the screen, the tracking target and the self
- the faster the speed the faster the self-balancing car is controlled to approach the tracking target, and the larger the size of the tracking template corresponding to the tracking target is, the closer the tracking target is to the self-balancing vehicle, the more the self-balancing vehicle is controlled.
- the slower the speed the more the self-balancing car collides with the tracking target.
- the image data collected by the camera is sent to the control device, so that the field of view of the self-balancing vehicle is shared with the user, so that the user can conveniently determine the target point in the screen output by the control device, thereby enabling the self-determination.
- the balance car can determine the tracking template of the tracking target based on the target point, and visually track the tracking target by using a visual tracking algorithm.
- the self-balancing vehicle can also automatically align with the target point, so that the user can click on any target point on the screen output by the control device to control the driving direction of the self-balancing vehicle. Convenient, and the user can watch the road condition information of the self-balancing car during driving in real time on the control device, and adjust the driving direction of the self-balancing car.
- the user can continuously click on different target points in the screen on the touch screen of the control device, thereby remotely controlling the self-balancing vehicle to travel freely.
- the present embodiment provides a self-balancing vehicle having a camera, the self-balancing vehicle further including the transmitting unit 201, the receiving unit 202, the determining unit 203, and the tracking shown in FIG. 2.
- Unit 204 among them,
- the sending unit 201 is configured to send image data collected by the camera to the control device;
- the receiving unit 202 is configured to receive location information of a target point sent by the control device, where the target point is a point selected by the user on the screen when the control device outputs a screen of image data through the display screen, and the target point is on the tracking target. Any point;
- a determining unit 203 configured to determine, according to location information of the target point, a tracking template corresponding to the tracking target;
- the tracking unit 204 is configured to perform visual tracking on the tracking target by using a visual tracking algorithm based on the tracking template corresponding to the tracking target.
- the determining unit 203 includes:
- a first determining module configured to determine a target point in the screen based on location information of the target point
- An analysis module configured to separately analyze a plurality of candidate tracking templates based on the saliency algorithm, and select a best candidate tracking template from the plurality of candidate tracking templates, wherein the optimal candidate tracking template is Track the tracking template corresponding to the target.
- the self-balancing vehicle further includes:
- the control unit 205 is configured to control the self-balancing vehicle to advance toward the target point during the visual tracking of the tracking target by using the visual tracking algorithm based on the tracking template, wherein the self-balancing vehicle is a movable self-balancing vehicle.
- control unit 205 includes:
- the adjustment module is configured to adjust the pitch angle of the camera according to the vertical deviation angle so that the target point is vertically close to the center of the screen.
- control unit 205 includes:
- a fourth determining module configured to determine a rotational angular velocity of the self-balancing vehicle according to the horizontal offset angle
- the first control module is configured to control the rotation of the self-balancing vehicle based on the rotational angular velocity such that the target point is horizontally close to the center of the screen.
- control unit 205 includes:
- a fifth determining module configured to determine a forward speed of the self-balancing vehicle according to a size of the tracking template corresponding to the tracking target in the screen
- the second control module is configured to control the self-balancing vehicle to advance toward the tracking target based on the forward speed.
- the self-balancing vehicle introduced in this embodiment is a mobile electronic device used in the implementation of the target tracking method in the embodiment of the present invention. Therefore, those skilled in the art can understand the present invention based on the target tracking method introduced in the embodiment of the present invention.
- the specific embodiment of the self-balancing vehicle of the embodiment and various modifications thereof are not described in detail herein for how the self-balancing vehicle implements the method in the embodiment of the present invention.
- the self-balancing vehicle used by the person skilled in the art to implement the target tracking method in the embodiment of the present invention is within the scope of the present invention.
- the interaction mode of the user defining the tracking template by clicking operation is integrated with the visual tracking algorithm of the online learning, and only the user needs to click on a target point on the tracking target to obtain the tracking target.
- Corresponding tracking templates are used to visually track the tracking target using a visual tracking algorithm of online learning. Since the user does not need to spend too much time, the user can complete the click operation in an instant, thereby completing the definition of the tracking template, and has the advantages of convenient operation and quick operation. Therefore, the visual tracking algorithm for online learning is effectively solved. Since the user-specified area is used to define the interaction mode of the tracking template, it takes a long time, and there is a technical problem that the object in motion cannot be visually tracked. The technical effect of visually tracking the moving object by integrating the interaction mode of the user to define the tracking template by clicking operation with the visual tracking algorithm of online learning is realized.
- the user can monitor whether the self-balancing vehicle has a target and a lost situation through the control device.
- the target point can be corrected in time by clicking the operation, thereby making up for a certain
- the image data collected by the camera is sent to the control device, so that the field of view of the self-balancing vehicle is shared with the user, so that the user can conveniently determine the target point in the screen output by the control device.
- the self-balancing vehicle can determine the tracking template of the tracking target based on the target point, and visually track the tracking target by using a visual tracking algorithm.
- the self-balancing vehicle can also automatically align with the target point, so that the user can click on any target point on the screen output by the control device to control the driving direction of the self-balancing vehicle.
- the user can watch the road condition information of the self-balancing car during driving in real time on the control device, and adjust the driving direction of the self-balancing car, which has the advantages of convenient control.
- the foregoing program may be stored in a computer readable storage medium, and the program is executed when executed.
- the foregoing storage device includes the following steps: the foregoing storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
- ROM read-only memory
- RAM random access memory
- magnetic disk or an optical disk.
- optical disk A medium that can store program code.
- the above-described integrated unit of the embodiment of the present invention may be stored in a computer readable storage medium if it is implemented in the form of a software function module and sold or used as a stand-alone product.
- the technical solution of the embodiments of the present invention may be embodied in the form of a software product in essence or in the form of a software product stored in a storage medium, including a plurality of instructions.
- a computer device (which may be a personal computer, server, or network device, etc.) is caused to perform all or part of the methods described in various embodiments of the present invention.
- the foregoing storage medium includes: a removable storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like, which can store program codes.
- an embodiment of the present invention further provides a computer readable storage medium, the storage medium comprising a set of computer executable instructions for performing a control method of a mobile electronic device according to an embodiment of the present invention.
- the computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device.
- the apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.
- These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device.
- the instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
- Image Analysis (AREA)
Abstract
本发明公开了一种目标跟踪方法，应用于电子设备中，该电子设备具有一摄像头，该方法包括：将摄像头采集到的图像数据发送给控制设备；接收控制设备发来的目标点的位置信息，其中，目标点为在控制设备通过显示屏输出图像数据的画面时用户在画面上选中的点，目标点为跟踪目标上的任一点；基于目标点的位置信息，确定跟踪目标对应的跟踪模板；基于跟踪目标对应的跟踪模板，利用视觉跟踪算法对跟踪目标进行视觉跟踪。本发明解决了现有技术中的在线学习的视觉跟踪算法，由于采用用户指定区域来定义跟踪模板的交互模式，存在无法对运动中的物体进行视觉跟踪的技术问题。同时，本发明还提供了一种电子设备、计算机存储介质。
Description
本发明涉及视觉跟踪技术领域，尤其涉及一种目标跟踪方法及电子设备、计算机存储介质。
视觉跟踪技术，是指对图像序列中的跟踪目标进行检测、提取、识别和跟踪，获得跟踪目标运动参数，如位置、速度、加速度和运动轨迹等，从而进行下一步的处理与分析，实现对跟踪目标的行为理解，以完成更高一级的检测任务。视觉跟踪技术主要目的是模仿生理视觉系统的运动感知功能，赋予机器辨识图像序列中物体运动及其相互关系的能力，为图像序列理解提供重要途径。视觉跟踪技术具有广阔的应用前景，如视频监控、视频分析、视频检索、基于视频的运动分析和合成、基于运动信息的身份识别等。
近几年，基于在线学习的视觉跟踪技术兴起，各种在线学习的视觉跟踪算法不断涌现，在没有任何离线学习的先验经验的情况下，根据在初始帧画面中定义的跟踪模板，训练模型用于后续视频中目标的跟踪，并在跟踪过程不断更新模型，以达到适应目标物体姿态变化，以及克服复杂背景干扰的目的。由于无需离线训练，使得在线学习的视觉跟踪技术具有很高的通用性，可以对用户指定的任何物体进行视觉跟踪。
现有的在线学习的视觉跟踪算法的交互模式，需要用户在视频画面中指定跟踪目标所在的区域(该区域通常为一个矩形区域，需要用户通过画框操作画出该区域)，从而将该区域定义跟踪模板。在线学习的视觉跟踪算法对跟踪模板的精度有较高的要求，过大的跟踪模板易包含较多的干扰背景，过小的跟踪模板不具有足够的区分度。这种基于用户指定区域来定义跟踪模板的交互模式，要求在用户指定区域过程中，跟踪目标必须处于静止状态，以给用户足够的时间来准确指定该区域。但是，若跟踪目标处于运动状态，用户则没有足够的时间来准确定义跟踪目标所在的区域，也就无法获得满足要求的跟踪模板，继而无法对运动中的跟踪目标进行视觉跟踪。
综上，现有技术中的在线学习的视觉跟踪算法，由于采用用户指定区域来定义跟踪模板的交互模式，存在无法对运动中的物体进行视觉跟踪的技术问题。
发明内容
本发明实施例通过提供一种目标跟踪方法及电子设备、计算机存储介质，
解决了现有技术中的在线学习的视觉跟踪算法，由于采用用户指定区域来定义跟踪模板的交互模式，存在无法对运动中的物体进行视觉跟踪的技术问题。
在本发明的一实施例中，提供了一种目标跟踪方法，应用于电子设备中，所述电子设备具有一摄像头，所述目标跟踪方法包括：
将所述摄像头采集到的图像数据发送给控制设备；
接收所述控制设备发来的目标点的位置信息，其中，所述目标点为在所述控制设备通过显示屏输出所述图像数据的画面时用户在所述画面上选中的点，所述目标点为跟踪目标上的任一点；
基于所述目标点的位置信息，确定所述跟踪目标对应的跟踪模板；
基于所述跟踪目标对应的跟踪模板，利用视觉跟踪算法对所述跟踪目标进行视觉跟踪。
在一可实施方式中，所述基于所述目标点的位置信息，确定所述跟踪目标对应的跟踪模板，包括：
基于所述目标点的位置信息，在所述画面中确定所述目标点；
在所述目标点周围生成多个备选跟踪模板；
基于显著性算法，分别对所述多个备选跟踪模板进行分析，并从所述多个备选跟踪模板中选出一最佳备选跟踪模板，其中，所述最佳备选跟踪模板即为所述跟踪目标对应的跟踪模板。
在一可实施方式中，在所述基于所述跟踪模板，利用视觉跟踪算法对所述跟踪目标进行视觉跟踪过程中，还包括：
控制所述电子设备对准所述目标点前进，其中，所述电子设备为可移动电子设备。
在一可实施方式中，所述控制所述电子设备对准所述目标点前进，包括：
基于等式Picth＝arctan(dy/f)，确定所述目标点与所述画面的中心在竖直方向上的竖直偏离角；其中，Picth为所述竖直偏离角，f为所述摄像头的焦距，dy为所述目标点到所述摄像头光轴的垂直偏移量；
根据所述竖直偏离角，调整所述摄像头的俯仰角，以使所述目标点在竖直方向靠近所述画面的中心。
在一可实施方式中，所述控制所述电子设备对准所述目标点前进，包括：
基于等式yaw＝arctan(dx/f)，确定所述目标点与所述画面的中心在水平方向上的水平偏移角；其中，yaw为所述水平偏移角，f为所述摄像头的焦距，dx为所述目标点到所述摄像头光轴的水平偏移量；
根据所述水平偏移角，确定所述电子设备的转动角速度；
基于所述转动角速度，控制所述电子设备转动，以使所述目标点在水平方向靠近所述画面的中心。
在一可实施方式中，所述控制所述电子设备对准所述目标点前进，包括：
根据所述跟踪目标对应的跟踪模板在所述画面中的尺寸，确定所述电子设备的前进速度；
基于所述前进速度，控制所述电子设备向所述跟踪目标前进。
在本发明的另一实施例中提供了一种电子设备，所述电子设备具有一摄像头，所述电子设备，还包括：
发送单元，配置为将所述摄像头采集到的图像数据发送给控制设备；
接收单元，配置为接收所述控制设备发来的目标点的位置信息，其中，所述目标点为在所述控制设备通过显示屏输出所述图像数据的画面时用户在所述画面上选中的点，所述目标点为跟踪目标上的任一点；
确定单元，配置为基于所述目标点的位置信息，确定所述跟踪目标对应的跟踪模板；
跟踪单元，配置为基于所述跟踪目标对应的跟踪模板，利用视觉跟踪算法对所述跟踪目标进行视觉跟踪。
在一可实施方式中，所述确定单元，包括：
第一确定模块，配置为基于所述目标点的位置信息，在所述画面中确定所述目标点；
生成模块，配置为在所述目标点周围生成多个备选跟踪模板；
分析模块，配置为基于显著性算法，分别对所述多个备选跟踪模板进行分析，并从所述多个备选跟踪模板中选出一最佳备选跟踪模板，其中，所述最佳备选跟踪模板即为所述跟踪目标对应的跟踪模板。
在一可实施方式中，所述电子设备，还包括：
控制单元，配置为在所述基于所述跟踪模板，利用视觉跟踪算法对所述跟踪目标进行视觉跟踪过程中，控制所述电子设备对准所述目标点前进，其中，所述电子设备为可移动电子设备。
在一可实施方式中，所述控制单元，包括：
第二确定模块，配置为基于等式Picth＝arctan(dy/f)，确定所述目标点与所述画面的中心在竖直方向上的竖直偏离角；其中，Picth为所述竖直偏离角，f为所述摄像头的焦距，dy为所述目标点到所述摄像头光轴的垂直偏移量；
调整模块，配置为根据所述竖直偏离角，调整所述摄像头的俯仰角，以使所述目标点在竖直方向靠近所述画面的中心。
在一可实施方式中，所述控制单元，包括：
第三确定模块，配置为基于等式yaw＝arctan(dx/f)，确定所述目标点与所述画面的中心在水平方向上的水平偏移角；其中，yaw为所述水平偏移角，f为所述摄像头的焦距，dx为所述目标点到所述摄像头光轴的水平偏移量；
第四确定模块，配置为根据所述水平偏移角，确定所述电子设备的转动角速度；
第一控制模块，配置为基于所述转动角速度，控制所述电子设备转动，以使所述目标点在水平方向靠近所述画面的中心。
在一可实施方式中，所述控制单元，包括：
第五确定模块，配置为根据所述跟踪目标对应的跟踪模板在所述画面中的尺寸，确定所述电子设备的前进速度；
第二控制模块，配置为基于所述前进速度，控制所述电子设备向所述跟踪目标前进。
在本发明的另一实施例中，还提供了一种计算机存储介质，所述计算机存储介质中存储有计算机可执行指令，所述计算机可执行指令配置为执行本发明实施例所述的目标跟踪方法。
本发明实施例中提供的一个或多个技术方案，至少具有如下技术效果或优点：
1、在本实施例中，将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合，仅需要用户点击跟踪目标上的一个目标点，即可获得跟踪目标对应的跟踪模板，从而利用在线学习的视觉跟踪算法对所述跟踪目标进行视觉跟踪。由于无需耗费过多时间，用户瞬间即可完成该点击操作，从而完成对跟踪模板的定义，具有操作方便、快捷的优点。所以有效地解决了现有技术中的在线学习的视觉跟踪算法，由于采用用户指定区域来定义跟踪模板的交互模式，耗时较长，存在无法对运动中的物体进行视觉跟踪的技术问题。实现了将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合，来对运动中的物体进行视觉跟踪的技术效果。
2、在本实施例中，用户可以通过控制设备监视电子设备是否出现目标跟丢的情况，在出现目标跟丢的情况时，可以通过点击操作及时对目标点进行修正，从而弥补了某些视觉跟踪算法(例如：长期跟踪算法)鲁棒性差的不足。
3、在本实施例中，将摄像头采集到的图像数据发送给控制设备，实现了将电子设备的视野与用户共享，使得用户可以方便地在控制设备所输出的画面中确定目标点，进而使得电子设备可以基于目标点确定跟踪目标的跟踪模板，并利用视觉跟踪算法对跟踪目标进行视觉跟踪。且在对跟踪目标进行视觉跟踪过程中，电子设备还可以自动对准目标点前进，这样，用户即可在控制设备输出的画面上点击任一目标点，来控制电子设备的行驶方向，同时，用户可以在控制设备上实时观看电子设备在行驶过程中的路况信息，并对电
子设备的行驶方向进行调整，具有操控方便的优点。
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本发明实施例中一种目标跟踪方法的流程图；
图2为本发明实施例中一种电子设备的结构图；
图3为本发明实施例中一种目标跟踪方法的流程图。
下面将参照附图更详细地描述本公开的示例性实施例。虽然附图中显示了本公开的示例性实施例，然而应当理解，可以以各种形式实现本公开而不应被这里阐述的实施例所限制。相反，提供这些实施例是为了能够更透彻地理解本公开，并且能够将本公开的范围完整的传达给本领域的技术人员。
本发明实施例通过提供一种目标跟踪方法及电子设备，解决了现有技术中的在线学习的视觉跟踪算法，由于采用用户指定区域来定义跟踪模板的交互模式，存在无法对运动中的物体进行视觉跟踪的技术问题。
本发明实施例的技术方案为解决上述技术问题，总体思路如下：
一种目标跟踪方法，应用于电子设备中，该电子设备具有一摄像头，该方法包括：将摄像头采集到的图像数据发送给控制设备；接收控制设备发来的目标点的位置信息，其中，目标点为在控制设备通过显示屏输出图像数据的画面时用户在画面上选中的点，目标点为跟踪目标上的任一点；基于目标点的位置信息，确定跟踪目标对应的跟踪模板；基于跟踪目标对应的跟踪模板，利用视觉跟踪算法对跟踪目标进行视觉跟踪。
为了更好的理解上述技术方案，下面将结合说明书附图以及具体的实施方式对上述技术方案进行详细的说明。
实施例一
本实施例提供了一种目标跟踪方法，应用于电子设备中，所述电子设备具有一摄像头，如图1所示，所述目标跟踪方法，包括：
步骤S101：将摄像头采集到的图像数据发送给控制设备。
在具体实施过程中，所述电子设备可以是一可移动电子设备，其包括一动力系统，能够驱动电子设备行驶(例如：直行，转弯，倒退，曲线行驶，
等等)。
在具体实施过程中，在电子设备上设置有一摄像头，用于采集图像信息。为了获得更为广阔的图像采集视角，对于地面移动电子设备而言，摄像头应尽量设置在靠近电子设备顶部的位置；对于空中移动电子设备(如无人机)而言，摄像头应尽量设置在靠近电子设备底部的位置。
在具体实施过程中，控制设备可以与该电子设备通过有线方式或无线方式连接，以使得控制设备可以与电子设备进行通信。在电子设备通过摄像头采集到图像数据后，会将采集到的图像数据实时发送给控制设备。
在具体实施过程中，该控制设备可以是与电子设备为相互独立的设备(例如：智能手机、或平板电脑、或笔记本电脑、或台式机电脑、或智能电视、或智能钥匙、等等)；或者，该控制设备还可以直接设置在电子设备上，作为电子设备的一部分。
步骤S102：接收控制设备发来的目标点的位置信息，其中，目标点为在控制设备通过显示屏输出图像数据的画面时用户在画面上选中的点。
在具体实施过程中，控制设备接收到电子设备发来的图像数据后，可以通过一触摸屏(例如：电容式触摸屏、或电阻式触摸屏)输出该图像数据对应的画面(即：传图画面)，从而实现了将电子设备的视野与用户共享。用户可以在控制设备输出的画面上寻找一跟踪目标(该跟踪目标可以是静止的物体，也可以是运动的物体)，并通过点击操作点击跟踪目标上的任意一个像素点，这个像素点就是所述目标点。控制设备获取到用户的点击操作后，即可将用户点击的目标点的位置信息(例如：目标点在屏幕上的二维坐标)发送给电子设备。对应地，电子设备接收控制设备发来的目标点的位置信息。
步骤S103：基于目标点的位置信息，确定跟踪目标对应的跟踪模板。
具体来讲，步骤S103，包括：
基于目标点的位置信息，在画面中确定目标点；在目标点周围生成多个备选跟踪模板；基于显著性算法，分别对多个备选跟踪模板进行分析，并从多个备选跟踪模板中选出一最佳备选跟踪模板，其中，最佳备选跟踪模板即为跟踪目标对应的跟踪模板。
在具体实施过程中，可以获取目标点、以及目标点附近其它像素点的颜色信息(例如：色调参数、亮度参数，或RBG参数)，并对这些像素点的颜色信息进行分析，由于构成跟踪目标的像素点可能具有相同或相近的颜色，所以可以将颜色相同或相近的像素点所组成的区域定义为一个备选跟踪模板，从而生成多个备选跟踪模板。
在具体实施过程中，可以对每个备选跟踪模板添加噪声和随机形变，生成每个备选跟踪模板的正样本；对每个备选跟踪模板周围的背景区域随机采
样，生成每个备选跟踪模板的负样本；基于每个备选跟踪模板的正样本和每个备选跟踪模板的负样本，生成每个备选跟踪模板的校验集。进一步，训练每个备选跟踪模板的跟踪模型；生成每个备选跟踪模板的校验集；再测试每个备选跟踪模板的跟踪模型在各自对应备选跟踪模板的检验集上的响应；基于正样本和负样本集上平均响应的比值，确定每个备选跟踪模板的跟踪模型的显著性；最后将显著性最高的跟踪模型对应的备选跟踪模板确定为最佳备选跟踪模板。
在具体实施过程中，每个备选跟踪模板都为矩形区域，这样最终得到的最佳备选跟踪模板也为矩形区域，从而满足了在线学习的视觉跟踪算法对跟踪模板的要求，进而实现了将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合。
步骤S104：基于跟踪目标对应的跟踪模板，利用视觉跟踪算法对跟踪目标进行视觉跟踪。
在具体实施过程中，视觉跟踪算法在一定时间内的具有很高的鲁棒性(具体受限于应用环境的复杂程度，大概在几十秒到十几分钟之间)，所以，视觉跟踪算法可以提供较高的自动化程度，在准确跟踪期间，无需用户控制，即可准确地对跟踪目标进行视觉跟踪。
在具体实施过程中，所述视觉跟踪算法是指在线学习的视觉跟踪算法，此处不限定具体的视觉跟踪算法，任何短期跟踪算法(short-term tracking)和长期跟踪算法(long-term tracking)都可以采用。但是，鉴于短期跟踪算法具有更好的鲁棒性且算法更加成熟，应尽量选择短期跟踪算法，例如，可以选择采用协同滤波器(Correlation Filter)的跟踪算法，从而获得更好的鲁棒性，并且降低算法的复杂度。
在具体实施过程中，在电子设备对跟踪目标进行视觉跟踪时，可以在摄像头采集到的图像数据中标记出跟踪目标，并将携带有标记的图像数据发送给控制设备。在控制设备输出图像数据对应的画面时，用户即可根据画面中的标记和跟踪目标的位置关系，判断电子设备是否出现目标跟丢的情况。例如，若画面中的标记与跟踪目标的位置重合，则表明没有跟丢；若画面中的标记与跟踪目标的位置不重合，甚至相距较远，则表明跟丢了。在用户发现电子设备出现目标跟丢的情况时，可以在控制设备的显示屏上重新点击跟踪目标的任一点，从而确定一新的目标点，以使电子设备基于新的目标点，重新确定跟踪目标对应的跟踪模板，并利用视觉跟踪算法对跟踪目标进行视觉跟踪。
在本实施例中，用户可以通过控制设备监视电子设备是否出现目标跟丢的情况，在出现目标跟丢的情况时，可以通过点击操作及时对目标点进行修
正，从而弥补了某些视觉跟踪算法(例如：长期跟踪算法)鲁棒性差的不足。
在本实施例中，将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合，仅需要用户点击跟踪目标上的一个目标点，即可获得跟踪目标对应的跟踪模板，从而利用在线学习的视觉跟踪算法对所述跟踪目标进行视觉跟踪。由于无需耗费过多时间，用户瞬间即可完成该点击操作，从而完成对跟踪模板的定义，具有操作方便、快捷的优点。所以有效地解决了在线学习的视觉跟踪算法，由于采用用户指定区域来定义跟踪模板的交互模式，耗时较长，存在无法对运动中的物体进行视觉跟踪的技术问题。实现了利用在线学习的视觉跟踪算法对运动中的物体进行视觉跟踪的技术效果。
作为一种可选的实施方式，在执行步骤S104过程中，还可以同时执行步骤S105。
步骤S105：控制电子设备对准目标点前进。
在具体实施过程中，可以根据目标点在画面中的位置，调整摄像头的俯仰角、以及控制电子设备转弯，使得目标点逐渐靠近画面的中心，从而控制电子设备对准目标点，再控制电子设备前进。
在具体实施过程中，可以基于等式(1)确定目标点与画面的中心在竖直方向上的竖直偏离角：
Picth＝arctan(dy/f)——等式(1)
其中，Picth为竖直偏离角，f为摄像头的焦距，dy为目标点到摄像头光轴的垂直偏移量；
再根据竖直偏离角Picth，调整摄像头的俯仰角，使竖直偏离角Picth逐渐减小趋于0，最终使目标点在竖直方向靠近画面的中心。
在具体实施过程中，可以基于等式(2)确定目标点与画面的中心在水平方向上的水平偏移角：
yaw＝arctan(dx/f)——等式(2)
其中，yaw为水平偏移角，f为摄像头的焦距，dx为目标点到摄像头光轴的水平偏移量；
再根据水平偏移角，确定电子设备的转动角速度；基于转动角速度，控制电子设备转动，使得水平偏移角yaw逐渐减小趋于0，最终使目标点在水平方向靠近画面的中心。
其中，转动角速度与水平偏移角成正比例关系，这样，在水平偏移角越大时，转动角速度也会越大，可以使目标点在水平方向快速靠近画面的中心。
在竖直偏离角Picth和水平偏移角yaw都为0时(或都接近0时)，目标点基
本就位于画面的中心点，此时，即可控制电子设备前进，逐渐靠近目标点，从而实现对目标点的跟踪行驶。
在具体实施过程中，可以根据跟踪目标对应的跟踪模板在画面中的尺寸，确定电子设备的前进速度；基于该前进速度，控制电子设备向跟踪目标前进。其中，可以使用sigmoid函数制定速度衰减模型。
在具体实施过程中，电子设备的前进速度与跟踪目标对应的跟踪模板在画面中的尺寸成反比例关系，这样，在跟踪目标对应的跟踪模板在画面中的尺寸越小时，说明跟踪目标与电子设备的距离越远，则控制电子设备的前进速度越快，从而控制电子设备快速靠近跟踪目标，而在跟踪目标对应的跟踪模板在画面中的尺寸越大时，说明跟踪目标与电子设备的距离越近，则控制电子设备的前进速度越慢，从而防止电子设备与跟踪目标碰撞。
在本实施例中，将摄像头采集到的图像数据发送给控制设备，实现了将电子设备的视野与用户共享，使得用户可以方便地在控制设备所输出的画面中确定目标点，进而使得电子设备可以基于目标点确定跟踪目标的跟踪模板，并利用视觉跟踪算法对跟踪目标进行视觉跟踪。在对跟踪目标进行视觉跟踪过程中，电子设备还可以自动对准目标点前进，这样，用户即可在控制设备输出的画面上点击任一目标点，来控制电子设备的行驶方向，操作方便，并且用户可以在控制设备上实时观看电子设备在行驶过程中的路况信息，并对电子设备的行驶方向进行调整。
在具体实施过程中，用户可以在控制设备的触摸屏上，连续点击画面中的不同目标点，从而遥控电子设备自由行驶。
基于同一发明构思，本实施例该提供了一种电子设备，该电子设备具有一摄像头，如图2所示，该电子设备，还包括：
发送单元201，用于将摄像头采集到的图像数据发送给控制设备；
接收单元202，用于接收控制设备发来的目标点的位置信息，其中，目标点为在控制设备通过显示屏输出图像数据的画面时用户在画面上选中的点，目标点为跟踪目标上的任一点；
确定单元203，用于基于目标点的位置信息，确定跟踪目标对应的跟踪模板；
跟踪单元204，用于基于跟踪目标对应的跟踪模板，利用视觉跟踪算法对跟踪目标进行视觉跟踪。
作为一种可选的实施方式，确定单元203，包括：
第一确定模块，用于基于目标点的位置信息，在画面中确定目标点；
生成模块，用于在目标点周围生成多个备选跟踪模板；
分析模块，用于基于显著性算法，分别对多个备选跟踪模板进行分析，
并从多个备选跟踪模板中选出一最佳备选跟踪模板，其中，最佳备选跟踪模板即为跟踪目标对应的跟踪模板。
作为一种可选的实施方式，电子设备还包括：
控制单元205，用于在基于跟踪模板，利用视觉跟踪算法对跟踪目标进行视觉跟踪过程中，控制电子设备对准目标点前进，其中，电子设备为可移动电子设备。
作为一种可选的实施方式，控制单元205包括：
第二确定模块，用于基于等式Picth＝arctan(dy/f)，确定目标点与画面的中心在竖直方向上的竖直偏离角；其中，Picth为竖直偏离角，f为摄像头的焦距，dy为目标点到摄像头光轴的垂直偏移量；
调整模块，用于根据竖直偏离角，调整摄像头的俯仰角，以使目标点在竖直方向靠近画面的中心。
作为一种可选的实施方式，控制单元205，包括：
第三确定模块，用于基于等式yaw＝arctan(dx/f)，确定目标点与画面的中心在水平方向上的水平偏移角；其中，yaw为水平偏移角，f为摄像头的焦距，dx为目标点到摄像头光轴的水平偏移量；
第四确定模块，用于根据水平偏移角，确定电子设备的转动角速度；
第一控制模块，用于基于转动角速度，控制电子设备转动，以使目标点在水平方向靠近画面的中心。
作为一种可选的实施方式，控制单元205，包括：
第五确定模块，用于根据跟踪目标对应的跟踪模板在画面中的尺寸，确定电子设备的前进速度；
第二控制模块，用于基于前进速度，控制电子设备向跟踪目标前进。
由于本实施例所介绍的电子设备为实施本发明实施例中目标跟踪方法所采用的电子设备，故而基于本发明实施例中所介绍的目标跟踪方法，本领域所属技术人员能够了解本实施例的电子设备的具体实施方式以及其各种变化形式，所以在此对于该电子设备如何实现本发明实施例中的方法不再详细介绍。只要本领域所属技术人员实施本发明实施例中目标跟踪方法所采用的电子设备，都属于本发明所欲保护的范围。
上述本发明实施例中的技术方案，至少具有如下的技术效果或优点：
1、在本发明实施例中，将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合，仅需要用户点击跟踪目标上的一个目标点，即可获得跟踪目标对应的跟踪模板，从而利用在线学习的视觉跟踪算法对所述跟踪目标进行视觉跟踪。由于无需耗费过多时间，用户瞬间即可完成该点击操作，从而完成对跟踪模板的定义，具有操作方便、快捷的优点。
所以有效地解决了在线学习的视觉跟踪算法，由于采用用户指定区域来定义跟踪模板的交互模式，耗时较长，存在无法对运动中的物体进行视觉跟踪的技术问题。实现了将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合，来对运动中的物体进行视觉跟踪的技术效果。
2、在本发明实施例中，用户可以通过控制设备监视电子设备是否出现目标跟丢的情况，在出现目标跟丢的情况时，可以通过点击操作及时对目标点进行修正，从而弥补了某些视觉跟踪算法(例如：长期跟踪算法)鲁棒性差的不足。
3、在本发明实施例中，将摄像头采集到的图像数据发送给控制设备，实现了将电子设备的视野与用户共享，使得用户可以方便地在控制设备所输出的画面中确定目标点，进而使得电子设备可以基于目标点确定跟踪目标的跟踪模板，并利用视觉跟踪算法对跟踪目标进行视觉跟踪。且在对跟踪目标进行视觉跟踪过程中，电子设备还可以自动对准目标点前进，这样，用户即可在控制设备输出的画面上点击任一目标点，来控制电子设备的行驶方向，同时，用户可以在控制设备上实时观看电子设备在行驶过程中的路况信息，并对电子设备的行驶方向进行调整，具有操控方便的优点。
实施例二
本实施例提供了一种目标跟踪方法，应用于自平衡车中，所述自平衡车具有一摄像头，如图3所示，所述目标跟踪方法，包括：
步骤S301：将摄像头采集到的图像数据发送给控制设备。
在具体实施过程中，所述自平衡车主要有独轮和双轮两类，其运作原理主要是建立在一种被称为“动态稳定”(Dynamic Stabilization)的基本原理上，利用车体内部的陀螺仪和加速度传感器，来检测车体姿态的变化，并利用伺服控制系统，精确地驱动电机进行相应的调整，以保持系统的平衡。自平衡车中包括一套动力系统，能够驱动自平衡车在路面上行驶(例如：直行，转弯，倒退，曲线行驶，等等)。
在具体实施过程中，在自平衡车上设置有一摄像头，用于采集图像信息。为了获得更为广阔的图像采集视角，摄像头应尽量设置在靠近自平衡车顶部的位置。
在具体实施过程中，控制设备可以与自平衡车通过有线方式或无线方式连接，以使得控制设备可以与自平衡车进行通信。在自平衡车通过摄像头采集到图像数据后，会将采集到的图像数据实时发送给控制设备。
在具体实施过程中，该控制设备可以是与自平衡车为相互独立的设备(例
如：智能手机、或平板电脑、或笔记本电脑、或台式机电脑、或智能电视、或智能钥匙等等)；或者，该控制设备还可以直接设置在自平衡车上，作为自平衡车的一部分。该控制设备在为智能钥匙时，可以控制自平衡车锁车、开锁，或开启防盗报警。
步骤S302：接收控制设备发来的目标点的位置信息，其中，目标点为在控制设备通过显示屏输出图像数据的画面时用户在画面上选中的点。
在具体实施过程中，控制设备接收到自平衡车发来的图像数据后，可以通过一触摸屏(例如：电容式触摸屏、或电阻式触摸屏)输出该图像数据对应的画面(即：传图画面)，从而实现了将自平衡车的视野与用户共享。用户可以在控制设备输出的画面上寻找一跟踪目标(该跟踪目标可以是静止的物体，也可以是运动的物体)，并通过点击操作点击跟踪目标上的任意一个像素点，这个像素点就是所述目标点。控制设备获取到用户的点击操作后，即可将用户点击的目标点的位置信息(例如：坐标信息)发送给自平衡车。对应地，自平衡车接收控制设备发来的目标点的位置信息。
步骤S303：基于目标点的位置信息，确定跟踪目标对应的跟踪模板。
具体来讲，步骤S303，包括：
基于目标点的位置信息，在画面中确定目标点；在目标点周围生成多个备选跟踪模板；基于显著性算法，分别对多个备选跟踪模板进行分析，并从多个备选跟踪模板中选出一最佳备选跟踪模板，其中，最佳备选跟踪模板即为跟踪目标对应的跟踪模板。
在具体实施过程中，可以获取目标点、以及目标点附近其它像素点的颜色信息(例如：色调参数、亮度参数，或RBG参数)，并对这些像素点的颜色信息进行分析，由于构成跟踪目标的像素点可能具有相同或相近的颜色，所以可以将颜色相同或相近的像素点所组成的区域定义为一个备选跟踪模板，从而生成多个备选跟踪模板。
在具体实施过程中，可以对每个备选跟踪模板添加噪声和随机形变，生成每个备选跟踪模板的正样本；对每个备选跟踪模板周围的背景区域随机采样，生成每个备选跟踪模板的负样本；基于每个备选跟踪模板的正样本和每个备选跟踪模板的负样本，生成每个备选跟踪模板的校验集。进一步，训练每个备选跟踪模板的跟踪模型；生成每个备选跟踪模板的校验集；再测试每个备选跟踪模板的跟踪模型在各自对应备选跟踪模板的检验集上的响应；基于正样本和负样本集上平均响应的比值，确定每个备选跟踪模板的跟踪模型的显著性；最后将显著性最高的跟踪模型对应的备选跟踪模板确定为最佳备选跟踪模板。
在具体实施过程中，每个备选跟踪模板都为矩形区域，这样最终得到的
最佳备选跟踪模板也为矩形区域，从而满足了在线学习的视觉跟踪算法对跟踪模板的要求，进而实现了将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合。
步骤S304：基于跟踪目标对应的跟踪模板，利用视觉跟踪算法对跟踪目标进行视觉跟踪。
在具体实施过程中，视觉跟踪算在一定时间内的具有很高的鲁棒性(具体受限于应用环境的复杂程度，大概在几十秒到十几分钟之间)，所以，视觉跟踪算法可以提供较高的自动化程度，在准确跟踪期间，无需用户控制，即可准确地对跟踪目标进行视觉跟踪。
在具体实施过程中，所述视觉跟踪算法是指在线学习的视觉跟踪算法，此处不限定具体的视觉跟踪算法，任何短期跟踪算法(short-term tracking)和长期跟踪算法(long-term tracking)都可以采用。但是，鉴于短期跟踪算法具有更好的鲁棒性且算法更加成熟，应尽量选择短期跟踪算法，例如，可以选择采用协同滤波器(Correlation Filter)的视觉跟踪算法，从而获得更好的鲁棒性，并且降低算法的复杂度。
在具体实施过程中，在自平衡车对跟踪目标进行视觉跟踪时，可以在摄像头采集到的图像数据中标记出该跟踪目标，并将携带有标记的图像数据发送给控制设备。在控制设备输出图像数据对应的画面时，用户即可根据画面中的标记和跟踪目标的位置关系，判断自平衡车是否出现目标跟丢的情况。例如，若画面中的标记与跟踪目标的位置重合，则表明没有跟丢；若画面中的标记与跟踪目标的位置不重合，甚至相距较远，则表明跟丢了。在用户发现自平衡车出现目标跟丢的情况时，可以在控制设备的显示屏上重新点击跟踪目标的任一点，从而确定一新的目标点，以使自平衡车基于新的目标点，重新确定跟踪目标对应的跟踪模板，并利用视觉跟踪算法对跟踪目标进行视觉跟踪。
在本实施例中，用户可以通过控制设备监视自平衡车是否出现目标跟丢的情况，在出现目标跟丢的情况时，可以通过点击操作及时对目标点进行修正，从而弥补了某些视觉跟踪算法(例如：长期跟踪算法)鲁棒性差的不足。
在本实施例中，将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合，仅需要用户点击跟踪目标上的一个目标点，即可获得跟踪目标对应的跟踪模板，从而利用在线学习的视觉跟踪算法对所述跟踪目标进行视觉跟踪。由于无需耗费过多时间，用户瞬间即可完成该点击操作，从而完成对跟踪模板的定义，具有操作方便、快捷的优点。所以有效地解决了在线学习的视觉跟踪算法，由于采用用户指定区域来定义跟踪模板的交互模式，耗时较长，存在无法对运动中的物体进行视觉跟踪的技术问
题。实现了利用在线学习的视觉跟踪算法对运动中的物体进行视觉跟踪的技术效果。
作为一种可选的实施方式，在执行步骤S304过程中，还可以同时执行步骤S305。
步骤S305：控制自平衡车对准目标点前进。
在具体实施过程中，可以根据目标点在画面中的位置，调整摄像头的俯仰角、以及控制自平衡车转弯，使得目标点逐渐靠近画面的中心，从而控制自平衡车对准目标点，再控制自平衡车前进。
在具体实施过程中，可以基于等式(1)确定目标点与画面的中心在竖直方向上的竖直偏离角：
Picth＝arctan(dy/f)——等式(1)
其中，Picth为竖直偏离角，f为摄像头的焦距，dy为目标点到摄像头光轴的垂直偏移量；
再根据竖直偏离角Picth，调整摄像头的俯仰角，使竖直偏离角Picth逐渐减小趋于0，最终使目标点在竖直方向靠近画面的中心。
在具体实施过程中，可以基于等式(2)确定目标点与画面的中心在水平方向上的水平偏移角：
yaw＝arctan(dx/f)——等式(2)
其中，yaw为水平偏移角，f为摄像头的焦距，dx为目标点到摄像头光轴的水平偏移量；
再根据水平偏移角，确定自平衡车的转动角速度；基于转动角速度，控制自平衡车转动，使得水平偏移角yaw逐渐减小趋于0，最终使目标点在水平方向靠近画面的中心。
其中，转动角速度与水平偏移角成正比例关系，这样，在水平偏移角越大时，转动角速度也会越大，可以使目标点在水平方向快速靠近画面的中心。
在竖直偏离角Picth和水平偏移角yaw都为0时(或都接近0时)，目标点基本就位于画面的中心点，此时，即可控制自平衡车前进，逐渐靠近目标点，从而实现对目标点的跟踪行驶。
在具体实施过程中，可以根据跟踪目标对应的跟踪模板在画面中的尺寸，确定自平衡车的前进速度；基于该前进速度，控制自平衡车向跟踪目标前进。其中，可以使用sigmoid函数制定速度衰减模型。
在具体实施过程中，自平衡车的前进速度与跟踪目标对应的跟踪模板在画面中的尺寸成反比例关系，这样，在跟踪目标对应的跟踪模板在画面中的尺寸越小时，说明跟踪目标与自平衡车的距离越远，则控制自平衡车的前进
速度越快，从而控制自平衡车快速靠近跟踪目标，而在跟踪目标对应的跟踪模板在画面中的尺寸越大时，说明跟踪目标与自平衡车的距离越近，则控制自平衡车的前进速度越慢，从而防止自平衡车与跟踪目标碰撞。
在本实施例中，将摄像头采集到的图像数据发送给控制设备，实现了将自平衡车的视野与用户共享，使得用户可以方便地在控制设备所输出的画面中确定目标点，进而使得自平衡车可以基于目标点确定跟踪目标的跟踪模板，并利用视觉跟踪算法对跟踪目标进行视觉跟踪。在对跟踪目标进行视觉跟踪过程中，自平衡车还可以自动对准目标点前进，这样，用户即可在控制设备输出的画面上点击任一目标点，来控制自平衡车的行驶方向，操作方便，并且用户可以在控制设备上实时观看自平衡车在行驶过程中的路况信息，并对自平衡车的行驶方向进行调整。
在具体实施过程中，用户可以在控制设备的触摸屏上，连续点击画面中的不同目标点，从而遥控自平衡车自由行驶。
基于同一发明构思，本实施例该提供了一种自平衡车，该自平衡车具有一摄像头，该自平衡车还包括图2中所示的发送单元201、接收单元202、确定单元203和跟踪单元204。其中，
发送单元201，用于将摄像头采集到的图像数据发送给控制设备；
接收单元202，用于接收控制设备发来的目标点的位置信息，其中，目标点为在控制设备通过显示屏输出图像数据的画面时用户在画面上选中的点，目标点为跟踪目标上的任一点；
确定单元203，用于基于目标点的位置信息，确定跟踪目标对应的跟踪模板；
跟踪单元204，用于基于跟踪目标对应的跟踪模板，利用视觉跟踪算法对跟踪目标进行视觉跟踪。
作为一种可选的实施方式，确定单元203，包括：
第一确定模块，用于基于目标点的位置信息，在画面中确定目标点；
生成模块，用于在目标点周围生成多个备选跟踪模板；
分析模块，用于基于显著性算法，分别对多个备选跟踪模板进行分析，并从多个备选跟踪模板中选出一最佳备选跟踪模板，其中，最佳备选跟踪模板即为跟踪目标对应的跟踪模板。
作为一种可选的实施方式，自平衡车，还包括：
控制单元205，用于在基于跟踪模板，利用视觉跟踪算法对跟踪目标进行视觉跟踪过程中，控制自平衡车对准目标点前进，其中，自平衡车为可移动自平衡车。
作为一种可选的实施方式，控制单元205包括：
第二确定模块，用于基于等式Picth＝arctan(dy/f)，确定目标点与画面的中心在竖直方向上的竖直偏离角；其中，Picth为竖直偏离角，f为摄像头的焦距，dy为目标点到摄像头光轴的垂直偏移量；
调整模块，用于根据竖直偏离角，调整摄像头的俯仰角，以使目标点在竖直方向靠近画面的中心。
作为一种可选的实施方式，控制单元205，包括：
第三确定模块，用于基于等式yaw＝arctan(dx/f)，确定目标点与画面的中心在水平方向上的水平偏移角；其中，yaw为水平偏移角，f为摄像头的焦距，dx为目标点到摄像头光轴的水平偏移量；
第四确定模块，用于根据水平偏移角，确定自平衡车的转动角速度；
第一控制模块，用于基于转动角速度，控制自平衡车转动，以使目标点在水平方向靠近画面的中心。
作为一种可选的实施方式，控制单元205，包括：
第五确定模块，用于根据跟踪目标对应的跟踪模板在画面中的尺寸，确定自平衡车的前进速度；
第二控制模块，用于基于前进速度，控制自平衡车向跟踪目标前进。
由于本实施例所介绍的自平衡车为实施本发明实施例中目标跟踪方法所采用的可移动电子设备，故而基于本发明实施例中所介绍的目标跟踪方法，本领域所属技术人员能够了解本实施例的自平衡车的具体实施方式以及其各种变化形式，所以在此对于该自平衡车如何实现本发明实施例中的方法不再详细介绍。只要本领域所属技术人员实施本发明实施例中目标跟踪方法所采用的自平衡车，都属于本发明所欲保护的范围。
上述本发明实施例中的技术方案，至少具有如下的技术效果或优点：
1、在本发明实施例中，将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合，仅需要用户点击跟踪目标上的一个目标点，即可获得跟踪目标对应的跟踪模板，从而利用在线学习的视觉跟踪算法对所述跟踪目标进行视觉跟踪。由于无需耗费过多时间，用户瞬间即可完成该点击操作，从而完成对跟踪模板的定义，具有操作方便、快捷的优点。所以有效地解决了在线学习的视觉跟踪算法，由于采用用户指定区域来定义跟踪模板的交互模式，耗时较长，存在无法对运动中的物体进行视觉跟踪的技术问题。实现了将用户通过点击操作来定义跟踪模板的这种交互模式与在线学习的视觉跟踪算法相融合，来对运动中的物体进行视觉跟踪的技术效果。
2、在本发明实施例中，用户可以通过控制设备监视自平衡车是否出现目标跟丢的情况，在出现目标跟丢的情况时，可以通过点击操作及时对目标点进行修正，从而弥补了某些视觉跟踪算法(例如：长期跟踪算法)鲁棒性差
的不足。
3、在本发明实施例中，将摄像头采集到的图像数据发送给控制设备，实现了将自平衡车的视野与用户共享，使得用户可以方便地在控制设备所输出的画面中确定目标点，进而使得自平衡车可以基于目标点确定跟踪目标的跟踪模板，并利用视觉跟踪算法对跟踪目标进行视觉跟踪。且在对跟踪目标进行视觉跟踪过程中，自平衡车还可以自动对准目标点前进，这样，用户即可在控制设备输出的画面上点击任一目标点，来控制自平衡车的行驶方向，同时，用户可以在控制设备上实时观看自平衡车在行驶过程中的路况信息，并对自平衡车的行驶方向进行调整，具有操控方便的优点。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：移动存储设备、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
或者，本发明实施例上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明实施例的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机、服务器、或者网络设备等)执行本发明各个实施例所述方法的全部或部分。而前述的存储介质包括：移动存储设备、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
鉴于此，本发明实施例还提供了一种计算机可读存储介质，所述存储介质包括一组计算机可执行指令，所述指令用于执行本发明实施例所述的移动电子设备的控制方法。
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功
能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
尽管已描述了本发明的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例作出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本发明范围的所有变更和修改。
显然，本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的精神和范围。这样，倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也意图包含这些改动和变型在内。
Claims (13)
- 一种目标跟踪方法，应用于电子设备中，所述电子设备具有一摄像头，其中，所述目标跟踪方法包括：将所述摄像头采集到的图像数据发送给控制设备；接收所述控制设备发来的目标点的位置信息，其中，所述目标点为在所述控制设备通过显示屏输出所述图像数据的画面时用户在所述画面上选中的点，所述目标点为跟踪目标上的任一点；基于所述目标点的位置信息，确定所述跟踪目标对应的跟踪模板；基于所述跟踪目标对应的跟踪模板，利用视觉跟踪算法对所述跟踪目标进行视觉跟踪。
- 如权利要求1所述的目标跟踪方法，其中，所述基于所述目标点的位置信息，确定所述跟踪目标对应的跟踪模板，包括：基于所述目标点的位置信息，在所述画面中确定所述目标点；在所述目标点周围生成多个备选跟踪模板；基于显著性算法，分别对所述多个备选跟踪模板进行分析，并从所述多个备选跟踪模板中选出一最佳备选跟踪模板，其中，所述最佳备选跟踪模板即为所述跟踪目标对应的跟踪模板。
- 如权利要求1或2所述的目标跟踪方法，其中，在所述基于所述跟踪模板，利用视觉跟踪算法对所述跟踪目标进行视觉跟踪过程中，还包括：控制所述电子设备对准所述目标点前进，其中，所述电子设备为可移动电子设备。
- 如权利要求3所述的目标跟踪方法，其中，所述控制所述电子设备对准所述目标点前进，包括：基于等式Picth＝arctan(dy/f)，确定所述目标点与所述画面的中心在竖直方向上的竖直偏离角；其中，Picth为所述竖直偏离角，f为所述摄像头的焦距，dy为所述目标点到所述摄像头光轴的垂直偏移量；根据所述竖直偏离角，调整所述摄像头的俯仰角，以使所述目标点在竖直方向靠近所述画面的中心。
- 如权利要求3所述的目标跟踪方法，其中，所述控制所述电子设备对准所述目标点前进，包括：基于等式yaw＝arctan(dx/f)，确定所述目标点与所述画面的中心在水平方向上的水平偏移角；其中，yaw为所述水平偏移角，f为所述摄像头的焦距，dx为所述目标点到所述摄像头光轴的水平偏移量；根据所述水平偏移角，确定所述电子设备的转动角速度；基于所述转动角速度，控制所述电子设备转动，以使所述目标点在水平 方向靠近所述画面的中心。
- 如权利要求3所述的目标跟踪方法，其中，所述控制所述电子设备对准所述目标点前进，包括：根据所述跟踪目标对应的跟踪模板在所述画面中的尺寸，确定所述电子设备的前进速度；基于所述前进速度，控制所述电子设备向所述跟踪目标前进。
- 一种电子设备，所述电子设备具有一摄像头，其中，所述电子设备，还包括：发送单元，配置为将所述摄像头采集到的图像数据发送给控制设备；接收单元，配置为接收所述控制设备发来的目标点的位置信息，其中，所述目标点为在所述控制设备通过显示屏输出所述图像数据的画面时用户在所述画面上选中的点，所述目标点为跟踪目标上的任一点；确定单元，配置为基于所述目标点的位置信息，确定所述跟踪目标对应的跟踪模板；跟踪单元，配置为基于所述跟踪目标对应的跟踪模板，利用视觉跟踪算法对所述跟踪目标进行视觉跟踪。
- 如权利要求7所述的电子设备，其中，所述确定单元，包括：第一确定模块，配置为基于所述目标点的位置信息，在所述画面中确定所述目标点；生成模块，配置为在所述目标点周围生成多个备选跟踪模板；分析模块，配置为基于显著性算法，分别对所述多个备选跟踪模板进行分析，并从所述多个备选跟踪模板中选出一最佳备选跟踪模板，其中，所述最佳备选跟踪模板即为所述跟踪目标对应的跟踪模板。
- 如权利要求7或8所述的电子设备，其中，所述电子设备，还包括：控制单元，配置为在所述基于所述跟踪模板，利用视觉跟踪算法对所述跟踪目标进行视觉跟踪过程中，控制所述电子设备对准所述目标点前进，其中，所述电子设备为可移动电子设备。
- 如权利要求9所述的电子设备，其中，所述控制单元，包括：第二确定模块，配置为基于等式Picth＝arctan(dy/f)，确定所述目标点与所述画面的中心在竖直方向上的竖直偏离角；其中，Picth为所述竖直偏离角，f为所述摄像头的焦距，dy为所述目标点到所述摄像头光轴的垂直偏移量；调整模块，配置为根据所述竖直偏离角，调整所述摄像头的俯仰角，以使所述目标点在竖直方向靠近所述画面的中心。
- 如权利要求9所述的电子设备，其中，所述控制单元，包括：第三确定模块，配置为基于等式yaw＝arctan(dx/f)，确定所述目标点与 所述画面的中心在水平方向上的水平偏移角；其中，yaw为所述水平偏移角，f为所述摄像头的焦距，dx为所述目标点到所述摄像头光轴的水平偏移量；第四确定模块，配置为根据所述水平偏移角，确定所述电子设备的转动角速度；第一控制模块，配置为基于所述转动角速度，控制所述电子设备转动，以使所述目标点在水平方向靠近所述画面的中心。
- 如权利要求9所述的电子设备，其中，所述控制单元，包括：第五确定模块，配置为根据所述跟踪目标对应的跟踪模板在所述画面中的尺寸，确定所述电子设备的前进速度；第二控制模块，配置为基于所述前进速度，控制所述电子设备向所述跟踪目标前进。
- 一种计算机存储介质，所述计算机存储介质中存储有计算机可执行指令，所述计算机可执行指令配置为执行权利要求1-6任一项所述的目标跟踪方法。
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610201474.1A CN105931263B (zh) | 2016-03-31 | 2016-03-31 | 一种目标跟踪方法及电子设备 |
| CN201610201474.1 | 2016-03-31 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2017167282A1 true WO2017167282A1 (zh) | 2017-10-05 |
Family
ID=56840484
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2017/079030 WO2017167282A1 (zh) | 2016-03-31 | 2017-03-31 | 一种目标跟踪方法及电子设备、计算机存储介质 |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN105931263B (zh) |
| WO (1) | WO2017167282A1 (zh) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108320298A (zh) * | 2018-04-28 | 2018-07-24 | 亮风台（北京）信息科技有限公司 | 一种视觉目标跟踪方法与设备 |
| CN110490155A (zh) * | 2019-08-23 | 2019-11-22 | 电子科技大学 | 一种禁飞空域无人机检测方法 |
| CN116196109A (zh) * | 2023-04-27 | 2023-06-02 | 北京碧莲盛不剃发植发医疗美容门诊部有限责任公司 | 基于图像识别的不剃发植发机械手 |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105931263B (zh) * | 2016-03-31 | 2019-09-20 | 纳恩博（北京）科技有限公司 | 一种目标跟踪方法及电子设备 |
| WO2018058305A1 (en) | 2016-09-27 | 2018-04-05 | SZ DJI Technology Co., Ltd. | System and method for controlling unmaned vehicle with presence of live object |
| CN106683123B (zh) * | 2016-10-31 | 2019-04-02 | 纳恩博（北京）科技有限公司 | 一种目标跟踪方法及目标跟踪装置 |
| CN106649825B (zh) * | 2016-12-29 | 2020-03-24 | 上海智臻智能网络科技股份有限公司 | 语音交互系统及其创建方法和装置 |
| CN106672345B (zh) * | 2016-12-30 | 2019-08-23 | 华南智能机器人创新研究院 | 一种工业机器人自动分拣的方法及系统 |
| CN108534771A (zh) * | 2017-03-06 | 2018-09-14 | 常州爱尔威智能科技有限公司 | 智能视觉追踪设备及其追踪选定目标的方法 |
| CN108874269B (zh) * | 2017-05-12 | 2020-12-29 | 北京臻迪科技股份有限公司 | 一种目标跟踪方法、装置及系统 |
| CN107577245A (zh) * | 2017-09-18 | 2018-01-12 | 深圳市道通科技股份有限公司 | 一种飞行器参数设定方法和装置及计算机可读存储介质 |
| CN107995435A (zh) * | 2017-12-11 | 2018-05-04 | 深圳市北航旭飞科技有限公司 | 无人机图像采集方法、装置、计算机设备及可读存储介质 |
| CN110120100B (zh) * | 2018-02-06 | 2023-07-14 | 广东虚拟现实科技有限公司 | 图像处理方法、装置及识别跟踪系统 |
| CN110347035A (zh) * | 2018-04-08 | 2019-10-18 | 北京京东尚科信息技术有限公司 | 自主跟踪方法及装置、电子设备、存储介质 |
| CN110291775B (zh) * | 2018-05-29 | 2021-07-06 | 深圳市大疆创新科技有限公司 | 一种跟踪拍摄方法、设备及存储介质 |
| CN108924494B (zh) * | 2018-07-13 | 2020-08-28 | 王新凤 | 基于地面的空中监测系统 |
| CN109360225B (zh) * | 2018-10-16 | 2020-12-18 | 北京理工大学 | 一种运动模型的优化系统及方法 |
| CN111386507A (zh) * | 2019-02-01 | 2020-07-07 | 深圳市大疆创新科技有限公司 | 数据处理方法、无人机、移动设备及系统 |
| CN110587621B (zh) * | 2019-08-30 | 2023-06-06 | 深圳智慧林网络科技有限公司 | 机器人、基于机器人的病人看护方法和可读存储介质 |
| CN110689556A (zh) * | 2019-09-09 | 2020-01-14 | 苏州臻迪智能科技有限公司 | 跟踪方法、装置及智能设备 |
| CN110930428B (zh) * | 2020-02-19 | 2020-08-14 | 成都纵横大鹏无人机科技有限公司 | 目标跟踪方法、装置、电子设备及存储介质 |
| CN113341971A (zh) * | 2021-06-01 | 2021-09-03 | 阿尔特汽车技术股份有限公司 | 一种自动跟踪行走的高尔夫车及自动跟踪行走的控制方法 |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102074016A (zh) * | 2009-11-24 | 2011-05-25 | 杭州海康威视软件有限公司 | 运动目标自动跟踪的装置和方法 |
| US20140205140A1 (en) * | 2013-01-24 | 2014-07-24 | Kineticor, Inc. | Systems, devices, and methods for tracking moving targets |
| CN105931263A (zh) * | 2016-03-31 | 2016-09-07 | 纳恩博（北京）科技有限公司 | 一种目标跟踪方法及电子设备 |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411368B (zh) * | 2011-07-22 | 2013-10-09 | 北京大学 | 机器人的主动视觉人脸跟踪方法和跟踪系统 |
| CN104700415B (zh) * | 2015-03-23 | 2018-04-24 | 华中科技大学 | 一种图像匹配跟踪中匹配模板的选取方法 |
-
2016
- 2016-03-31 CN CN201610201474.1A patent/CN105931263B/zh active Active
-
2017
- 2017-03-31 WO PCT/CN2017/079030 patent/WO2017167282A1/zh active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102074016A (zh) * | 2009-11-24 | 2011-05-25 | 杭州海康威视软件有限公司 | 运动目标自动跟踪的装置和方法 |
| US20140205140A1 (en) * | 2013-01-24 | 2014-07-24 | Kineticor, Inc. | Systems, devices, and methods for tracking moving targets |
| CN105931263A (zh) * | 2016-03-31 | 2016-09-07 | 纳恩博（北京）科技有限公司 | 一种目标跟踪方法及电子设备 |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108320298A (zh) * | 2018-04-28 | 2018-07-24 | 亮风台（北京）信息科技有限公司 | 一种视觉目标跟踪方法与设备 |
| CN110490155A (zh) * | 2019-08-23 | 2019-11-22 | 电子科技大学 | 一种禁飞空域无人机检测方法 |
| CN110490155B (zh) * | 2019-08-23 | 2022-05-17 | 电子科技大学 | 一种禁飞空域无人机检测方法 |
| CN116196109A (zh) * | 2023-04-27 | 2023-06-02 | 北京碧莲盛不剃发植发医疗美容门诊部有限责任公司 | 基于图像识别的不剃发植发机械手 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105931263A (zh) | 2016-09-07 |
| CN105931263B (zh) | 2019-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2017167282A1 (zh) | 一种目标跟踪方法及电子设备、计算机存储介质 | |
| US10970859B2 (en) | Monitoring method and device for mobile target, monitoring system and mobile robot | |
| US11915502B2 (en) | Systems and methods for depth map sampling | |
| Filipenko et al. | Comparison of various slam systems for mobile robot in an indoor environment | |
| Guindel et al. | Automatic extrinsic calibration for lidar-stereo vehicle sensor setups | |
| Bounini et al. | Autonomous vehicle and real time road lanes detection and tracking | |
| EP3980969A1 (en) | Cross-modal sensor data alignment | |
| Bakthavatchalam et al. | Photometric moments: New promising candidates for visual servoing | |
| US12047674B2 (en) | System for generating a three-dimensional scene of a physical environment | |
| CN110148177A (zh) | 用于确定相机的姿态角的方法、装置、计算设备、计算机可读存储介质以及采集实体 | |
| CN116659518B (zh) | 智能轮椅自主导航方法、装置、终端及介质 | |
| Cai et al. | Gaze estimation driven solution for interacting children with ASD | |
| Frank et al. | Using tablets in the vision-based control of a ball and beam test-bed | |
| US20230091536A1 (en) | Camera Placement Guidance | |
| US11763492B1 (en) | Apparatus and methods to calibrate a stereo camera pair | |
| Zheng et al. | Vision-based autonomous vehicle control using the two-point visual driver control model | |
| Santos et al. | Indoor waypoint UAV navigation using a RGB-D system | |
| Yeh et al. | Research on intuitive gesture recognition control and navigation system of UAV | |
| Lakshan | Blind navigation in outdoor environments: Head and torso level thin-structure based obstacle detection | |
| Liu | Intelligent Patrol Robot Based on Visual Machine Deep Learning | |
| Madani et al. | Machine-awareness in indoor environment: A pseudo-3D vision-based approach combining multi-resolution visual information | |
| Nguyen | Use of Image Processing Technology to Identify the Position of Indoor Robot | |
| Kulas et al. | Image-based path following and target tracking control for a mobile robot | |
| Shourove et al. | Advanced artificial intelligence self balanced robot for home security | |
| Salih et al. | Using Monocular Vision for Navigation of LEGO Robot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17773312 Country of ref document: EP Kind code of ref document: A1 |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 17773312 Country of ref document: EP Kind code of ref document: A1 |