CN105185140A - Auxiliary driving method and system - Google Patents

Auxiliary driving method and system Download PDF

Info

Publication number
CN105185140A
CN105185140A CN201510641531.3A CN201510641531A CN105185140A CN 105185140 A CN105185140 A CN 105185140A CN 201510641531 A CN201510641531 A CN 201510641531A CN 105185140 A CN105185140 A CN 105185140A
Authority
CN
China
Prior art keywords
vehicle
traffic lights
event
image
driver
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201510641531.3A
Other languages
Chinese (zh)
Other versions
CN105185140B (en
Inventor
黄修源
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SHANGHAI XIUYUAN NETWORK TECHNOLOGY Co Ltd
Original Assignee
SHANGHAI XIUYUAN NETWORK TECHNOLOGY Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SHANGHAI XIUYUAN NETWORK TECHNOLOGY Co Ltd filed Critical SHANGHAI XIUYUAN NETWORK TECHNOLOGY Co Ltd
Priority to CN201510641531.3A priority Critical patent/CN105185140B/en
Publication of CN105185140A publication Critical patent/CN105185140A/en
Application granted granted Critical
Publication of CN105185140B publication Critical patent/CN105185140B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Traffic Control Systems (AREA)
  • Navigation (AREA)

Abstract

The invention discloses an auxiliary driving method and system. The method comprises: detecting the distance between the position of a vehicle and that of a traffic light, and if the distance is less than a first threshold, obtaining an image of the front of the vehicle; analyzing the image and judging whether the traffic light is in the image; if the traffic light is in the image, extracting related information of the traffic light including the color of the traffic light; and responding to a detected first event which happens, and prompting a driver through a vibration device of a steering wheel of the vehicle or voice, wherein the first event is that the color of the traffic light changes from red to green.

Description

A kind of auxiliary driving method and system
Technical field
The present invention relates to the auxiliary driving of vehicle.
Background technology
In road traffic, traffic lights are used to control traffic order and flow.In most countries, when traffic lights are shown as red light, instruction vehicle must not cross stop line, and be shown as green light and then indicate vehicle can by stop line, amber light becomes the transition in red light process as green light in addition.But, the driver of vehicle usually can not make traffic lights due to a variety of causes and correctly and timely reacting.Such as, driver may due to absent minded, the speed of a motor vehicle is too fast or sight line is blocked and makes a dash across the red light, or not start in time after red light becomes green light.These situations gently then can reduce traffic efficiency, heavy then traffic hazard can occur.
DAS (Driver Assistant System) can help driver to make traffic lights correctly and timely to judge.Such as, following technical scheme is disclosed in Japanese Patent Laid-Open 2004-252718 publication, namely, from road side equipment, mobile unit is sent to the signal of the lighting lamp state representing teleseme, when the mobile unit that have received this signal receives the signal of lighting lamp state representing red signal, to vehicle driver carry out urging stop inform action.But, this technical scheme needs to configure corresponding wireless signal transmitting device separately on traffic lights, onboard system also needs to have the ability with traffic lights radio communication simultaneously, therefore the enforcement of this technical scheme needs to carry out large-scale redevelopment to existing traffic lights, except increasing great amount of cost, this construction reformation is also unrealistic in many circumstances.
Therefore, the auxiliary driving solution of a kind of low cost, high intelligence is expected.
Summary of the invention
For this reason, the invention provides a kind of new scheme to try hard to solve or at least alleviate Problems existing above.
According to an aspect of the present invention, a kind of auxiliary driving method is provided, comprises: detect the position of vehicle and the distance of traffic lights position; If distance is less than first threshold, then obtain the image of vehicle front; Analyze this image and judge whether there is traffic lights in image; If there is traffic lights, then extract traffic lights relevant information, described traffic lights relevant information comprises the color of traffic lights; In response to the generation the first event being detected, by vibrating device or the voice reminder driver of steering wheel for vehicle, wherein said first event is that the color of traffic lights becomes green from redness.First threshold can be 100 meters.
Described in response to detecting that the generation of the first event can also be comprised by the vibrating device of steering wheel for vehicle or the step of voice reminder driver: in response to the generation the first event being detected, to judge whether vehicle present speed meets first condition; If meet first condition, then by vibrating device or the voice reminder driver of steering wheel for vehicle, described first condition is present speed is zero.
The method can also comprise: in response to generation second event being detected, judge whether position and the speed of now vehicle meet second condition, wherein said second event is that the color of traffic lights becomes yellow or redness from green, and described second condition is that vehicle location is less than Second Threshold relative to the distance of traffic lights position and the ratio of speed; And if meet second condition, then by vibrating device or the voice reminder driver of steering wheel for vehicle.Second Threshold can be 2 seconds.
The method can also comprise: if there is not traffic lights in image, then whether there is in-plant large area shelter in recognition image; If there is in-plant large area shelter, then in response to the generation the 3rd event being detected, by vibrating device or the voice reminder driver of steering wheel for vehicle, wherein said 3rd event is that this large area shelter changes relative to the distance of vehicle.The comprising of described detection the 3rd event detect this large area shelter by graphical analysis or Distnace determination device to change relative to the distance of vehicle.Described in response to detecting that the generation of the 3rd event can be comprised by the vibrating device of steering wheel for vehicle or the step of voice reminder driver: in response to the generation the 3rd event being detected, to judge whether vehicle present speed meets first condition; If meet first condition, then by vibrating device or the voice reminder driver of steering wheel for vehicle.
Traffic lights relevant information can also comprise traffic light kind, and described type comprises circular and arrow pattern traffic lights.The method can also comprise: if described traffic light kind is arrow pattern traffic lights, then according to the track at vehicle location and described image determination vehicle place; If track, place is corresponding with arrow pattern traffic lights, then in response to the generation the first event being detected, by vibrating device or the voice reminder driver of steering wheel for vehicle.Step according to the track at vehicle location and described image determination vehicle place can comprise: at least one track that may be positioned at according to the vehicle position information determination vehicle of location; By the track at the marker determination vehicle place in recognition image.
According to a further aspect in the invention, a kind of DAS (Driver Assistant System) is provided, comprises: image unit, be suitable for the image of catching vehicle front; Navigation positioning unit, is suitable for determining the position of vehicle and detects the position of vehicle and the distance of traffic lights position, if distance is less than first threshold, then indicating described image unit to catch the image of vehicle front; Image analyzing unit, is suitable for analysis chart picture and judges whether there is traffic lights in image, and if there is traffic lights, then extract traffic lights relevant information, described traffic lights relevant information comprises the color of traffic lights; Assisting automobile driver unit, is suitable for the vibrating device by steering wheel for vehicle or voice reminder driver.Assisting automobile driver unit is suitable in response to image analyzing unit detects the generation of the first event and reminds driver, and wherein said first event is that the color of traffic lights becomes green from redness.
Assisting automobile driver unit can also be suitable for the generation the first event being detected in response to image analyzing unit, judge whether vehicle present speed meets first condition by navigation positioning unit, if and meet first condition, then remind driver, described first condition is present speed is zero.
Assisting automobile driver unit can also be suitable for generation second event being detected in response to image analyzing unit, judge whether the position of now vehicle and speed meet second condition by navigation positioning unit, wherein said second event is that the color of traffic lights becomes yellow or redness from green, and described second condition is that vehicle location is less than Second Threshold relative to the distance of traffic lights position and the ratio of speed; And if meet second condition, then remind driver.
If image analyzing unit can also be suitable for there is not traffic lights in image, then whether there is in-plant large area shelter in recognition image, if described assisting automobile driver unit can also be suitable for there is in-plant large area shelter, then in response to the generation the 3rd event being detected, remind driver, wherein said 3rd event is that this large area shelter changes relative to the distance of vehicle.The generation of the 3rd event can be the analyzing and detecting by described image analyzing unit.
This DAS (Driver Assistant System) can also comprise Distnace determination device, is suitable for the generation detecting described 3rd event.
Assisting automobile driver unit can also be suitable for the generation in response to the 3rd event being detected, judges whether vehicle present speed meets first condition by navigation positioning unit, and if meet first condition, then remind driver.
Traffic lights relevant information can also comprise traffic light kind, described type comprises circular and arrow pattern traffic lights, if it is arrow pattern traffic lights that described navigation positioning unit can also be suitable for described traffic light kind, then according to the track at vehicle location and described image determination vehicle place, if it is corresponding with arrow pattern traffic lights that described assisting automobile driver unit can also be suitable for track, place, then in response to detecting that driver is reminded in the generation of the first event.Navigation positioning unit is also suitable for vehicle position information determination vehicle at least one track that may be positioned at according to location, and determines the track at vehicle place by the marker in image analyzing unit recognition image.
Accompanying drawing explanation
In order to realize above-mentioned and relevant object; combine description below and accompanying drawing herein to describe some illustrative aspect; these aspects indicate the various modes can putting into practice principle disclosed herein, and all aspects and equivalent aspect thereof are intended to fall in the scope of theme required for protection.Read detailed description below in conjunction with the drawings, above-mentioned and other object of the present disclosure, Characteristics and advantages will become more obvious.Throughout the disclosure, identical Reference numeral is often referred to for identical parts or element.
Fig. 1 illustrates the schematic block diagram of DAS (Driver Assistant System) 100 according to an embodiment of the invention;
Fig. 2 illustrates flow process Figure 200 of auxiliary driving method according to an embodiment of the invention.
Embodiment
Should understand, term used herein " vehicle " or other similar terms comprise common motor vehicle, such as, comprise the passenger car of Multifunctional bicycle (SUV), bus, truck, various commercial vehicle, the water carrier comprising various ship and boats and ships, aircraft etc., and comprise hybrid electric vehicle, electric motor car, plug-in hybrid electric vehicles, hydrogen-powered vehicle, fuel-cell vehicle and other alternate fuel car.
Although exemplary embodiment is described to use multiple unit to perform exemplary flow, should be appreciated that exemplary flow also can be performed by one or more module.In addition, should be appreciated that term " controller/control module " refers to the hardware unit comprising storer and processor.This storer is configured to store above-mentioned module, and processor is specifically configured to perform above-mentioned module, to perform the one or more processes further described below.
In addition, steering logic of the present invention also can be embodied as the non-transient computer-readable medium on computer-readable medium, and this computer-readable medium comprises the executable program instructions performed by processor, controller/control module etc.The example of computer-readable medium includes but not limited to ROM, RAM, CD-ROM (read-only optical disc), tape, floppy disk, flash memory disk, smart card and optical data storage device.Computer readable recording medium storing program for performing also can be distributed in the computer system of interconnection network, such as to be stored and execution computer-readable medium with distribution form by remote server or controller local area network (CAN).
Term used herein is only used to the object of illustrated example embodiment instead of is intended to limit the present invention.As used herein, singulative ", one " is also intended to comprise plural form, indicates unless clear in context.It will also be appreciated that, the term used in the description " comprises " and/or " comprising " refers to there are described feature, step, operation, element and/or parts, but does not get rid of existence or add one or more further feature, step, operation, element, parts and/or its group.As used herein, term "and/or" comprises any of one or more relevant Listed Items and all combinations.
Below with reference to accompanying drawings exemplary embodiment of the present disclosure is described in more detail.Although show exemplary embodiment of the present disclosure in accompanying drawing, however should be appreciated that can realize the disclosure in a variety of manners and not should limit by the embodiment set forth here.On the contrary, provide these embodiments to be in order to more thoroughly the disclosure can be understood, and complete for the scope of the present disclosure can be conveyed to those skilled in the art.
Fig. 1 illustrates the schematic block diagram of DAS (Driver Assistant System) 100 according to an embodiment of the invention.
As shown in Figure 1, this DAS (Driver Assistant System) comprises image unit 110, navigation positioning unit 120, image analyzing unit 130 and assisting automobile driver unit 140.
Image unit 110 can catch the image of vehicle front.It can comprise optical lens and sensor, and camera lens preferably uses the wide-angle lens of more than 120 ° visual ranges, and sensor can be CMOS or ccd sensor.Its lens speed can reach F2.8, and imaging resolution is not less than 800*600 pixel, and still has good imaging quality under comparatively half-light line.In order to further Enhanced Imaging effect, flashlamp can also be equipped with.In one embodiment, image unit 110 can realize as a part for vehicle-mounted traveling recorder, its image of catching comprises still image and dynamic image (i.e. video), is stored in the storage medium of such as safe digital card (SD card) and so on.Navigation positioning unit 120 can determine the position of vehicle.In addition, navigation positioning unit 120 have built-in storage map or can the map of online access, combining cartographic information can show the position of vehicle in map and detect the position of vehicle and the distance of upcoming traffic lamp/position, crossing.Such as, navigation positioning unit 120 can calculate distance between the two according to the line between the absolute position of vehicle and the absolute position of traffic lights.In order to save electric power, image unit 110 most time can be in holding state, when the position of vehicle and the distance of upcoming traffic lamp/position, crossing are less than first threshold, during as 100 meters, navigation positioning unit 120 just indicates image unit 110 to work and catches the image of vehicle front.Image analyzing unit 130 analysis chart picture also judges whether there is traffic lights in image, if there is traffic lights, then extracts traffic lights relevant information.The design standards of traffic lights all has corresponding specification in various countries, such as, in China by regulations such as GB GB14886-2006.Common automobile traffic lamp forms one group by the circular pattern unit that red, yellow, green three geometric positions are discrete, or by presenting redness in time, yellow, a green pattern-free circular cell form.In addition, also have blinker, be made up of arrow pattern unit in the same way in discrete by red, yellow, green three geometric positions.The installation of traffic lights also has various putting in order, as laterally and vertically installed.Should be appreciated that the present invention is not limited to specific traffic lights model, but go for various possible traffic lights.Image analyzing unit 130 is according to the existing knowledge about traffic lights, profile, color and brightness etc. can be utilized to judge whether to exist in image traffic lights, such as, typical traffic light feature is a circular luminous object, or three transverse directions or vertically disposed circular object (one of them is higher than other two brightness).If there is traffic lights, image analyzing unit 130 can extract traffic lights relevant information, comprises the color of traffic lights.It should be noted that, color mentioned here refers to the color corresponding with the effective right of way signal indicated by traffic lights, such as, when a traffic lights, this traffic lights to present at that time green then mean allow current, and when forming one group of traffic lights by three traffic lights, the lamp display useful signal of a color may be only had, as one of them lamp presents redness, mean that this direction does not allow to pass through, and other two lamps are not lighted, now also claim the color of traffic lights for red.When three traffic lights represent the right of way signal in different track respectively, the color of traffic lights then refers to the color of the traffic lights corresponding to track, vehicle place, and this will discuss later further.
The various events that assisting automobile driver unit 140 detects according to image analyzing unit 130 and/or navigation positioning unit 120 are reminded driver, such as, by vibrating device or the voice mode of steering wheel for vehicle.In one embodiment, image analyzing unit 130 detects that the color of traffic lights becomes green from redness, mean that the vehicle before crossing can allow to pass through, assisting automobile driver unit 140 couples of drivers remind, wherein the vibration of bearing circle is than better suited mode, be enough to the attention causing driver, excessive interference can not be caused to it again.Be parked in the driver before crossing and can start vehicle in time by crossing after acquisition is reminded, thus the noise of blowing a whistle of the traffic congestion avoided not starting vehicle in time and cause and below vehicle.Here, the color of traffic becomes green from redness and can comprise single traffic lights and become from redness the lamp that lights green situation and one group of traffic lights to be become green light situation from red colored lamp.When detecting that the color of traffic lights becomes green from redness, navigation positioning unit 120 can also judge whether vehicle present speed is zero, if speed is zero, assisting automobile driver unit 140 couples of drivers remind.The accuracy of prompting can be improved like this, because when vehicle is in transport condition close to crossing, driver generally can not lose notice, even if red light becomes green light, also do not need to remind driver specially.
In another embodiment, image analyzing unit 130 detects that the color of traffic lights becomes yellow or redness from green, and now navigation positioning unit 120 judges that vehicle location is less than Second Threshold relative to the distance of traffic lights position and the ratio of speed, such as, and 2 seconds.If be less than Second Threshold, mean that vehicle gets final product exterior traffic lamp position within the time being shorter than Second Threshold, the speed of a motor vehicle is too fast, when traffic lights becomes yellow or redness from green, driver is as absent minded situation about may make a dash across the red light, cause danger, therefore assisting automobile driver unit 140 can be reminded to slow down in time driver.
In another embodiment, if image analyzing unit 130 does not detect traffic lights in the picture, may be so because junction ahead does not arrange traffic lights or traffic lights fault, also may be because vehicle front gear has other vehicles, particularly oversize vehicle.Whether image analyzing unit 130 can exist in-plant large area shelter in recognition image.If there is in-plant large area shelter, then when this large area shelter changes relative to the distance of vehicle, be likely the vehicle in front due to traffic lights from redness become green and start, assisting automobile driver unit 140 can be reminded driver, so that vehicle passes through crossing in time.By the means of graphical analysis, such as, can compare the position of shelter in two images, or detect this large area shelter by Distnace determination device and change relative to the distance of vehicle.In addition, in order to improve the accuracy of prompting further, navigation positioning unit 120 can judge whether vehicle present speed is zero, if be zero, assisting automobile driver unit 140 couples of drivers remind.
Above embodiment is all discussed for the example of circular pattern traffic lights.In addition, traffic lights may also have arrow pattern, the right of way signal that this arrow pattern often indicates the vehicle in corresponding track to turn left or turn right.If traffic light kind is arrow pattern traffic lights, then navigation positioning unit 120 can according to the track at vehicle location determination vehicle place, if track, place is corresponding with arrow pattern traffic lights, such as, vehicle is positioned at left turn lane, corresponding is left-turning traffic lamp with it, and so when this left-turning traffic lamp becomes green light from red light, assisting automobile driver unit 140 couples of drivers remind.For using the navigation positioning unit of civil navigation positioning chip (as GPS), because its positioning precision is limited, the graphical analysis of image analyzing unit 130 can be coordinated to determine track, vehicle place.Such as, the precision of location may be within the scope of 5 meters, so can determine that vehicle may be positioned at wherein at least one track according to vehicle position information.Such as, at road, there are before crossing four tracks, wherein left turn lane, two Through Lanes and a right-turn lane, the track distributed intelligence of road can obtain from map, according to the position of vehicle, navigation positioning unit 120 can determine that vehicle may be positioned on the Through Lane of left turn lane and next-door neighbour's left turn lane.Now, can analyze by combining image analytic unit 130 pairs of images, determined the track at vehicle place by the marker in recognition image.Such as, when vehicle is positioned at left turn lane, generally there will be the marker such as isolation strip, fence in left front in the image that image unit 110 is taken, and when being positioned at right-turn lane, in image, generally there will be the marker such as road edge, pedestrian walkway signals in right front.
In solution of the present invention, by graphical analysis and in conjunction with the feature of the position of vehicle, speed and traffic lights, appropriate prompting can be carried out to driver when vehicle is positioned at before traffic lights, driver is helped to tackle the various situations of traffic lights, improve traffic efficiency, reduce accident risk, do not need to carry out any transformation to existing means of transportation simultaneously, save cost.
Although it should be noted that and DAS (Driver Assistant System) 100 is shown for comprising multiple independently assembly in FIG, should be appreciated that these assemblies also can be partly integrated or together fully-integrated, such as, realize in the mode of onboard system.These assemblies can be connected by gateway (not shown).Gateway can pass through controller local area network (CAN) and be connected with panel board controller, motor controller, entire car controller, battery management system (BMS) etc., allows mutually to exchange data and instruction between these parts.The speed of vehicle not only can be obtained by navigation positioning unit, also can be obtained from controller by CAN via gateway, equally, also can be sent to corresponding vibrating device or speech player by gateway to the prompting (vibration or voice) of user.
Fig. 2 illustrates flow process Figure 200 of auxiliary driving method according to an embodiment of the invention.
In step S210, detect the position of vehicle and the distance of traffic lights position.In step S220, if distance is less than first threshold, then obtain the image of vehicle front.In step S230, analyze this image and judge whether there is traffic lights in image.In step S240, if there is traffic lights, then extract traffic lights relevant information, traffic lights relevant information comprises the color of traffic lights.In step S250, in response to the generation the first event being detected, by vibrating device or the voice reminder driver of steering wheel for vehicle, wherein the first event is that the color of traffic lights becomes green from redness.The generation in response to the first event being detected can be comprised in step S250, judge whether vehicle present speed meets first condition, if meet first condition, then by vibrating device or the voice reminder driver of steering wheel for vehicle, first condition is present speed is zero.In addition, the method can also comprise the generation in response to second event being detected, judge whether position and the speed of now vehicle meet second condition, wherein second event is that the color of traffic lights becomes yellow or redness from green, and second condition is that vehicle location is less than Second Threshold relative to the distance of traffic lights position and the ratio of speed.If meet second condition, then by vibrating device or the voice reminder driver of steering wheel for vehicle.If judge to there is not traffic lights in image in step S230, then whether there is in-plant large area shelter in recognition image.If there is in-plant large area shelter, then in response to the generation the 3rd event being detected, by vibrating device or the voice reminder driver of steering wheel for vehicle, wherein the 3rd event is that this large area shelter changes relative to the distance of vehicle.Detect comprising of the 3rd event to detect this large area shelter by graphical analysis or Distnace determination device and change relative to the distance of vehicle.In response to the generation the 3rd event being detected, can also judge whether vehicle present speed meets first condition.If meet first condition, then by vibrating device or the voice reminder driver of steering wheel for vehicle.Traffic lights relevant information can also comprise traffic light kind, and described type comprises circular and arrow pattern traffic lights.If described traffic light kind is arrow pattern traffic lights, then can according to the track at vehicle location and described image determination vehicle place.If track, place is corresponding with arrow pattern traffic lights, then in response to the generation the first event being detected, by vibrating device or the voice reminder driver of steering wheel for vehicle.When determining track, vehicle place, at least one track that may be able to be positioned at according to the vehicle position information determination vehicle of location, and pass through the track at the marker determination vehicle place in recognition image.
In instructions provided herein, describe a large amount of detail.But can understand, embodiments of the invention can be put into practice when not having these details.In some instances, be not shown specifically known method, structure and technology, so that not fuzzy understanding of this description.
Similarly, be to be understood that, in order to simplify the disclosure and to help to understand in each inventive aspect one or more, in the description above to exemplary embodiment of the present invention, each feature of the present invention is grouped together in single embodiment, figure or the description to it sometimes.But, the method for the disclosure should be construed to the following intention of reflection: namely the present invention for required protection requires than the feature more multiple features clearly recorded in each claim.Or rather, as claims below reflect, all features of disclosed single embodiment before inventive aspect is to be less than.Therefore, the claims following embodiment are incorporated to this embodiment thus clearly, and wherein each claim itself is as independent embodiment of the present invention.
Those skilled in the art are to be understood that the module of the equipment in example disclosed herein or unit or assembly can be arranged in equipment as depicted in this embodiment, or alternatively can be positioned in one or more equipment different from the equipment in this example.Module in aforementioned exemplary can be combined as a module or can be divided into multiple submodule in addition.
Those skilled in the art are appreciated that and adaptively can change the module in the equipment in embodiment and they are arranged in one or more equipment different from this embodiment.Module in embodiment or unit or assembly can be combined into a module or unit or assembly, and multiple submodule or subelement or sub-component can be put them in addition.Except at least some in such feature and/or process or unit be mutually repel except, any combination can be adopted to combine all processes of all features disclosed in this instructions (comprising adjoint claim, summary and accompanying drawing) and so disclosed any method or equipment or unit.Unless expressly stated otherwise, each feature disclosed in this instructions (comprising adjoint claim, summary and accompanying drawing) can by providing identical, alternative features that is equivalent or similar object replaces.
The present invention also comprises: A6, method as described in A4 or 5, wherein said in response to detecting that the generation of the 3rd event is comprised by the vibrating device of steering wheel for vehicle or the step of voice reminder driver: in response to the generation the 3rd event being detected, to judge whether vehicle present speed meets first condition; If meet first condition, then by vibrating device or the voice reminder driver of steering wheel for vehicle.A7, method as described in A1, wherein traffic lights relevant information also comprises traffic light kind, described type comprises circular and arrow pattern traffic lights, the method also comprises: if described traffic light kind is arrow pattern traffic lights, then according to the track at vehicle location and described image determination vehicle place; If track, place is corresponding with arrow pattern traffic lights, then in response to the generation the first event being detected, by vibrating device or the voice reminder driver of steering wheel for vehicle.A8, method as described in A7, wherein comprise according to the step in the track at vehicle location and described image determination vehicle place: at least one track that may be positioned at according to the vehicle position information determination vehicle of location; By the track at the marker determination vehicle place in recognition image.A9, method as described in A1, wherein said first threshold is 100 meters.A10, method as described in A3, wherein said Second Threshold is 2 seconds.
B16, DAS (Driver Assistant System) as described in B15, also comprise Distnace determination device, be suitable for the generation detecting described 3rd event.B17, as the DAS (Driver Assistant System) in B14-16 as described in any one, wherein said assisting automobile driver unit is also suitable for the generation in response to the 3rd event being detected, judge whether vehicle present speed meets first condition by navigation positioning unit, and if meet first condition, then remind driver.B18, DAS (Driver Assistant System) as described in B11, wherein traffic lights relevant information also comprises traffic light kind, described type comprises circular and arrow pattern traffic lights, if it is arrow pattern traffic lights that described navigation positioning unit is also suitable for described traffic light kind, then according to the track at vehicle location and described image determination vehicle place, if it is corresponding with arrow pattern traffic lights that described assisting automobile driver unit is also suitable for track, place, then in response to detecting that driver is reminded in the generation of the first event.B19, DAS (Driver Assistant System) as described in B18, described navigation positioning unit is also suitable for vehicle position information determination vehicle at least one track that may be positioned at according to location, and determines the track at vehicle place by the marker in image analyzing unit recognition image.B20, DAS (Driver Assistant System) as described in B11, wherein said first threshold is 100 meters.B21, DAS (Driver Assistant System) as described in B13, wherein said Second Threshold is 2 seconds.
In addition, those skilled in the art can understand, although embodiments more described herein to comprise in other embodiment some included feature instead of further feature, the combination of the feature of different embodiment means and to be within scope of the present invention and to form different embodiments.Such as, in the following claims, the one of any of embodiment required for protection can use with arbitrary array mode.
In addition, some in described embodiment are described as at this can by the processor of computer system or the method implemented by other device performing described function or the combination of method element.Therefore, there is the device of processor formation for implementing the method or method element of the necessary instruction for implementing described method or method element.In addition, the element described herein of device embodiment is the example as lower device: this device is for implementing the function performed by the element of the object in order to implement this invention.
As used in this, unless specifically stated so, use ordinal number " first ", " second ", " the 3rd " etc. to describe plain objects and only represent the different instances relating to similar object, and be not intended to imply the object be described like this must have the time upper, spatially, sequence aspect or in any other manner to definite sequence.
Although the embodiment according to limited quantity describes the present invention, benefit from description above, those skilled in the art understand, in the scope of the present invention described thus, it is contemplated that other embodiment.In addition, it should be noted that the language used in this instructions is mainly in order to object that is readable and instruction is selected, instead of select to explain or limiting theme of the present invention.Therefore, when not departing from the scope and spirit of appended claims, many modifications and changes are all apparent for those skilled in the art.For scope of the present invention, be illustrative to disclosing of doing of the present invention, and nonrestrictive, and scope of the present invention is defined by the appended claims.

Claims (10)

1. an auxiliary driving method, comprising:
Detect the position of vehicle and the distance of traffic lights position;
If distance is less than first threshold, then obtain the image of vehicle front;
Analyze this image and judge whether there is traffic lights in image;
If there is traffic lights, then extract traffic lights relevant information, described traffic lights relevant information comprises the color of traffic lights;
In response to the generation the first event being detected, by vibrating device or the voice reminder driver of steering wheel for vehicle, wherein said first event is that the color of traffic lights becomes green from redness.
2. the method for claim 1, wherein said in response to detecting that the generation of the first event is also comprised by the vibrating device of steering wheel for vehicle or the step of voice reminder driver:
In response to the generation the first event being detected, judge whether vehicle present speed meets first condition;
If meet first condition, then by vibrating device or the voice reminder driver of steering wheel for vehicle, described first condition is present speed is zero.
3. the method for claim 1, also comprises:
In response to generation second event being detected, judge whether position and the speed of now vehicle meet second condition, wherein said second event is that the color of traffic lights becomes yellow or redness from green, and described second condition is that vehicle location is less than Second Threshold relative to the distance of traffic lights position and the ratio of speed; And
If meet second condition, then by vibrating device or the voice reminder driver of steering wheel for vehicle.
4. the method for claim 1, also comprises:
If there is not traffic lights in image, then whether there is in-plant large area shelter in recognition image;
If there is in-plant large area shelter, then in response to the generation the 3rd event being detected, by vibrating device or the voice reminder driver of steering wheel for vehicle, wherein said 3rd event is that this large area shelter changes relative to the distance of vehicle.
5. method as claimed in claim 4, the comprising of wherein said detection the 3rd event detects this large area shelter by graphical analysis or Distnace determination device and changes relative to the distance of vehicle.
6. a DAS (Driver Assistant System), comprising:
Image unit, is suitable for the image of catching vehicle front;
Navigation positioning unit, is suitable for determining the position of vehicle and detects the position of vehicle and the distance of traffic lights position, if distance is less than first threshold, then indicating described image unit to catch the image of vehicle front;
Image analyzing unit, is suitable for analysis chart picture and judges whether there is traffic lights in image, and if there is traffic lights, then extract traffic lights relevant information, described traffic lights relevant information comprises the color of traffic lights;
Assisting automobile driver unit, is suitable for the vibrating device by steering wheel for vehicle or voice reminder driver,
Wherein assisting automobile driver unit is suitable in response to image analyzing unit detects the generation of the first event and reminds driver, and wherein said first event is that the color of traffic lights becomes green from redness.
7. DAS (Driver Assistant System) as claimed in claim 6, wherein said assisting automobile driver unit is also suitable for the generation the first event being detected in response to image analyzing unit, judge whether vehicle present speed meets first condition by navigation positioning unit, if and meet first condition, then remind driver, described first condition is present speed is zero.
8. DAS (Driver Assistant System) as claimed in claim 6, wherein said assisting automobile driver unit is also suitable for generation second event being detected in response to image analyzing unit, judge whether the position of now vehicle and speed meet second condition by navigation positioning unit, wherein said second event is that the color of traffic lights becomes yellow or redness from green, and described second condition is that vehicle location is less than Second Threshold relative to the distance of traffic lights position and the ratio of speed; And if meet second condition, then remind driver.
9. DAS (Driver Assistant System) as claimed in claim 6, if wherein said image analyzing unit is also suitable for there is not traffic lights in image, then whether there is in-plant large area shelter in recognition image, if described assisting automobile driver unit is also suitable for there is in-plant large area shelter, then in response to the generation the 3rd event being detected, remind driver, wherein said 3rd event is that this large area shelter changes relative to the distance of vehicle.
10. DAS (Driver Assistant System) as claimed in claim 9, wherein said 3rd event be by the analyzing and detecting of described image analyzing unit.
CN201510641531.3A 2015-09-30 2015-09-30 A kind of auxiliary driving method and system Expired - Fee Related CN105185140B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510641531.3A CN105185140B (en) 2015-09-30 2015-09-30 A kind of auxiliary driving method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510641531.3A CN105185140B (en) 2015-09-30 2015-09-30 A kind of auxiliary driving method and system

Publications (2)

Publication Number Publication Date
CN105185140A true CN105185140A (en) 2015-12-23
CN105185140B CN105185140B (en) 2018-07-06

Family

ID=54907178

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510641531.3A Expired - Fee Related CN105185140B (en) 2015-09-30 2015-09-30 A kind of auxiliary driving method and system

Country Status (1)

Country Link
CN (1) CN105185140B (en)

Cited By (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105654755A (en) * 2016-01-27 2016-06-08 苏州寅初信息科技有限公司 Bicycle braking system based on traffic signal lamp discrimination
CN106004438A (en) * 2016-07-19 2016-10-12 广汽菲亚特克莱斯勒汽车有限公司 Linking method and system for vehicle-mounted intelligent traffic light
CN106504554A (en) * 2016-09-30 2017-03-15 乐视控股(北京)有限公司 The method and device of identification traffic light status information
CN106600978A (en) * 2017-02-15 2017-04-26 赵立 Monitoring method of vehicle red light running
WO2017161666A1 (en) * 2016-03-23 2017-09-28 京东方科技集团股份有限公司 Blind-man navigation method and blind-man navigation device
CN107316488A (en) * 2017-08-23 2017-11-03 苏州豪米波技术有限公司 The recognition methods of signal lamp, device and system
CN107544296A (en) * 2016-06-28 2018-01-05 奥迪股份公司 Electronic-controlled installation and method for vehicle
CN107657821A (en) * 2016-07-26 2018-02-02 奥迪股份公司 Electronic-controlled installation and method for vehicle
CN107664764A (en) * 2016-07-29 2018-02-06 高德信息技术有限公司 A kind of navigation object determines method and device
CN108407806A (en) * 2018-02-26 2018-08-17 宁波吉利汽车研究开发有限公司 Auxiliary driving method and device
CN108536156A (en) * 2018-03-21 2018-09-14 深圳臻迪信息技术有限公司 Target Tracking System and method for tracking target
CN108802762A (en) * 2018-05-17 2018-11-13 四川斐讯信息技术有限公司 A kind of obstacle avoidance apparatus and barrier-avoiding method of Intelligent luggage carrier
CN108894136A (en) * 2018-07-12 2018-11-27 中国联合网络通信集团有限公司 A kind of generation method and device of virtual zebra stripes
CN109492507A (en) * 2017-09-12 2019-03-19 百度在线网络技术(北京)有限公司 The recognition methods and device of the traffic light status, computer equipment and readable medium
CN109801495A (en) * 2019-01-29 2019-05-24 合肥革绿信息科技有限公司 A kind of signal lamp prompt system and method based on image recognition
CN109859509A (en) * 2018-11-13 2019-06-07 惠州市德赛西威汽车电子股份有限公司 Lane state based reminding method and equipment
CN109920261A (en) * 2017-12-20 2019-06-21 问众智能信息科技(北京)有限公司 A kind of method, apparatus of information alert, mobile unit and readable storage medium storing program for executing
CN109935100A (en) * 2017-12-18 2019-06-25 福特全球技术公司 To the vehicle monitoring of infrastructure illumination
CN110097773A (en) * 2018-01-31 2019-08-06 上海飞田通信股份有限公司 Vehicle front traffic light system device
CN110660256A (en) * 2019-10-22 2020-01-07 北京地平线机器人技术研发有限公司 Method and device for estimating state of signal lamp
CN110675645A (en) * 2019-11-01 2020-01-10 常州工学院 Monitoring method and device
CN110768720A (en) * 2019-10-30 2020-02-07 南京工程学院 Vehicle-mounted traffic light identification and broadcast system based on LED visible light communication
CN110867088A (en) * 2019-11-21 2020-03-06 宝能汽车有限公司 Method and device for acquiring signal lamp state
CN110941975A (en) * 2018-09-21 2020-03-31 广州汽车集团股份有限公司 Image acquisition method, angle adjustment device and driving system
CN111583697A (en) * 2019-02-18 2020-08-25 丰田自动车株式会社 Driving support system and server device
CN111695546A (en) * 2020-06-28 2020-09-22 北京京东乾石科技有限公司 Traffic signal lamp identification method and device for unmanned vehicle
CN111723625A (en) * 2019-03-22 2020-09-29 上海海拉电子有限公司 Traffic light image recognition processing method and device, auxiliary traffic system and storage medium
CN112001235A (en) * 2020-07-13 2020-11-27 浙江大华汽车技术有限公司 Vehicle traffic information generation method and device and computer equipment
CN112017456A (en) * 2020-08-14 2020-12-01 上海擎感智能科技有限公司 Early warning method, vehicle-mounted terminal and computer readable storage medium
CN112061132A (en) * 2019-05-24 2020-12-11 阿里巴巴集团控股有限公司 Driving assistance method and driving assistance device
CN112885126A (en) * 2019-11-30 2021-06-01 华为技术有限公司 Driving prompting method and related device
TWI742597B (en) * 2020-04-03 2021-10-11 富智捷股份有限公司 Driving assistance method based on hd map, and vehicle electronic apparatus
CN113492858A (en) * 2020-04-03 2021-10-12 深圳市超捷通讯有限公司 Driving assistance method based on high-precision map and vehicle-mounted device
CN113506437A (en) * 2021-05-20 2021-10-15 云度新能源汽车有限公司 Traffic light pre-judging method based on vehicle-mounted intelligent camera and big data platform and storage device
CN114333380A (en) * 2021-12-13 2022-04-12 重庆长安汽车股份有限公司 Traffic light identification method and system based on camera and V2x, and vehicle
CN114333289A (en) * 2020-09-28 2022-04-12 沃尔沃汽车公司 Vehicle starting reminding device, system and method

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115131764A (en) * 2021-03-26 2022-09-30 宝马股份公司 Method, system and device for detecting traffic light for vehicle

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06251291A (en) * 1993-02-22 1994-09-09 Sawako Amahisa Traffic signal checker
CN202126738U (en) * 2011-07-03 2012-01-25 烟台汽车工程职业学院 Device for automatically reminding passengers in bus of traffic light conversion
CN102682618A (en) * 2012-06-12 2012-09-19 中国人民解放军军事交通学院 Traffic light detection and reminding device for safe driving
CN103236178A (en) * 2013-04-07 2013-08-07 江苏物联网研究发展中心 Signal lamp mode recognition reminding system and method
CN103295411A (en) * 2012-12-27 2013-09-11 天津森宇科技发展有限公司 Signal lamp monitoring and prompting device in process of driving
CN104252793A (en) * 2013-06-27 2014-12-31 比亚迪股份有限公司 Signal lamp state detecting method, signal lamp state detecting system and vehicle-mounted control device
CN104821094A (en) * 2015-05-22 2015-08-05 广东欧珀移动通信有限公司 Traffic signal lamp reminding method and device
CN104916150A (en) * 2015-05-12 2015-09-16 三星电子(中国)研发中心 Traffic safety presentation method applied to electronic device and electronic device

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06251291A (en) * 1993-02-22 1994-09-09 Sawako Amahisa Traffic signal checker
CN202126738U (en) * 2011-07-03 2012-01-25 烟台汽车工程职业学院 Device for automatically reminding passengers in bus of traffic light conversion
CN102682618A (en) * 2012-06-12 2012-09-19 中国人民解放军军事交通学院 Traffic light detection and reminding device for safe driving
CN103295411A (en) * 2012-12-27 2013-09-11 天津森宇科技发展有限公司 Signal lamp monitoring and prompting device in process of driving
CN103236178A (en) * 2013-04-07 2013-08-07 江苏物联网研究发展中心 Signal lamp mode recognition reminding system and method
CN104252793A (en) * 2013-06-27 2014-12-31 比亚迪股份有限公司 Signal lamp state detecting method, signal lamp state detecting system and vehicle-mounted control device
CN104916150A (en) * 2015-05-12 2015-09-16 三星电子(中国)研发中心 Traffic safety presentation method applied to electronic device and electronic device
CN104821094A (en) * 2015-05-22 2015-08-05 广东欧珀移动通信有限公司 Traffic signal lamp reminding method and device

Cited By (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105654755A (en) * 2016-01-27 2016-06-08 苏州寅初信息科技有限公司 Bicycle braking system based on traffic signal lamp discrimination
WO2017161666A1 (en) * 2016-03-23 2017-09-28 京东方科技集团股份有限公司 Blind-man navigation method and blind-man navigation device
US10509961B2 (en) 2016-03-23 2019-12-17 Boe Technology Group Co., Ltd. Blindman navigation method and blindman navigation apparatus
CN107544296B (en) * 2016-06-28 2020-03-17 奥迪股份公司 Electronic control device and method for vehicle
CN107544296A (en) * 2016-06-28 2018-01-05 奥迪股份公司 Electronic-controlled installation and method for vehicle
CN106004438A (en) * 2016-07-19 2016-10-12 广汽菲亚特克莱斯勒汽车有限公司 Linking method and system for vehicle-mounted intelligent traffic light
CN107657821A (en) * 2016-07-26 2018-02-02 奥迪股份公司 Electronic-controlled installation and method for vehicle
CN107664764A (en) * 2016-07-29 2018-02-06 高德信息技术有限公司 A kind of navigation object determines method and device
CN106504554A (en) * 2016-09-30 2017-03-15 乐视控股(北京)有限公司 The method and device of identification traffic light status information
CN106600978A (en) * 2017-02-15 2017-04-26 赵立 Monitoring method of vehicle red light running
CN106600978B (en) * 2017-02-15 2018-04-17 长沙硕铠电子科技有限公司 A kind of monitoring method of running red light for vehicle
CN107316488A (en) * 2017-08-23 2017-11-03 苏州豪米波技术有限公司 The recognition methods of signal lamp, device and system
US11080539B2 (en) 2017-09-12 2021-08-03 Baidu Online Network Technology (Beijing) Co., Ltd. Traffic light state recognizing method and apparatus, computer device and readable medium
CN109492507A (en) * 2017-09-12 2019-03-19 百度在线网络技术(北京)有限公司 The recognition methods and device of the traffic light status, computer equipment and readable medium
CN109935100A (en) * 2017-12-18 2019-06-25 福特全球技术公司 To the vehicle monitoring of infrastructure illumination
CN109920261A (en) * 2017-12-20 2019-06-21 问众智能信息科技(北京)有限公司 A kind of method, apparatus of information alert, mobile unit and readable storage medium storing program for executing
CN110097773A (en) * 2018-01-31 2019-08-06 上海飞田通信股份有限公司 Vehicle front traffic light system device
CN108407806A (en) * 2018-02-26 2018-08-17 宁波吉利汽车研究开发有限公司 Auxiliary driving method and device
CN108536156A (en) * 2018-03-21 2018-09-14 深圳臻迪信息技术有限公司 Target Tracking System and method for tracking target
CN108802762A (en) * 2018-05-17 2018-11-13 四川斐讯信息技术有限公司 A kind of obstacle avoidance apparatus and barrier-avoiding method of Intelligent luggage carrier
CN108894136A (en) * 2018-07-12 2018-11-27 中国联合网络通信集团有限公司 A kind of generation method and device of virtual zebra stripes
CN108894136B (en) * 2018-07-12 2020-10-13 中国联合网络通信集团有限公司 Virtual zebra crossing generation method and device
CN110941975A (en) * 2018-09-21 2020-03-31 广州汽车集团股份有限公司 Image acquisition method, angle adjustment device and driving system
CN109859509A (en) * 2018-11-13 2019-06-07 惠州市德赛西威汽车电子股份有限公司 Lane state based reminding method and equipment
WO2020098004A1 (en) * 2018-11-13 2020-05-22 惠州市德赛西威汽车电子股份有限公司 Lane traffic status reminder method and device
CN109801495A (en) * 2019-01-29 2019-05-24 合肥革绿信息科技有限公司 A kind of signal lamp prompt system and method based on image recognition
CN111583697A (en) * 2019-02-18 2020-08-25 丰田自动车株式会社 Driving support system and server device
CN111583697B (en) * 2019-02-18 2022-08-05 丰田自动车株式会社 Driving support system and server device
CN111723625B (en) * 2019-03-22 2023-08-08 上海海拉电子有限公司 Traffic light image recognition processing method and device, auxiliary traffic system and storage medium
CN111723625A (en) * 2019-03-22 2020-09-29 上海海拉电子有限公司 Traffic light image recognition processing method and device, auxiliary traffic system and storage medium
CN112061132A (en) * 2019-05-24 2020-12-11 阿里巴巴集团控股有限公司 Driving assistance method and driving assistance device
CN110660256A (en) * 2019-10-22 2020-01-07 北京地平线机器人技术研发有限公司 Method and device for estimating state of signal lamp
CN110660256B (en) * 2019-10-22 2021-04-27 北京地平线机器人技术研发有限公司 Method and device for estimating state of signal lamp
CN110768720A (en) * 2019-10-30 2020-02-07 南京工程学院 Vehicle-mounted traffic light identification and broadcast system based on LED visible light communication
CN110675645A (en) * 2019-11-01 2020-01-10 常州工学院 Monitoring method and device
CN110867088A (en) * 2019-11-21 2020-03-06 宝能汽车有限公司 Method and device for acquiring signal lamp state
CN112885126A (en) * 2019-11-30 2021-06-01 华为技术有限公司 Driving prompting method and related device
TWI742597B (en) * 2020-04-03 2021-10-11 富智捷股份有限公司 Driving assistance method based on hd map, and vehicle electronic apparatus
CN113492858A (en) * 2020-04-03 2021-10-12 深圳市超捷通讯有限公司 Driving assistance method based on high-precision map and vehicle-mounted device
US11562648B2 (en) 2020-04-03 2023-01-24 Mobile Drive Netherlands B.V. Method for providing assistance to driver based on HD map, and vehicle apparatus applying method
CN111695546A (en) * 2020-06-28 2020-09-22 北京京东乾石科技有限公司 Traffic signal lamp identification method and device for unmanned vehicle
CN112001235A (en) * 2020-07-13 2020-11-27 浙江大华汽车技术有限公司 Vehicle traffic information generation method and device and computer equipment
CN112017456A (en) * 2020-08-14 2020-12-01 上海擎感智能科技有限公司 Early warning method, vehicle-mounted terminal and computer readable storage medium
CN114333289A (en) * 2020-09-28 2022-04-12 沃尔沃汽车公司 Vehicle starting reminding device, system and method
CN114333289B (en) * 2020-09-28 2023-12-22 沃尔沃汽车公司 Vehicle starting reminding equipment, system and method
CN113506437A (en) * 2021-05-20 2021-10-15 云度新能源汽车有限公司 Traffic light pre-judging method based on vehicle-mounted intelligent camera and big data platform and storage device
CN114333380A (en) * 2021-12-13 2022-04-12 重庆长安汽车股份有限公司 Traffic light identification method and system based on camera and V2x, and vehicle

Also Published As

Publication number Publication date
CN105185140B (en) 2018-07-06

Similar Documents

Publication Publication Date Title
CN105185140A (en) Auxiliary driving method and system
US11961162B2 (en) Imaging apparatus, image processing apparatus, display system, and vehicle
US20220299994A1 (en) Configuring Motion Planning for a Self-Driving Tractor Unit
US10847032B2 (en) Apparatus for informing parking position and method thereof
JP6411595B2 (en) Method for controlling the speed of a vehicle
CN114375467B (en) System and method for detecting an emergency vehicle
CN108737992A (en) Pass through the diagnosis for the vehicle car light that the video camera of inter-vehicular communication assists
CN112298207A (en) Maintaining road safety in the presence of an autonomous vehicle that loses its ability to travel
CN105809095B (en) Determination of traffic intersection passage state
US10977503B2 (en) Fault isolation for perception systems in autonomous/active safety vehicles
CN108960044A (en) Infrastructure lamp state is determined using vehicle
WO2021065626A1 (en) Traffic control system, traffic control method, and control device
CN105632218A (en) Motorway vehicle-mounted group navigation system based on GPS and GSM platform
CN205644987U (en) On -vehicle crowd navigation of highway based on GPS and GSM platform
CN110271554A (en) Driving assistance system and method for vehicle
CN116745831A (en) Map generation device, map generation program, and in-vehicle device
CN110422176B (en) Intelligent transportation system and automobile based on V2X
KR102252913B1 (en) System and method of detecting signal light
CN110349403B (en) Driving assistance system and method for vehicle
JP7507707B2 (en) Control device, mobile object, control method and program
CN115063987B (en) Vehicle control method and device, vehicle and storage medium
US20230368546A1 (en) Methods and Systems for Parking Detection
WO2022230780A1 (en) Notification control device and notification control method for vehicles
JP2024158562A (en) Information processing device, system, and program
CN116533872A (en) Driving support device, vehicle, driving support method, and storage medium

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20180706

Termination date: 20210930