CN111191607A

CN111191607A - Method, apparatus, and storage medium for determining steering information of vehicle

Info

Publication number: CN111191607A
Application number: CN201911421098.7A
Authority: CN
Inventors: 周康明; 申影影
Original assignee: Shanghai Eye Control Technology Co Ltd
Current assignee: Shanghai Eye Control Technology Co Ltd
Priority date: 2019-12-31
Filing date: 2019-12-31
Publication date: 2020-05-22

Abstract

The invention provides a method, a device and a storage medium for determining vehicle steering information. The method comprises the following steps: acquiring an original image; respectively identifying the interest areas and the relative positions of the target vehicle in the panoramic pictures by using an identification model to generate foreground pictures; constructing a multitask segmentation network to segment the plurality of panoramic pictures to obtain road segmentation information and intersection segmentation information of a scene around the target vehicle, and fusing the road segmentation information and the intersection segmentation information with a given transparency to generate a background picture; combining the foreground picture and the background picture to generate forward data of the target vehicle; and performing forward operation on the forward data of the target vehicle based on the steering classification network to determine the steering information of the target vehicle.

Description

Method, apparatus, and storage medium for determining steering information of vehicle

Technical Field

The present invention relates to the field of intelligent detection, and more particularly, to a method for determining steering information of a vehicle, an apparatus implementing such method, and a computer-readable storage medium.

Background

With the continuous development of social economy and the continuous improvement of the living standard of people, the quantity of motor vehicles in cities is rapidly increased. The number of electronic police (i.e. camera) snapshot systems for motor vehicle violation has also increased rapidly. The traditional vehicle illegal electronic police snapshot picture auditing is mainly implemented through manual auditing, the workload is large, and a large number of invalid snapshot pictures can be generated under the condition that the electronic police cannot normally work in special weather or road reconstruction and the like, so that the workload of manual auditing is huge.

At present, a large number of off-site intelligent illegal auditing technologies are available, and intelligent illegal auditing is performed on a picture captured by an electric police camera at one time, namely, target vehicle attitude behavior and surrounding scene information are identified in a limited number of pictures in an acquired specific time period. The existing vehicle posture recognition technology mainly comprises the following steps:

the method comprises the steps of capturing and cutting a target vehicle region of interest in a limited picture. The method is greatly influenced by road conditions, and information is easy to lose, so that the accuracy is low.

And secondly, adding road information to overcome the defects of the first method, and finding a point (called vanishing point) at the end of the road to be connected with the midpoint of the target vehicle frame.

And thirdly, with the help of vanishing points, the target vehicle interesting region and the target license plate interesting region, specifying a judgment threshold value of vehicle postures such as left-right turning and straight going by utilizing geometric constraint so as to judge the vehicle postures. But the accuracy of the technology depends on the accuracy of the license plate detection.

Although the vehicle attitude classification can be realized in partial scenes, the method can not meet the engineering requirements, and the method can not meet the actual requirements even if the three methods are embedded into engineering by using multi-model fusion, and the accuracy is only 93.31%. The accuracy of vehicle posture identification usually restricts the accuracy of intelligent illegal auditing, so how to accurately and quickly identify the vehicle posture in a snapshot picture of an electronic police officer of vehicle illegal activities, and meanwhile, the defects of information loss, information redundancy and the like are avoided, and the method is a technical problem which needs to be solved urgently at present.

Disclosure of Invention

In view of the above problems, the present invention provides a solution for determining steering information of a vehicle, which can quickly and accurately analyze and identify a vehicle photo captured by an electronic police to determine steering information thereof and further determine an illegal state thereof.

According to one aspect of the present invention, a method for determining steering information of a vehicle is provided. The method comprises the following steps: acquiring an original image, wherein the original image comprises a plurality of panoramic pictures and at least one close-up picture of a target vehicle, and each panoramic picture comprises the target vehicle and a scene around the target vehicle; respectively identifying the interest areas and the relative positions of the target vehicle in the panoramic pictures by using an identification model to generate foreground pictures; constructing a multitask segmentation network to segment the plurality of panoramic pictures to obtain road segmentation information and intersection segmentation information of a scene around the target vehicle, and fusing the road segmentation information and the intersection segmentation information with a given transparency to generate a background picture; combining the foreground picture and the background picture to generate forward data of the target vehicle; and performing a forward operation on the forward data of the target vehicle based on the steering classification network to determine steering information of the target vehicle, wherein the steering information comprises one of left turning, right turning and straight going.

According to another aspect of the present invention, there is provided an apparatus for determining steering information of a vehicle. The apparatus comprises: a memory having computer program code stored thereon; and a processor configured to execute the computer program code to perform the method as described above.

According to yet another aspect of the present invention, a computer-readable storage medium is provided. The computer readable storage medium has stored thereon a computer program code which, when executed, performs the method as described above.

By utilizing the scheme of the invention, through constructing the improved multitask segmentation network, more information of the target vehicle, such as road segmentation information, can be acquired under the condition of not obviously increasing the burden of the display card, so that the judgment on the steering information of the target vehicle is quicker and more accurate.

Drawings

FIG. 1 shows a flow diagram of a method for determining steering information of a vehicle according to an embodiment of the invention;

FIGS. 2A and 2B illustrate schematic structural diagrams of a conventional PSPnet split network and an improved PSPnet split network according to an embodiment of the present invention, respectively;

FIGS. 3A and 3B show schematic structural diagrams of a conventional GoogleLeNet network and an improved GoogleLeNet V3 network, respectively, according to embodiments of the present invention;

FIGS. 4A through 4F illustrate a simulated experimental scenario for determining steering information for a vehicle in accordance with the present invention; and

FIG. 5 shows a schematic block diagram of an example device that may be used to implement an embodiment of the invention.

Detailed Description

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings in order to more clearly understand the objects, features and advantages of the present invention. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of the present invention, but are merely intended to illustrate the spirit of the technical solution of the present invention.

In the following description, for the purposes of illustrating various inventive embodiments, certain specific details are set forth in order to provide a thorough understanding of the various inventive embodiments. One skilled in the relevant art will recognize, however, that the embodiments may be practiced without one or more of the specific details. In other instances, well-known devices, structures and techniques associated with this application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

Throughout the specification and claims, the word "comprise" and variations thereof, such as "comprises" and "comprising," are to be understood as an open, inclusive meaning, i.e., as being interpreted to mean "including, but not limited to," unless the context requires otherwise.

Reference throughout this specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in some embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

As used in the specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It should be noted that the term "or" is generally employed in its sense including "and/or" unless the context clearly dictates otherwise.

FIG. 1 shows a flow diagram of a method 100 for determining steering information of a vehicle, according to an embodiment of the invention.

In the method 100, first, an original image is acquired at step 110. The original image includes a plurality of panoramic pictures and at least one close-up picture of the target vehicle. Each panoramic picture includes the target vehicle itself and the scene surrounding the target vehicle. The original image may be, for example, an image captured by a camera of a road violation inspection agency, such as an electronic police mounted in a traffic police department. For the judgment of an illegal state related to the turning of a vehicle, it is not sufficient to rely only on a still picture of the vehicle, and therefore, a plurality of temporally continuous panoramic pictures taken at the same photographing position are generally required. In addition, since there may be multiple vehicles on a picture, a close-up photograph of at least one target vehicle is also needed to identify the target vehicle from the panoramic picture.

Currently, an original image for illegal review generally includes three panoramic pictures and one close-up picture, and the four pictures are spliced into one original image for background manual review.

Next, in step 120, the regions of interest and the relative positions of the target vehicle in the plurality of panoramic pictures are respectively identified from the plurality of panoramic pictures by using the identification model to generate foreground pictures. Here, since the plurality of panoramic pictures are a plurality of temporally continuous pictures taken at the same shooting position (like a camera), the positions of the same target vehicle in the plurality of panoramic pictures are usually different from each other, and since the vehicle is in the process of traveling, the postures in the plurality of panoramic pictures may be different from each other, for example, some panoramic pictures are taken of a rear view of the target vehicle, some panoramic pictures are taken of a view in which the target vehicle is inclined toward the left or right at different angles, and the like. Therefore, it is necessary to find the region where the target vehicle is located, i.e., the region of interest, from the panoramic image, and determine the positions of the respective regions of interest as the respective positions of the target vehicle. Theoretically, the more the number of the foreground pictures is, the more accurate the simulated vehicle running track is, but in practical engineering application, generally three pictures can meet basic steering judgment, and the image data processing complexity can be reduced.

Specifically, in one embodiment, step 120 may comprise: and acquiring different positions of the target vehicle from the plurality of panoramic pictures respectively based on the close-up pictures of the target vehicle by using the recognition model, and connecting the positions of the target vehicle according to the time sequence of the plurality of panoramic pictures to indicate the running track of the target vehicle.

Here, the recognition model may be a recognition model known in the art, such as a license plate recognition model, a vehicle ReID model, or the like. However, those skilled in the art will understand that the identification model herein may also be various other models that can be identified from the complex picture containing the background information based on the specific information of the target vehicle, and this is not described in detail in the present invention.

Next, at step 130, a multitask segmentation network is constructed to segment the plurality of panoramic pictures to obtain road segmentation information and intersection segmentation information of a scene around the target vehicle, and the road segmentation information and the intersection segmentation information are fused with a given transparency to generate a background picture.

In the present invention, the multitask split network is an improved split network (e.g., PSPnet (Scene parsing network)). Specifically, one branch of the multitask split network is to reuse the existing split network (reuse network parameters/weights) to split intersection information from the panoramic picture, and the other branch is to be added separately to split road information from the panoramic picture.

Specifically, in some embodiments, step 130 may include sub-step 132 (not shown in the figures) in which a branch of a multitask splitting network is constructed to split one of a plurality of panoramic pictures to generate intersection splitting information. Here, the intersection segmentation information indicates lane lines, traffic lights, and the like in a scene around the target vehicle, or may further include zebra crossing information of the current intersection. In the present invention, the branch of the multitask split network can reuse the existing split network, such as the existing split network (e.g. PSPnet) for intersection information splitting. In the existing segmentation network, only the information of the intersection where the target vehicle is currently located is considered, such as lane line information (e.g., white solid line, white dotted line, double yellow lines, etc.) and traffic light information of the current intersection and zebra crossing information (e.g., zebra crossing information in front of the vehicle head when the traffic light is red) of the current intersection, and other road and non-road information and zebra crossing information in other directions around the current intersection are not considered.

However, the intersection segmentation information obtained by such a segmentation network is not sufficient to accurately judge the turning information and the illegal state of the vehicle, and therefore, in an embodiment according to the present invention, step 130 further includes a sub-step 134 (not shown in the figure) of constructing another branch of the multitask segmentation network to segment one of the panoramic pictures to generate road segmentation information. Herein, the road segmentation information indicates a road, a non-road, and at least two zebra crossings in a scene around the target vehicle. In some prior arts, the intersection segmentation information may include information of a zebra crossing closest to the vehicle (current zebra crossing), which is not enough for dynamically determining the steering information of the vehicle. Therefore, the road segmentation information of the present invention includes information of at least two zebra crossings in a scene around the target vehicle, for example, information of the zebra crossings at other positions, such as the zebra crossings at opposite intersections and/or both intersections, in addition to the information of the current zebra crossing. In this way, more reference information about the surroundings of the vehicle can be obtained to accurately judge the steering information of the vehicle.

Here, another branch of the multitask split network may be a pruned PSPnet split network, which may be generated by pruning 30 convolutional layers of the PSPnet split network (such as the network structure disclosed by PSPnet official network) to 4, replacing the convolution kernel of 3 × 3 of each PSPnet split network with 1 × 1, 3 × 3, 1 × 1 three convolution kernels, introducing a hole convolution in the convolution network of which 3 × 3 is a convolution kernel, and adding an auxiliary loss function. Fig. 2A and 2B illustrate structural diagrams of a conventional PSPnet split network and an improved PSPnet split network according to an embodiment of the present invention, respectively.

By pruning 30 convolutional layers to 4 convolutional layers, the resource consumption and the time consumption of a display card can be obviously saved, a hole breaking network is added in a PSPnet segmentation network, the hole breaking problem of large target segmentation can be solved, and the receptive field is enlarged under the conditions of not losing the resolution and not increasing parameters. On the one hand, the larger receptive field can be used for detecting and segmenting a larger target, and on the other hand, the resolution is improved to more accurately position the target.

In an embodiment of the invention, the clipped PSPnet split network is a trained clipped PSPnet split network. Training the cropped PSPnet split network may include: extracting a plurality of intersection images; respectively segmenting a road part, a non-road part and a zebra crossing part from each intersection image and respectively marking; and sending the marked road part, the marked non-road part and the marked zebra part into the cut PSPnet segmentation network for training to obtain the trained cut PSPnet segmentation network. That is, the cropped PSPnet split network is trained with a large number of labeled intersection images (similar to the panoramic picture in the original image) to obtain a trained cropped PSPnet split network for subsequent use in generating road split information.

Here, the road division information and the intersection division information may be obtained based on the same panoramic picture of the plurality of panoramic pictures or may be obtained based on different panoramic pictures of the plurality of panoramic pictures.

In addition, step 130 may further include a sub-step 136 (not shown in the figure) in which the road segmentation information is fused with a first transparency and the intersection segmentation information is fused with a second transparency to generate the background picture. Here, fusing the road segmentation information and the intersection segmentation information with corresponding transparency means that the Alpha channel of the picture of the road segmentation information and the Alpha channel of the intersection segmentation information are set to present a distinction degree on a foreground picture formed by fusing. Typically, the first transparency is less than the second transparency, e.g., the first transparency can be 0.4, the second transparency can be 0.6, etc.

Continuing with the method 100, at step 140, the foreground pictures from step 120 and the background pictures from step 130 are combined to produce forward data for the target vehicle.

Specifically, in one embodiment, the interest areas of the target vehicle may be respectively cut from the plurality of panoramic pictures and superimposed on the background picture according to respective positions, and the centers of the interest areas of the plurality of panoramic pictures may be connected to indicate the driving track of the target vehicle. The image resulting from the superimposition may be referred to as forward data of the target vehicle.

Next, at step 150, forward data of the target vehicle may be operated forward based on the steering classification network to determine steering information of the target vehicle. Here, the steering information may include, for example, one of a left turn, a right turn, and a straight line.

In some embodiments, the steering classification network is a tailored google lenetv3 network, which may be formed by replacing 10 inceptionV1 modules of a modular inclusion structure of a google lenet network (such as the network structure disclosed by the google lenet official network) with 3 inceptionV3 modules, and deleting the auxiliary loss function. Fig. 3A and 3B show schematic structural diagrams of a conventional google lenet network and an improved google lenet v3 network according to an embodiment of the invention, respectively.

Fig. 4A to 4F show a simulation experiment scenario for determining steering information of a vehicle according to the present invention. Wherein, fig. 4A shows a simulated experiment scenario using only intersection partition information according to the present invention, fig. 4B shows a simulated experiment scenario hiding other vehicles on the basis of fig. 4A, fig. 4C shows a simulated experiment scenario in the case where intersection partition information is added, fig. 4D shows a simulated experiment scenario where a turn indicator line (e.g., left turn, straight going) and a track line are added on the basis of fig. 4A, fig. 4E shows another simulated experiment scenario using only intersection partition information according to the present invention, and fig. 4F shows a simulated experiment scenario using only intersection partition information according to the present invention and road partition information and an auxiliary line.

Simulation experiment results show that the Google LeNetV3 network cut by the method can realize balance on model precision and running speed, and ensure the accuracy and robustness of classification effect. The size of the ca ffemodel of the cut Google LeNetV3 is only 3.5M, and the accuracy of the determination of the vehicle steering information can be 98% in the actual engineering test, which is increased by 5-6 percentage points compared with the best model fusion method in the prior art, and the speed is doubled.

In one embodiment, the tailored google lenetv3 network may be a trained tailored google lenetv3 network. Training the tailored google lenetv3 network includes: a training image is acquired. The training image includes a plurality of panoramic training pictures and a close-up photograph of a training vehicle, each panoramic training picture including the training vehicle and a scene surrounding the training vehicle. And acquiring training vehicles in the plurality of panoramic training pictures by using the license plate recognition model or the vehicle ReID model, and capturing the training vehicles and corresponding positions from the panoramic training pictures as foreground training pictures. Next, the above-mentioned multitask segmentation network may be constructed or utilized to segment the panoramic training picture to obtain road segmentation information and intersection segmentation information of the scene around the training vehicle, and to fuse at different transparencies to produce a background training picture. And superposing the foreground training picture and the background training picture to generate a classification training picture, and performing classification marking on the training vehicles on the classification training picture, wherein the classification marking comprises left-turn, right-turn or straight-going marking. This is done for a large number of training images and a large number of labeled classified training pictures are fed into the cropped google lenetv3 network to produce a trained cropped google lenetv3 network.

Although some vehicle driving classification models are based on deep learning, the data acquisition has the problems of improper pinching, loss (only a single target vehicle is taken), redundancy (an entire image is directly trained), and the like, so that the accuracy of the final result is low. In practical application, various logic constraints have to be added to customize various parameters, so that the robustness of the algorithm is poor. The invention adopts a multi-task network constructed based on an intersection segmentation network PSPnet in the original engineering to segment roads (for example, three types of roads, non-roads and zebra crossings are segmented), combines the uniqueness of a vehicle posture classification task, fuses target vehicles based on the segmentation result, and performs classification training by using GoogleLeNet. And a hole breaking network and the like are added in the network design to solve the problem of large target segmentation and hole breaking, and the compression network is pruned to save the resource consumption and the time consumption of the display card.

Through the above steps 110 to 150, for any given original image including the target vehicle, the steering information of the target vehicle, i.e., whether the target vehicle is turning left, turning right, or going straight, can be accurately detected.

However, in the actual illicit-state auditing application, in addition to determining the steering information of the target vehicle, it is often necessary to determine the illicit state of the target vehicle, such as whether the target vehicle is illicit and what is its specific illicit behavior, in conjunction with the steering information. In this case, the method 100 may further include: and inputting the steering information of the target vehicle into an illegal type judgment module to determine the illegal state of the target vehicle.

For example, an illegal type judgment model for judging illegal behaviors such as traffic light running and guidance violation based on steering information is generally included in an illegal auditing system, and in this case, the process of determining the steering information of the target vehicle can be regarded as a basic module for illegal type judgment. The method can be used for more accurately and quickly judging the steering information of the target vehicle, so that the accuracy of judging main illegal types such as traffic light running, guidance violation and the like in the conventional illegal auditing system is improved.

FIG. 5 shows a schematic block diagram of an example device 200 that may be used to implement an embodiment of the invention. The apparatus 200 may be part of a law violation review system of a traffic police department, for example. As shown, device 200 may include one or more Central Processing Units (CPUs) 210 (only one shown schematically) that may perform various suitable actions and processes in accordance with computer program instructions stored in a Read Only Memory (ROM)220 or loaded from a storage unit 280 into a Random Access Memory (RAM) 230. In the RAM 230, various programs and data required for the operation of the device 200 can also be stored. The CPU 210, ROM 220, and RAM 230 are connected to each other through a bus 240. An input/output (I/O) interface 250 is also connected to bus 240.

A number of components in device 200 are connected to I/O interface 250, including: an input unit 260 such as a keyboard, a mouse, etc.; an output unit 270 such as various types of displays, speakers, and the like; a storage unit 280 such as a magnetic disk, an optical disk, or the like; and a communication unit 290 such as a network card, modem, wireless communication transceiver, etc. The communication unit 290 allows the device 200 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The method 100 described above may be performed, for example, by the processing unit 210 of one device 200 or a plurality of devices 200. For example, in some embodiments, the method 100 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 280. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 200 via the ROM 220 and/or the communication unit 290. When the computer program is loaded into RAM 230 and executed by CPU 210, one or more of the operations of method 100 described above may be performed. Further, the communication unit 290 may support wired or wireless communication functions.

The method 100 and the apparatus 200 for determining steering information of a vehicle according to the present invention are described above with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that the performance of the steps of the method 100 is not limited to the order shown in the figures and described above, but may be performed in any other reasonable order. For example, the

above steps

120 and 130 may be performed in a different order than shown or in parallel. Further, the device 200 also need not include all of the components shown in fig. 5, it may include only some of the components necessary to perform the functions described in the present invention, and the manner in which these components are connected is not limited to the form shown in the drawings.

The present invention may be methods, apparatus, systems and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therein for carrying out aspects of the present invention.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

The computer program instructions for carrying out operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing an electronic circuit, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), with state information of computer-readable program instructions, which can execute the computer-readable program instructions.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processing unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining steering information for a vehicle, comprising:

acquiring an original image, wherein the original image comprises a plurality of panoramic pictures and at least one close-up picture of a target vehicle, and each panoramic picture comprises the target vehicle and a scene around the target vehicle;

respectively identifying the interest area and the relative position of the target vehicle in the plurality of panoramic pictures by utilizing an identification model to generate a foreground picture;

constructing a multitask segmentation network to segment the plurality of panoramic pictures to obtain road segmentation information and intersection segmentation information of a scene around the target vehicle, and fusing the road segmentation information and the intersection segmentation information with a given transparency to generate a background picture;

combining the foreground picture and the background picture to generate forward data for the target vehicle; and

forward data of the target vehicle is subjected to a forward operation based on a steering classification network to determine steering information of the target vehicle, wherein the steering information comprises one of left turn, right turn and straight movement.

2. The method of claim 1, wherein generating a foreground picture comprises:

obtaining, with the recognition model, different locations of the target vehicle from the plurality of panoramic pictures, respectively, based on the close-up pictures of the target vehicle; and

and connecting the positions of the target vehicle according to the time sequence of the plurality of panoramic pictures to indicate the running track of the target vehicle.

3. The method of claim 1, wherein generating a background picture comprises:

constructing a branch of the multitask segmentation network to segment one panoramic picture of the panoramic pictures to generate intersection segmentation information, wherein the intersection segmentation information indicates a lane line in a scene around the target vehicle, a zebra crossing of a current intersection and a traffic light;

constructing another branch of the multitask segmentation network to segment one of the plurality of panoramic pictures to generate road segmentation information indicating roads, non-roads, and at least two zebra crossings in a scene surrounding the target vehicle; and

and fusing the road segmentation information with a first transparency and the intersection segmentation information with a second transparency to generate the background picture.

4. The method of claim 3, wherein another branch of the multitask split network is a pruned PSPnet split network, wherein

The clipped PSPnet split network is generated by pruning 30 convolutional layers of the PSPnet split network to 4, replacing the convolution kernel of each 3 x 3 of the PSPnet split network with three convolution kernels of 1 x 1, 3 x 3, 1 x 1, introducing a hole convolution in the convolution network with convolution kernel 3 x 3, and adding an auxiliary loss function.

5. The method of claim 4, wherein the clipped PSPnet split network is a trained clipped PSPnet split network, and wherein training the clipped PSPnet split network comprises:

extracting a plurality of intersection images;

respectively segmenting a road part, a non-road part and a zebra crossing part from each intersection image and respectively marking; and

and sending the marked road part, the marked non-road part and the marked zebra part into the cut PSPnet segmentation network for training to obtain the trained cut PSPnet segmentation network.

6. The method of claim 1, wherein said steering classification network is a tailored google lenetv3 network, wherein said tailored google lenetv3 network is formed by replacing 10 inceptionV1 modules of a modular inclusion structure of a google lenet network with 3 inceptionV3 modules and deleting ancillary loss functions.

7. The method of claim 6, wherein the tailored GoogleLeNetV3 network is a trained tailored GoogleLeNetV3 network, and wherein training the tailored GoogleLeNetV3 network comprises:

acquiring a training image, wherein the training image comprises a plurality of panoramic training pictures and a close-up photo of a training vehicle, and each panoramic training picture comprises the training vehicle and a scene around the training vehicle;

acquiring the training vehicles in the panoramic training pictures by using a license plate recognition model or a license plate ReID model, and intercepting the training vehicles and corresponding positions from the panoramic training pictures as foreground training pictures;

segmenting the panoramic training picture by using the multitask segmentation network to obtain road segmentation information and intersection segmentation information of scenes around the training vehicle, and fusing with different transparencies to generate a background training picture;

superimposing the foreground training picture and the background training picture to generate a classification training picture;

carrying out classification marking on the training vehicles on the classification training pictures, wherein the classification marking comprises left-turn, right-turn or straight-going marking; and

the labeled classified training pictures are fed into the cropped google lenetv3 network to produce a trained cropped google lenetv3 network.

8. The method of claim 1, wherein the method further comprises:

and inputting the steering information of the target vehicle into an illegal type judgment module to determine the illegal state of the target vehicle.

9. An apparatus for determining steering information of a vehicle, comprising:

a memory having computer program code stored thereon; and

a processor configured to execute the computer program code to perform the method of any of claims 1 to 8.

10. A computer-readable storage medium having computer program code stored thereon, the computer program code, when executed, performing the method of any of claims 1 to 8.