WO2022088766A1 - Image processing method, medium, device, and image processing system - Google Patents

Abstract

Provided are an image processing method, a medium, a device, and an image processing system. The image processing method comprises: acquiring a panoramic image around a vehicle and current travel information of the vehicle, wherein the panoramic image includes angle information corresponding to presentation content of the panoramic image; acquiring an image display requirement that matches the current travel information of the vehicle, wherein the image display requirement comprises a display angle; and on the basis of the angle information that is consistent with the display angle, extracting a panoramic image portion corresponding to the angle information, so as to present, in an in-vehicle display device and in combination with a change in the image display requirement, the panoramic image portion. By means of providing a panoramic image, an image of each angle of a vehicle can be displayed, and the collected panoramic image is processed, such that a better safety guarantee is brought about for the life of a vehicle owner, and more convenient and fast information is provided.

Description

Image processing method, medium, device and image processing system technical field

The invention belongs to the technical field of image processing, and relates to an image processing method, medium, equipment and image processing system.

Background technique

With the development of imaging technology, obtaining panoramic images around the vehicle brings more convenience to users, especially when driving through a 360-degree circle, which is regarded as a better panoramic visual experience for users.

However, when using the panoramic images generated in the prior art for image processing and analysis, there are the following defects: the panoramic images generated by the existing vehicle-mounted 360 surround view system are mostly realized by splicing and synthesizing 4 to 8 cameras, and the When the panoramic image is displayed, there will be problems such as lack of three-dimensional image, smear, superimposition, deformation, and display dead angle. Further, the car owner has a poor experience when viewing the displayed panoramic image or obtaining relevant information from the panoramic image.

SUMMARY OF THE INVENTION

In view of the above-mentioned shortcomings of the prior art, one object of the present invention is to provide an image processing method, medium, electronic device and image processing system, the advantages of which are that the panoramic image around the vehicle can be highly processed in combination with the current driving information of the vehicle. Partial image display of quality picture.

Another object of the present invention is to provide an image processing method, medium, electronic device and image processing system, the advantages of which are that the display angle of the collected panoramic image can be automatically adjusted according to the traveling direction of the vehicle.

Another object of the present invention is to provide an image processing method, medium, electronic device and image processing system, the advantages of which are that the display angle of the collected panoramic image can be automatically adjusted according to the position of the target object outside the vehicle.

Another object of the present invention is to provide an image processing method, medium, electronic device and image processing system, the advantages of which are that a complete and clear image at any angle in a panoramic image can be manually adjusted and viewed according to user requirements.

Another object of the present invention is to provide an image processing method, medium, electronic device and image processing system, the advantages of which are that when a panoramic image is partially displayed, the lack of three-dimensionality, smear, ghosting, and deformation of the image is avoided. , There are problems such as display dead corners.

Another object of the present invention is to provide an image processing method, medium, electronic device and image processing system, the advantages of which are that the panoramic images collected by the vehicle can be shared on different terminals in real time for viewing by different target groups.

Another object of the present invention is to provide an image processing method, a medium, an electronic device and an image processing system, which have the advantage of being able to identify the congestion situation and prompt the owner to move the car in a parking situation.

Another object of the present invention is to provide an image processing method, medium, electronic device and image processing system, the advantages of which are that the identity and intention of persons approaching the vehicle can be identified and the vehicle owner is prompted.

Another object of the present invention is to provide an image processing system, the advantage of which is that, through the communication between the panoramic image device and the electronic device, the panoramic image generated by the panoramic image device is sent to the electronic device for image processing, so that in the subsequent The electronic device can automatically adjust the display angle of the panoramic image according to the traveling direction of the vehicle/position of the target object in the parking space, wherein the installation position of the panoramic image device of the present invention on the vehicle is optimized, so that the probability of the panoramic image device being collided is the lowest in the whole vehicle , and the integrated panoramic lens greatly reduces the difficulty of installation and post-maintenance.

In order to achieve the above object and other related objects, one aspect of the present invention provides an image processing method, the method includes the following steps: acquiring a panoramic image around the vehicle and current driving information of the vehicle; angle information; obtain image display requirements that match the current driving information of the vehicle; the image display requirements include a display angle; based on the angle information consistent with the display angle, extract the panoramic image portion corresponding to the angle information , so that the panoramic image portion is presented in the in-vehicle display device in combination with the change of the image display requirements.

In an embodiment of the present invention, the current driving information of the vehicle includes at least current steering wheel angle information, travel speed information and gear information of the vehicle. Therefore, the corresponding partial panoramic image can be displayed in combination with the current steering wheel angle information, driving speed information and gear information of the vehicle.

In an embodiment of the present invention, the image display requirement further includes at least a refresh direction and a refresh frequency; the vehicle determines the current steering wheel angle information and gear information of the vehicle through preset sensors and/or navigation positioning data. and driving speed information; the acquiring the image display requirement matching the current driving information of the vehicle includes the following steps: calculating the display angle of the panoramic image based on the sensor and/or the navigation and positioning data; The bit information detects the advancing and retreating state of the vehicle, and then determines the refresh direction of the panoramic image based on the advancing and retreating state; and determines the refresh frequency of the panoramic image based on the traveling speed information. Therefore, the image display requirement can be determined through the information provided by the vehicle data collected by the sensor and/or the navigation and positioning data, so that the displayed partial panoramic image changes with the state of the vehicle.

In an embodiment of the present invention, the extracting the panoramic image portion corresponding to the angle information based on the angle information consistent with the display angle includes the following steps: establishing an angle coordinate system based on the angle information to The angular coordinate system is superimposed on the panoramic image; the display angle is positioned in the angular coordinate system; and the picture to be displayed at the position where the display angle is positioned is extracted. Therefore, the deflection angle of the vehicle can be associated with the angle information of the panoramic image through the angle coordinate system, so as to locate the content to be displayed in the panoramic image through the determined display angle.

In an embodiment of the present invention, the method further includes the step of: presenting the panoramic image via an electronic device. In this way, the electronic device for image processing can be associated with the server to provide multi-channel display of different electronic devices for the generated panoramic image, and to share the image with others; in addition, the generated panoramic image can also be backed up storage and system management.

In an embodiment of the present invention, the method further includes: when the vehicle is in a parked state, performing image recognition on the panoramic image to recognize the passing situation outside the vehicle; and performing voice recognition on the panoramic image , to recognize the voice content of the person outside the vehicle; based on the meeting situation and the voice content, determine whether the car needs to be moved; and in response to determining that the car needs to be moved, send a car-moving warning to the owner of the vehicle. In this way, when the vehicle is in a parked state, the “sentry” mode can be turned on through the recognition of the panoramic image to intelligently identify the traffic jam, so as to issue a car-moving warning to the car owner, so that the car owner can move the vehicle at the first time, and the road can be viewed more quickly. Clear the situation.

In an embodiment of the present invention, the method further includes the following steps: when the vehicle is in a parked state, performing image recognition on the panoramic image through machine learning to identify the identity features of persons outside the vehicle; identity feature, infer the intention of the person outside the vehicle to approach the vehicle; and in response to determining the intention of the person outside the vehicle to approach the vehicle, send prompt information to the owner of the vehicle for indicating the person outside the vehicle The intent of the person to approach the vehicle and/or the recommended response. In this way, when the vehicle is in a parked state, people around the vehicle can be intelligently identified through the recognition of panoramic images, the intention of people outside the vehicle to approach the vehicle can be predicted, and a prompt message can be sent to the vehicle owner, so that the vehicle owner can take timely measures for various abnormal situations. Countermeasures can also make the owner try to avoid abnormal parking.

To achieve the above objects and other related objects, another aspect of the present invention provides a non-transitory computer-readable storage medium having stored thereon machine-executable instructions that, when executed, cause a machine to execute the steps of the method.

To achieve the above objects and other related objects, another aspect of the present invention provides an electronic device, comprising: a memory configured to store one or more computer programs; and a processor coupled to the memory and configured to execute the The one or more computer programs cause the electronic device to perform the method.

In order to achieve the above object and other related objects, a final aspect of the present invention provides an image processing system, the image processing system includes: a panoramic imaging device and the electronic device; the panoramic imaging device is connected to the electronic device; The panoramic image device is used to generate a panoramic image and send it to the electronic device; the electronic device is used to obtain the panoramic image around the vehicle and the current driving information of the vehicle; angle information; obtain image display requirements that match the current driving information of the vehicle; the image display requirements include a display angle; based on the angle information consistent with the display angle, extract the panoramic image portion corresponding to the angle information , so that the panoramic image portion is presented in the in-vehicle display device in combination with the change of the image display requirements.

Description of drawings

FIG. 1 is a schematic flowchart of an image processing method of the present invention in an embodiment;

FIG. 2 is a schematic diagram of driving information in an embodiment of the image processing method of the present invention;

FIG. 3 is a flowchart showing an image display requirement acquisition process in an embodiment of the image processing method of the present invention;

FIG. 4A is a schematic diagram showing the calculation of a display angle in an embodiment of the image processing method of the present invention;

FIG. 4B is a schematic diagram showing the calculation of the display angle in another embodiment of the image processing method of the present invention;

5 is a schematic diagram illustrating panoramic image extraction in an embodiment of the image processing method of the present invention;

FIG. 6A shows a flow chart of establishing an angular coordinate system in an embodiment of the image processing method of the present invention;

6B is a schematic diagram showing the establishment of an angular coordinate system in an embodiment of the image processing method of the present invention;

7 is a flowchart showing a panoramic image display in an embodiment of the image processing method of the present invention;

FIG. 8 is a schematic diagram showing the monitoring of persons outside the vehicle in an embodiment of the image processing method of the present invention;

9 is a schematic diagram showing a traffic jam detection in an embodiment of the image processing method of the present invention;

FIG. 10 is a schematic diagram of abnormal parking in an embodiment of the image processing method of the present invention;

11 is a schematic diagram showing monitoring of children and carried items in an embodiment of the image processing method of the present invention;

FIG. 12 is a schematic diagram showing the structural connection of the electronic device of the present invention in an embodiment;

FIG. 13 is a schematic structural diagram of an image processing system of the present invention in an embodiment;

FIG. 14 is a schematic diagram illustrating an application of the image processing system of the present invention in an embodiment.

Component label description

1 Electronic equipment

11 Processor

12 Memory

2 Panoramic imaging equipment

Steps S11～S14

Steps S121～S123

Steps S131～S133

S131A～S131C Steps

Steps S141～S142

Detailed ways

The embodiments of the present invention are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other under the condition of no conflict.

It should be noted that the diagrams provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic way, so the diagrams only show the components related to the present invention rather than the number, shape and For dimension drawing, the type, quantity and proportion of each component can be changed at will in actual implementation, and the component layout may also be more complicated.

The image processing method of the present invention can display images from various angles of the vehicle by providing panoramic images, and process the collected panoramic images, thereby bringing more safety assurance and more convenient information to the life of the vehicle owner.

The principle and implementation of an image processing method, medium, device and image processing system of this embodiment will be described in detail below with reference to FIGS. 1 to 14 , so that those skilled in the art can understand the image of this embodiment without creative work. Processing method, medium, device and image processing system.

Please refer to FIG. 1 , which is a schematic flow chart of the image processing method of the present invention in one embodiment. As shown in Figure 1, the image processing method specifically includes the following steps:

S11: Acquire a panoramic image around the vehicle and current driving information of the vehicle; the panoramic image includes angle information corresponding to its presentation content.

Specifically, the panoramic image is acquired through a 720° panoramic lens. The 720° panoramic lens can be produced by integrating two front and rear wide-angle fisheye lenses. The obtained image is a full-angle image of 720° up, down, left, and right. The fixation scheme of the panoramic lens is preferably installed on the top of the vehicle, and the picture quality can be improved. At the 4K level, there is no problem of lack of three-dimensionality and ghosting. Since the panoramic lens is installed at the top of the vehicle, the collision probability is the lowest in the whole vehicle; in addition, the difficulty of installation and post-maintenance is greatly reduced by having only one panoramic lens.

Please refer to FIG. 2 , which is a schematic diagram of driving information in an embodiment of the image processing method of the present invention. As shown in Figure 2, the current driving information of the vehicle includes at least current steering wheel angle information, travel speed information and gear information of the vehicle.

S12: Acquire an image display requirement matching the current driving information of the vehicle; the image display requirement includes a display angle.

In an embodiment, the image display requirement further includes at least a refresh direction and a refresh frequency; the vehicle determines the current steering wheel angle information, gear information and driving speed of the vehicle through preset sensors and/or navigation positioning data. information. In an implementation of this embodiment, the steering wheel angle information is detected by a steering wheel angle sensor of the vehicle, the gear information is detected by a transmission gear sensor of the vehicle, and the odometer sensor or wheel speed sensor of the vehicle is used to detect all the information. For the driving speed information, the display angle, refresh direction, and refresh frequency of the panoramic image are determined correspondingly through the detected information. In another implementation of this embodiment, when the steering wheel rotates for many times, the small rotational displacement of the vehicle is determined by the high-precision navigation positioning data, so that the rotational direction of the vehicle is estimated from the side, and then the display angle is determined according to the rotational direction, Combined with the gear information and driving speed information measured by the vehicle sensors, the image display requirements of the panoramic image are comprehensively determined. For example, if the user turns the vehicle to the left at an intersection, at this time, the steering wheel is turned to the left for many times, and the tangent direction of each point on the entire trajectory of the vehicle's U-turn is taken as the turning direction of the vehicle. Further, the centerline of the road can be selected. As a reference line, the angle of the tangential direction of each point is determined from this, and the steering angle of the vehicle is then known. In another implementation of this embodiment, the three parameters of the steering wheel angle information, gear information and driving speed information are inferred from the navigation and positioning data, and then the image display requirements are determined according to the three parameters, wherein the estimated value of the navigation and positioning data is determined. The method can be that the vehicle communicates with the navigation and positioning device, and transmits the steering wheel angle information, gear information and driving speed information to the navigation and positioning device to obtain, specifically, the collected steering wheel angle information, gear information and driving speed through the vehicle sensor. The information is transmitted to the navigation and positioning device through Bluetooth, wifi and other communication methods (for example, when the execution device of the image processing method is a mobile phone, the mobile phone receives the steering wheel angle information, gear information and driving speed information collected by the vehicle sensor to determine the display angle of the panoramic image) ; It can also be the navigation positioning data to determine the vehicle's deflection direction, forward and backward, or travel speed and other information according to the change of the vehicle's position on the map. For example, the vehicle is compared to a point in the navigation map. The position on the map is constantly changing. When the point is shifted to the left relative to the current direction, it is determined that the vehicle is turning left. When the point moves backward relative to the current position, it is determined that the vehicle is moving backward. To a certain extent, it can reflect the speed of the vehicle.

Please refer to FIG. 3 , which is a flow chart of acquiring an image display requirement in an embodiment of the image processing method of the present invention. As shown in Figure 3, S12 includes:

S121: Calculate the display angle of the panoramic image based on the sensor and/or the navigation and positioning data. For example, when the vehicle is driving forward, the front screen is displayed, and when the vehicle turns right, the screen can be automatically rotated with the rotation of the steering wheel.

Please refer to FIG. 4A , which is a schematic diagram showing the calculation of the display angle in an embodiment of the image processing method of the present invention. As shown in FIG. 4A , the gear information includes D gear and R gear, the steering wheel angle information is detected by a steering wheel angle sensor of the vehicle, and the steering wheel angle information includes a left turn angle and a right turn angle. The calculation of the display angle is illustrated by the following four cases.

In response to determining that the steering wheel angle information is the D gear and the left turn angle, subtract the left turn angle from the first preset angle as the display angle. Wherein, the first preset angle is 360°. For example, if the gear is D-forward and the steering wheel angle is 30 degrees to the left, the corresponding coordinate image is the direction of the display angle of 360°-30°=330°.

In response to determining that the steering wheel angle information is the D gear and the right turn angle, the right turn angle is used as the display angle. For example, if the gear is D-forward and the steering wheel angle is 20 degrees to the right, the corresponding coordinate image is the direction of the display angle of 20 degrees.

In response to determining that the steering wheel angle information is the R gear and the left turn angle, the second preset angle is added to the left turn angle as the display angle. Wherein, the second preset angle is 180°. For example, if the gear is R-reverse, and the steering wheel angle is 40 degrees to the left, the corresponding coordinate image is the direction of the display angle of 180°+40°=220°.

In response to determining that the steering wheel angle information is the R gear and the right turn angle, the second preset angle is subtracted from the right turn angle as the display angle. For example, if the gear is R-reverse, and the steering wheel angle is 10 degrees to the right, the corresponding coordinate image is the direction of the display angle of 180°-10°=170°.

Please refer to FIG. 4B , which is a schematic diagram of calculating a display angle in another embodiment of the image processing method of the present invention. As shown in FIG. 4B , the display angle of the panoramic image is calculated through the navigation and positioning data, so as to realize the tracking and shooting of the driving image. During the driving process, the panoramic image can be used to record the surrounding images, especially the target objects or objects that the owner is interested in. Predict objects that may be of interest to car owners.

Specifically, the POI (Point of Information, information point or point of interest) location can be identified through the panoramic image, and the driver can be prompted by voice according to the surrounding POI location data, "Your right front is the Dongfeng Pearl TV Tower, whether you need to track and shoot", or The driver can use voice control or click the screen position to circle the surrounding POI content for tracking shooting. After the shooting instruction is output, the POI is identified in the obtained panoramic image based on the GPS coordinates, outline, and obvious marks of the POI location, so as to track and shoot the POI location image.

In practical applications, as shown in (a) in Figure 4B, the POI point is the Oriental Pearl TV Tower, and the angle between the POI point and the vehicle is calculated according to the latitude and longitude: the positioning information of the vehicle is determined through the navigation and positioning data to obtain the vehicle coordinates. The vehicle coordinates and the coordinates of the POI location are determined in the navigation map, and the number of included angles can be calculated through angle calculation. For example, if the vehicle coordinates are (0, 0) and the POI point position is (x, y), then tan∠a=|x|÷|y|, so ∠a=arctan(|x|÷|y |). According to the included angle, the panoramic image of the corresponding angle can be displayed.

Further, as shown in (b) in FIG. 4B , the POI image is displayed in real time by calculating the angle every second, and the POI point image is displayed on the vehicle. For example, the clip between the Oriental Pearl TV Tower and the vehicle is determined by the navigation and positioning data. The angle gradually changes from 30 degrees to 75 degrees, and accordingly, the display angle of the panoramic image also changes from 30 degrees to 75 degrees.

Further, while tracking shooting and panoramic image display for the Oriental Pearl TV Tower, it provides users with voice commentary of the information related to the Oriental Pearl TV Tower, playing the role of a "smart tour guide".

In this way, the POI can be intelligently identified according to the owner's preferences and the instructions issued, and the display angle of the POI in the panoramic image can be determined, and then the corresponding panoramic image part can be continuously tracked and displayed; and the POI coordinates in the map can be intelligently predicted through contour recognition or map. The vehicle passes through landmarks (targets or events) in the area, such as landmark buildings that the owner may be interested in, so as to realize automatic tracking shooting and display of panoramic images and provide audio guide functions.

S122: Detect the forward/backward state of the vehicle based on the gear information, and then determine the refresh direction of the panoramic image based on the forward/backward state.

S123: Determine the refresh frequency of the panoramic image based on the traveling speed information.

S13 , based on the angle information consistent with the display angle, extract the panoramic image part corresponding to the angle information, so as to present the panoramic image part in the in-vehicle display device in combination with the change of the image display requirement.

Please refer to FIG. 5 , which is a schematic diagram of panoramic image extraction in an embodiment of the image processing method of the present invention. As shown in Figure 5, S13 includes:

S131 , establishing an angle coordinate system based on the angle information, so as to superimpose the angle coordinate system on the panoramic image.

Please refer to FIG. 6A , which is a flow chart of establishing an angular coordinate system in an embodiment of the image processing method of the present invention. As shown in Figure 6A, S131 includes:

S131A, converting the panoramic image into a spherical image. Specifically, the panoramic image is a 2:1 ratio image, and the image is folded into a spherical image through a deformed graphics processing algorithm. Specifically, the implementation steps of the deformed image processing algorithm include: first, enlarging the panoramic image into a rectangular image to perform local transformation, for example, after a source pixel point l(x, y) of the input panoramic image is enlarged by the Seam Carving algorithm, The position is moved to l'(x, y), thus, its motion field (optical flow field) u(x, y)=l'(x, y)-l(x, y) can be calculated to calculate the panorama After the image becomes a rectangular image, the u(x, y) motion vector of each pixel point; then, the rectangular image is meshed, and the new position of each mesh vertex is recorded according to -u(x, y); finally , the new position of the mesh vertex is used as the control vertex, and the panorama image is globally deformed by the deformation method of similarity transformation to obtain a spherical image.

S131B, in the spherical image, the horizontal line is used as the center line, and the center horizontal image is determined.

S131C, establishing the angle coordinate system for the center horizontal image.

S132. Position the display angle in the angle coordinate system.

S133, extracting the picture to be displayed with the display angle at the location.

Please refer to FIG. 6B , which is a schematic diagram of establishing an angular coordinate system in an embodiment of the image processing method of the present invention. As shown in Fig. 6B, Fig. 6B presents a real view of the center horizontal image, and the vehicle is located at the center of the circular angular coordinate system, so that all scenes around the circumference can be viewed. For example, when the vehicle is driving, the gear is D-forward and the steering wheel angle is 90 degrees to the left. At this time, the orientation of the vehicle is positioned to the 270° direction in FIG. 6B , so the 270° direction is used as the display angle. , a tall building can be photographed directly in front of the vehicle as the part of the presented panoramic image corresponding to the display angle, that is, the picture to be displayed at the current moment.

Please refer to FIG. 7 , which is a flow chart of displaying a panoramic image in an embodiment of the image processing method of the present invention. As shown in FIG. 7 , the image processing method further includes S14. In one embodiment, S14 includes:

S141 , continuously determining the pictures to be displayed at different times in combination with changes in the image display requirements.

S142: Display a plurality of the pictures to be displayed in chronological order, so as to perform corresponding display of different parts of the panoramic image according to the real-time changes of the image display requirements.

In one embodiment, the image processing method further includes:

S15. Present the panoramic image via an electronic device.

In one embodiment, the panoramic image is sent to an electronic device for displaying the panoramic image, so as to present the panoramic image.

In another embodiment, the panoramic image is sent to a server, so as to send the panoramic image to an electronic device for displaying the panoramic image via the server, so as to present the panoramic image.

Specifically, the panoramic images uploaded on the cloud can be used by family members and friends to view the panoramic videos around driving. As the content support for online travel, family members can see all the scenery on the road at home. In practical applications, the viewing method of the panoramic image may be through a car, a mobile phone, an IOT (The Internet of Things) device, a HUD (Head Up Display, head-up display) device, a holographic image device, or other devices with display. function of the device is displayed. The panoramic image can be viewed anytime, anywhere, not only when driving under safe conditions, such as in traffic jams or when driving normally, to view the images in the blind spot of the driving, but also through the in-vehicle display device when parking, or when relatives and friends are in the car. It can be viewed at home or other venues through mobile phones or IoT devices, or when a traffic accident occurs, it can be retrieved and viewed through the work equipment of the traffic department.

The panoramic image described in the present invention performs monitoring in the "sentry" mode during the parking period of the vehicle owner, and judges the possible abnormality or sudden situation of the vehicle by recognizing the shooting content of the panoramic image.

In practical application scenarios, the panoramic image records abnormal vehicle movement, collision, scratches and other information throughout the entire process, providing evidence for later traceability.

In practical application scenarios, when the car owner temporarily stops (going to buy medicine or buy urgently needed items), the recording and identification of panoramic images can be used to identify people outside the car approaching the vehicle. In the actual situation, the person outside the car can be the owner of the car who temporarily stops to block the road and urgently needs to contact the owner to move the car. It can be the person outside the car who advertises, the traffic police with the ticket, or the security personnel of the nearby building, or the person who is about to enter the vehicle when parking. and those who got out of the vehicle. Please refer to FIG. 8 , which is a schematic diagram of monitoring of persons outside the vehicle in an embodiment of the image processing method of the present invention. As shown in Figure 8, a 720-degree panoramic lens captures a person outside the vehicle approaching the vehicle, and then performs image recognition on the panoramic image to analyze the approaching intention of the person outside the vehicle.

On the one hand, the people outside the car stopped temporarily for the car owner to block the road, and urgently needed to contact the car owner to move the car. Please refer to FIG. 9 , which is a schematic diagram of traffic jam detection in an embodiment of the image processing method of the present invention. As shown in Figure 9, when the car owner temporarily parked and left the vehicle, the panoramic image records the surrounding road conditions of the vehicle. The image information in the panoramic image shows that the distance between car B and car C and the owner's vehicle is relatively small. The lateral distance between the rear of the owner's vehicle and the fronts of cars B and C is small, and there is a possibility of collision when continuing to drive, and it is determined that the meeting is poor. Specifically, the rear of the vehicle in the panoramic image and the front of car B Perform contour extraction with the front of car C, calculate the lateral distance D1 between the rear of the vehicle and the front of car B, calculate the lateral distance D2 between the rear of the vehicle and the front of car C, and calculate D1 and D2 according to the shooting ratio of the panoramic image. Converted to the actual horizontal distances d1 and d2, when there is a distance d1 or d2 less than 10cm, it is judged that the car meeting is poor; the people outside the car (the driver who walked out of the C car) provided by the audio information in the panoramic image. The conversation information makes a judgment on whether the car owner needs to move the car. For example, when a person outside the car is saying "the current distance cannot be passed, you need to contact the car owner to move the car". Through speech content recognition, the keywords "distance" and "moving car" in the speech are extracted, and then the natural language processing method is used to analyze the The extracted keywords are semantically recognized to determine that the car owner needs to move the car. Specifically include the following steps:

(A1) When the vehicle is in a temporary parking state, image recognition is performed on the panoramic image to identify the meeting outside the vehicle, which specifically includes the following steps:

(A1-1) Identifying the other person's vehicle in the panoramic image, and calculating the distance between the other person's vehicle and the vehicle according to the rendering ratio of the panoramic image and the real scene.

(A1-2) Compare the vehicle distance with a preset vehicle distance threshold to evaluate the possibility that the other person's vehicle will pass normally.

(A1-3) In response to determining that the vehicle distance is smaller than the preset vehicle distance threshold, it is determined that the meeting situation outside the vehicle is congestion.

(A2) Perform voice recognition on the panoramic image to recognize the voice content of people outside the vehicle.

(A3) Determine whether it is necessary to move the car according to the meeting situation and the voice content.

(A4) In response to determining that a manoeuvring is required, a manoeuvring alert is sent to the owner of the vehicle.

On the other hand, the person outside the vehicle is the person outside the vehicle who advertises, the traffic police who post the ticket, or the security guard of a nearby building. Please refer to FIG. 10 , which is a schematic diagram of abnormal parking in an embodiment of the image processing method of the present invention. As shown in Figure 10, in the panoramic image obtained by the 720-degree panoramic lens, it is recognized that there is a person outside the vehicle in work clothes approaching the vehicle. Further identify the work clothes as security clothes, volunteer clothes, traffic police clothes or company shop clothes, and identify the carry-on items of people outside the car. For example, in Figure 10, the person outside the vehicle is a building security guard, holding a flashlight, passing by at night to check for abnormally parked vehicles in the parking space of the building. Specifically include the following steps:

(B1) When the vehicle is in a temporarily parked state, image recognition is performed on the panoramic image through machine learning, so as to identify the identity features of persons outside the vehicle. Specifically include the following steps:

(B1-1) Perform still image extraction on the panoramic image.

(B1-2) Input the static picture into a pre-trained person identification model to perform image recognition, and the factors of the image recognition include the clothing of the person and the identification items carried by the person.

(B2) According to the identity feature, infer the intention of the person outside the vehicle to approach the vehicle.

(B3) In response to determining the intention of the person outside the vehicle to approach the vehicle, sending prompt information to the owner of the vehicle for indicating the intention of the person outside the vehicle to approach the vehicle and/or recommending countermeasures.

On the other hand, persons outside the vehicle are persons who are about to enter the vehicle and persons who get out of the vehicle when the vehicle is parked. Please refer to FIG. 11 , which is a schematic diagram of monitoring children and carried items in an embodiment of the image processing method of the present invention. As shown in Figure 11, it is recognized through the panoramic image that the car owner carries a document when entering the car and is accompanied by children, and only the car owner leaves when leaving the car. Combined with the time of vehicle ignition data and flameout data, extract the image when the car owner enters the car and the image when the car leaves the car in the panoramic image at the corresponding time, and compare the images, so as to judge whether the car owner forgot to bring documents or children when he left the car to the car, so as to send a reminder to the car owner, so that the car owner can flexibly process the reminder information and make a judgment as to whether it is true that the item or the child has left in a short time, and return immediately. In this embodiment, the intelligent monitoring of the panoramic image is adopted. With image recognition, a forgetting reminder is sent to the user, thereby greatly reducing the possibility that the owner of the vehicle must return the forgotten items to the vehicle and take it out again, causing delays in important matters, or reducing the risk of danger caused by the owner forgetting children in the car. possibility.

The protection scope of the image processing method of the present invention is not limited to the execution order of the steps listed in this embodiment, and all the solutions implemented by adding and subtracting steps and replacing steps in the prior art based on the principles of the present invention are included in the present invention. within the scope of protection.

This embodiment provides a non-transitory computer-readable storage medium on which machine-executable instructions are stored, the machine-executable instructions, when executed, cause a machine to execute the image processing method. The image processing method includes: acquiring a panoramic image around the vehicle and current driving information of the vehicle; the panoramic image includes angle information corresponding to its presentation content; acquiring an image display requirement matching the current driving information of the vehicle; the The image display requirement includes a display angle; based on the angle information consistent with the display angle, extract the panoramic image part corresponding to the angle information, so as to present the image display device in the in-vehicle display device in combination with the change of the image display requirement Panoramic image section.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by hardware related to computer programs. The aforementioned computer program may be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the aforementioned computer readable storage medium includes: ROM, RAM, magnetic disk or optical disk and other computer storage media that can store program codes.

Please refer to FIG. 12 , which is a schematic diagram of the structure and connection of the electronic device of the present invention in an embodiment. As shown in FIG. 12, the electronic device 1 of the present invention includes: a processor 11 and a memory 12; the memory 12 is configured to store one or more computer programs; the processor 11 is coupled to the memory 12 and It is configured to execute the one or more computer programs to cause the electronic device 1 to perform the image processing methods shown in FIGS. 1-5 , 6A and 7 . The electronic device includes, but is not limited to, a tablet computer, a smart phone, a Personal Digital Assistant (PDA for short), a car terminal or a smart glasses, a smart watch or other wearable devices that can obtain panoramic images through a network connection.

The above-mentioned processor 11 can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (Digital Signal Processing, referred to as DSP) ), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. The processor 11 is used to execute the computer program of the image processing method shown in FIG. 1-FIG. 5, FIG. 6A, and FIG. 7, and the image processing method includes: acquiring panoramic images around the vehicle and current driving information of the vehicle; The panoramic image includes angle information corresponding to its presentation content; obtains image display requirements that match the current driving information of the vehicle; the image display requirements include a display angle; based on the angle information consistent with the display angle, extracting The panoramic image portion corresponding to the angle information is used to present the panoramic image portion in the in-vehicle display device in combination with the change of the image display requirements.

The above-mentioned memory 12 may include random access memory (Random Access Memory, RAM for short), and may also include non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 12 is used to store the computer program of the image processing method shown in FIG. 1-FIG. 5, FIG. 6A, and FIG. 7. The image processing method includes: acquiring panoramic images around the vehicle and current driving information of the vehicle; The panoramic image includes angle information corresponding to its presentation content; obtains an image display requirement that matches the current driving information of the vehicle; the image display requirement includes a display angle; based on the angle information consistent with the display angle, extracting and The panoramic image portion corresponding to the angle information is used to present the panoramic image portion in the in-vehicle display device in combination with changes in the image display requirements.

In one embodiment, the electronic device is a car machine, which consists of a microprocessor (MCU), a memory (ROM, RAM), an input/output interface (I/O), an analog-to-digital converter (A/D), and a plasticizer. , driver and other large-scale integrated circuits. Wherein, the microprocessor (MCU) is used for executing the computer program of the image processing method, and the memory (ROM, RAM) is used for storing the computer program of the image processing method. The image processing method includes: acquiring a panoramic image around the vehicle and current driving information of the vehicle; the panoramic image includes angle information corresponding to its presentation content; acquiring an image display requirement matching the current driving information of the vehicle; the The image display requirement includes a display angle; based on the angle information consistent with the display angle, extract the panoramic image part corresponding to the angle information, so as to present the image display device in the in-vehicle display device in combination with the change of the image display requirement Panoramic image section.

Please refer to FIG. 13 , which is a schematic structural diagram of an image processing system according to an embodiment of the present invention. As shown in FIG. 13 , the image processing system of the present invention includes: an electronic device 1 and a panoramic imaging device 2 ; the panoramic imaging device 2 is connected to the electronic device 1 .

The panoramic image device 2 is used to generate a panoramic image and send it to the electronic device 1 .

The electronic device 1 is used to obtain a panoramic image around the vehicle and current driving information of the vehicle; the panoramic image includes angle information corresponding to its presentation content; obtains image display requirements that match the current driving information of the vehicle; the The image display requirement includes a display angle; based on the angle information consistent with the display angle, extract the panoramic image part corresponding to the angle information, so as to present the image display device in the in-vehicle display device in combination with the change of the image display requirement Panoramic image section.

Please refer to FIG. 14 , which is a schematic diagram of an application of the image processing system of the present invention in an embodiment. As shown in FIG. 14 , in practical applications, the panoramic imaging device is a panoramic camera, and the electronic device is a vehicle end. The panoramic camera can be installed on the top of the vehicle before the vehicle leaves the factory, or the panoramic camera can be added in the later modification process. Preferably, the panoramic camera is installed in the middle of the roof of the vehicle, and is connected to the Tbox controller of the vehicle through a USB data cable, so as to transmit the obtained panoramic image to the vehicle for image processing. Specifically, the panoramic image is uniformly calculated, processed, and output to the central control screen through the on-board controller (ie, the Tbox controller) at the vehicle end. Further, the obtained panoramic images can be stored in a storage device (for example: U disk) associated with the data transmission of the Tbox controller, for the owner or the relatives and friends of the owner who has the authority and the personnel who need to view it in the follow-up. View the driving panorama video recording video. The Tbox controller can also upload the imaged panoramic images to the cloud server for storage in the cloud through the 4G-SIM card, so that the uploaded panoramic images can be stored and backed up, systematically managed, and distributed to the electronic devices to be displayed according to the viewing requirements. corresponding panoramic image.

The image processing system of the present invention can implement the image processing method of the present invention, but the implementation devices of the image processing method of the present invention include but are not limited to the structures of the image processing system listed in this embodiment. The structural deformation and replacement of the prior art made by the principle of the invention are all included in the protection scope of the present invention.

To sum up, the image processing method, medium, device and image processing system of the present invention obtain a panoramic image through a panoramic lens, which can be dynamically imaged at 720° on a display device, and can display images from various angles of driving. By combining the steering wheel angle data of the vehicle, the image that can be automatically converted to the image while driving with the vehicle can be realized. By providing a 720° panoramic image record of the whole vehicle when driving or parking, it can feed back prompt information to the owner of the abnormal situation in the monitored panoramic image, and can also provide picture evidence in the event of an accident, violation of regulations, or vehicle scratching. A cloud account can be used to log in to the application interface associated with the panoramic lens, so that users with permissions can view the surrounding of the vehicle from anywhere. The invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

The above-mentioned embodiments merely illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Anyone skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed in the present invention should still be covered by the claims of the present invention.

Claims (15)

1. An image processing method, characterized in that the method comprises the following steps:

   Obtain a panoramic image around the vehicle and current driving information of the vehicle, where the panoramic image includes angle information corresponding to its presentation content;

   acquiring an image display requirement matching the current driving information of the vehicle, where the image display requirement includes a display angle; and

   Based on the angle information consistent with the display angle, the panoramic image part corresponding to the angle information is extracted, so that the panoramic image part is presented in the in-vehicle display device in combination with the change of the image display requirements.
2. The method according to claim 1, wherein the current driving information of the vehicle includes at least current steering wheel angle information, travel speed information and gear information of the vehicle.
3. The method according to claim 2, wherein the image display requirement further includes at least a refresh direction and a refresh frequency; the vehicle determines the current steering wheel angle information, gear speed, etc. of the vehicle through preset sensors and/or navigation positioning data. position information and driving speed information; the acquisition of the image display requirements matching the current driving information of the vehicle includes the following steps:

   Calculate the display angle of the panoramic image based on the sensor and/or the navigation and positioning data;

   Detecting the forward and backward state of the vehicle based on the gear information, and then determining the refresh direction of the panoramic image based on the forward and backward state; and

   The refresh rate of the panoramic image is determined based on the travel speed information.
4. The method according to claim 3, wherein the gear information includes D gear and R gear, the steering wheel angle information includes a left turn angle and a right turn angle; and the display of the panoramic image is calculated based on the steering wheel angle information Angle includes at least one of the following steps:

   In response to determining that the steering wheel angle information is the D gear and the left turn angle, subtract the left turn angle from the first preset angle as the display angle;

   In response to determining that the steering wheel angle information is the D gear and the right turn angle, using the right turn angle as the display angle;

   In response to determining that the steering wheel angle information is the R gear and the left turn angle, adding the second preset angle to the left turn angle as the display angle; and

   In response to determining that the steering wheel angle information is the R gear and the right turn angle, the second preset angle is subtracted from the right turn angle as the display angle.
5. The method according to claim 1, wherein, based on the angle information consistent with the display angle, extracting the panoramic image part corresponding to the angle information comprises the following steps:

   establishing an angle coordinate system based on the angle information, so as to superimpose the angle coordinate system in the panoramic image;

   positioning the display angle in the angular coordinate system; and

   Extract the picture to be displayed with the display angle at the location.
6. The method according to claim 5, wherein the establishing an angle coordinate system based on the angle information comprises the following steps:

   converting the panoramic image into a spherical image;

   Using the horizontal line as the center mark in the spherical image, determining a center horizontal image; and

   The angular coordinate system is established for the center horizontal image.
7. The method of claim 1, wherein the method further comprises the steps of:

   Based on changes in the image display requirements, continuously determining the pictures to be displayed at different times; and

   A plurality of the pictures to be displayed are displayed in chronological order, so as to perform corresponding display of different parts of the panoramic image based on the real-time changes of the image display requirements.
8. The method of claim 1, wherein the method further comprises the step of: presenting the panoramic image via an electronic device.
9. The method according to any one of claims 1-8, wherein the method further comprises the steps of:

   When the vehicle is in a parked state, image recognition is performed on the panoramic image to recognize the passing situation outside the vehicle;

   Perform voice recognition on the panoramic image to recognize the voice content of people outside the vehicle;

   Based on the meeting situation and the voice content, determining whether the car needs to be moved; and

   In response to determining that a manoeuvring is required, a manoeuvring alert is sent to the owner of the vehicle.
10. The method according to claim 9, wherein the performing image recognition on the panoramic image to recognize the passing situation outside the vehicle comprises the following steps:

    Identifying the other person's vehicle in the panoramic image, and calculating the distance between the other person's vehicle and the vehicle based on the rendering ratio of the panoramic image and the real scene;

    comparing the vehicle distance to a preset distance threshold to assess the likelihood that the other person's vehicle will pass normally; and

    In response to determining that the vehicle distance is less than the preset vehicle distance threshold, it is determined that the meeting situation outside the vehicle is congestion.
11. The method according to any one of claims 1-8, wherein the method further comprises the steps of:

    When the vehicle is in a parked state, image recognition is performed on the panoramic image through machine learning to identify the identity features of persons outside the vehicle;

    Based on the identity feature, infer the intention of the person outside the vehicle to approach the vehicle; and

    In response to determining the intention of the person outside the vehicle to approach the vehicle, prompt information is sent to the owner of the vehicle for indicating the intention of the person outside the vehicle to approach the vehicle and/or recommending countermeasures.
12. The method according to claim 11, wherein the performing image recognition on the panoramic image through machine learning to identify the identity features of persons outside the vehicle comprises the following steps:

    performing still picture extraction on the panoramic image; and

    The static picture is input into a pre-trained person identification model for image recognition, and the factors of the image recognition include the clothing of the person and the identification items carried by the person.
13. A non-transitory computer-readable storage medium having stored thereon machine-executable instructions that, when executed, cause a machine to perform the steps of the method of any of claims 1-12.
14. An electronic device comprising: a memory configured to store one or more computer programs; and a processor coupled to the memory and configured to execute the one or more computer programs to cause the electronic device to perform a claim The method of any one of claims 1-12.
15. An image processing system, characterized in that the image processing system comprises: a panoramic imaging device and the electronic device according to claim 14; the panoramic imaging device is connected to the electronic device;

    The panoramic image device is used to generate a panoramic image and send it to the electronic device;

    The electronic device is used to obtain a panoramic image around the vehicle and current driving information of the vehicle; the panoramic image includes angle information corresponding to its presentation content; obtains image display requirements matching the current driving information of the vehicle; the image The display requirement includes a display angle; based on the angle information consistent with the display angle, extracting the panoramic image part corresponding to the angle information, so as to present the panoramic image in the in-vehicle display device in combination with the change of the image display requirement video part.

Images