WO2021161614A1

WO2021161614A1 - Image transmission system

Info

Publication number: WO2021161614A1
Application number: PCT/JP2020/043733
Authority: WO
Inventors: 直浩平岩
Original assignee: アイシン・エィ・ダブリュ株式会社
Priority date: 2020-02-13
Filing date: 2020-11-25
Publication date: 2021-08-19
Also published as: JP7392506B2; JP2021128532A

Abstract

[Problem] To provide a technology for reducing the possibility that the amount of information of an unimportant object becomes excessive. [Solution] An image transmission system is configured to comprise: an image capturing unit which is mounted on a vehicle and captures an image of the road on which the vehicle is traveling; an information reduction processing unit which, for important objects in a specific state among objects present in the image, performs an information reduction process on the image such that the amount of information reduction per unit area of the important objects is smaller than the amount of information reduction per unit area of unimportant objects other than the important objects; and an image transmission unit which transmits the image after an information reduction process to an external device.

Description

Image transmission system

The present invention relates to an image transmission system.

Conventionally, a technique for compressing an image taken in a vehicle is known. For example, Patent Document 1 discloses a technique for compressing a designated portion of captured image data, for example, a license plate, a traffic light, or a wall of a traveling vehicle at a compression rate lower than that of other portions.

Japanese Unexamined Patent Publication No. 2009-175848

In the conventional technique, when the license plate is specified, the portion of the license plate becomes low compression. However, uniform low compression of specific objects often did not reduce the amount of information about non-essential objects. For example, when transmitting an image of a traffic jam scene for analyzing a congested convoy, if the image of the vehicle is uniformly low-compressed, the amount of information of the image of the vehicle not related to the traffic jam becomes excessive.
The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for reducing the possibility that the amount of information of an unimportant object becomes excessive.

In order to achieve the above object, the image transmission system is mounted on a vehicle and is important to be in a specific state among an image capturing unit that captures an image of the road on which the vehicle is traveling and an object existing in the image. Information reduction processing that performs information reduction processing on the image so that the amount of information reduction per unit area of the important object is smaller than the amount of information reduction per unit area of non-important objects other than the important object. A unit and an image transmission unit that transmits the image after information reduction processing to an external device are provided.

That is, in the image transmission system, important objects are determined in advance. Therefore, in the image transmission system, when an important object is included in the captured image, the information reduction process is performed so that the information reduction amount of the important object is smaller than the information reduction amount of the other non-important objects. As a result, the possibility that the amount of information of important objects is insufficient is reduced, and the possibility that the amount of information of non-important objects is excessive is reduced. Therefore, it is possible to transmit an image in which the amount of information of the important object is likely to be sufficient, and the image of the important object can be used in a sufficient amount of information at the destination of the image.

A block diagram showing the configuration of an image transmission system. Flowchart of image transmission processing. FIG. 3A is a diagram showing an example of a captured image, and FIG. 3B is a diagram showing a plane including a vehicle width axis and a vehicle length axis. FIG. 4A is a scene determination process for a specific lane congestion scene, and FIGS. 4B to 4D are diagrams for explaining a specific lane congestion scene. FIG. 5A is a scene determination process for an obstacle existence scene, and FIG. 5B is a diagram for explaining an obstacle existence scene. FIG. 6A is a scene determination process for a map non-recording scene, and FIG. 6B is a diagram for explaining a map non-recording scene.

Here, embodiments of the present invention will be described in the following order.
(1) Image transmission system configuration:
(2) Image transmission processing:
(2-1) Scene judgment processing (specific lane congestion scene):
(2-2) Scene judgment processing (obstacle existence scene):
(2-3) Scene judgment processing (map non-recording scene):
(3) Other embodiments:

(1) Image transmission system configuration:
FIG. 1 is a block diagram showing a configuration of a navigation system 10 including an image transmission system according to an embodiment of the present invention. The navigation system 10 is provided in the vehicle, and includes a control unit 20 including a CPU, RAM, ROM, and the like, and a recording medium 30. In the navigation system 10, the control unit 20 can execute a program stored in the recording medium 30 or the ROM. Map information 30a is recorded in advance on the recording medium 30.

The map information 30a is information used for specifying the position of an intersection, route guidance, etc., and specifies node data indicating the position of a node set on the road on which the vehicle travels, and the shape of the road between the nodes. It includes shape interpolation point data indicating the position of the shape interpolation point, link data indicating the connection between nodes, and feature data indicating the position and shape of features existing on the road and its surroundings. In this embodiment, the nodes indicate intersections.

In addition, the link data is associated with information indicating the number of lanes existing in the road section indicated by the link data and the width of the lane. In the present embodiment, the position indicated by the node or the shape interpolation point indicates the position of the center line on the road section, and the position of the lane and the range in which the lane exists can be specified by the position, the number of lanes, and the width of the lane. be. Feature data is data indicating the existence of various features. In this embodiment, the feature includes at least a traffic sign. That is, the feature data indicating the traffic sign includes the position of the traffic sign and the identification information of the traffic sign. The position of the traffic sign may be indicated by coordinates (latitude, longitude, etc.), or may be indicated as a position on a road section indicated by a link, for example, by a distance from a node or the like.

The vehicle in this embodiment includes a camera 40, a GNSS receiving unit 41, a vehicle speed sensor 42, a gyro sensor 43, a user I / F unit 44, and a communication unit 45. The GNSS receiving unit 41 is a device that receives signals from the Global Navigation Satellite System, receives radio waves from navigation satellites, and outputs a signal for calculating the current position of the vehicle via an interface (not shown). The control unit 20 acquires this signal and acquires the current position (latitude, longitude, etc.) of the vehicle in the coordinate system of the map. The vehicle speed sensor 42 outputs a signal corresponding to the rotational speed of the wheels provided in the vehicle. The control unit 20 acquires this signal via an interface (not shown) to acquire the vehicle speed. The gyro sensor 43 detects the angular acceleration for turning in the horizontal plane of the vehicle and outputs a signal corresponding to the direction of the vehicle. The control unit 20 acquires this signal and acquires the traveling direction of the vehicle. The vehicle speed sensor 42, the gyro sensor 43, and the like are used to identify the traveling locus of the vehicle, and in the present embodiment, the current position is specified based on the starting point and the traveling locus of the vehicle, and the departure point and the traveling are The current position of the vehicle specified based on the trajectory is corrected based on the output signal of the GNSS receiving unit 41.

The camera 40 is a device that acquires an image in the field of view directed to the front of the vehicle. The optical axis of the camera 40 is fixed to the vehicle, and the direction of the optical axis may be known in the navigation system 10. In the present embodiment, the camera 40 is attached to the vehicle in a posture in which the vehicle width direction and the center of the optical axis are perpendicular to each other and the front of the vehicle in the traveling direction is included in the field of view. The control unit 20 can detect other vehicles (peripheral vehicles) existing in the vicinity of the vehicle by acquiring the image output by the camera 40 and analyzing the image by extracting the feature amount or the like.

The user I / F unit 44 is an interface unit for inputting user instructions and providing various information to the user, and includes a touch panel type display, a speaker, and the like (not shown). That is, the user I / F unit 44 includes an image and sound output unit and an input unit for instructions by the user. The communication unit 45 is a device for communicating with an external device. In the present embodiment, the control unit 20 communicates with the server 50 via the communication unit 45.

The control unit 20 receives the input of the destination by the user via the input unit of the user I / F unit 44 (not shown) by the function of the navigation program (not shown), and from the current position of the vehicle to the destination based on the map information 30a. Search for the planned travel route. Further, the control unit 20 controls the user I / F unit 44 by the function of the navigation program, and executes guidance for traveling along the planned travel route. In the present embodiment, the control unit 20 can execute a function of transmitting an image taken by the camera 40 to the server 50 as an additional function of the navigation program.

The navigation program includes an image transmission program 21 for transmitting the image. The image transmission program 21 includes an image capturing unit 21a, a scene determination unit 21b, an information reduction processing unit 21c, and an image transmission unit 21d in order to realize an image transmission function.

The image capturing unit 21a is a program module mounted on the vehicle and causing the control unit 20 to execute a function of capturing an image of the road on which the vehicle is traveling. In the present embodiment, the control unit 20 controls the camera 40 in the process of traveling the vehicle, and photographs the landscape including the road in front of the vehicle. The image output from the camera 40 by shooting is recorded on the recording medium 30 as image information 30b. In the present embodiment, the control unit 20 acquires the image information 30b at regular intervals while the vehicle is running, but of course, the image is taken under various conditions, for example, a specific road type or a road section. It may be.

The scene determination unit 21b is a program module that causes the control unit 20 to execute a function of determining which of the predetermined scenes the image captured by the function of the image capture unit 21a is. In the present embodiment, the server 50 acquires an image, and the server 50 performs various analyzes based on the image. For this reason, a predetermined characteristic image is targeted for analysis, and the characteristic image is defined in advance as a scene to be analyzed.

The scene may be an analysis target and is not limited, but in the present embodiment, the explanation will be made mainly assuming that three types of scenes are the analysis target. One of the three types of scenes is a scene in which a specific lane on the road is congested and there is no congestion in a lane other than the specific lane (referred to as a specific lane congestion scene). The remaining two scenes are a scene in which an obstacle exists on the road on which the vehicle travels (called an obstacle existence scene) and a scene in which the presence of a traffic sign included in the image is not shown in the map information (map not shown). It is called a recording scene).

The control unit 20 recognizes peripheral vehicles based on the image information 30b in order to specify whether or not the scene is a specific lane congestion scene. Further, the control unit 20 specifies the distance and direction of the peripheral vehicle as seen from the vehicle based on the image information 30b, and specifies the relative position of the peripheral vehicle as seen from the vehicle. Further, the control unit 20 recognizes a white line on the road based on the image information 30b, and identifies the lane in which each peripheral vehicle exists. Then, the control unit 20 determines that there is a traffic jam in a specific lane when peripheral vehicles existing in the specific lane form a convoy and the vehicle speed is equal to or less than a predetermined speed (for example, 20 km / h). Identify. When there is another lane in which there is no traffic jam on the road on which the vehicle travels, the control unit 20 determines that the image indicated by the image information 30b is a specific lane traffic jam scene.

In order to identify whether or not the scene has an obstacle, the control unit 20 recognizes the area occupied by the image of the road on which the vehicle travels based on the image information 30b. Further, the control unit 20 detects an area surrounded by the image of the road, and if there is a stationary area having a predetermined size or more, determines that an obstacle on the road is photographed in the area. .. In this case, the control unit 20 determines that the image indicated by the image information 30b is an obstacle presence scene.

The control unit 20 recognizes a traffic sign based on the image information 30b in order to identify whether or not the scene is a map non-recording scene. Further, the control unit 20 identifies the current location of the vehicle when the image information 30b is captured, and searches for a traffic sign existing within a predetermined range from the current location with reference to the map information 30a. Then, when the recognized traffic sign is not included in the map information 30a, the control unit 20 determines that the image indicated by the image information 30b is a map non-recording scene.

When the scene is specified based on the image information 30b, the control unit 20 associates the scene identification information with the image information 30b. The identification information may be any information indicating a scene, and examples thereof include an ID and the like. It may be determined that the same image information 30b corresponds to two or more scenes.

When the object existing in the image corresponds to the important object in a specific state, the information reduction processing unit 21c determines that the amount of information reduction per unit area of the important object is the information per unit area of the non-important object other than the important object. This is a program module that causes the control unit 20 to execute a function of performing information reduction processing on an image so as to be less than the reduction amount. That is, the control unit 20 performs image processing to reduce the amount of information in order to reduce the amount of communication when transmitting the image information 30b to the server 50 as much as possible by the function of the information reduction processing unit 21c.

The image information 30b obtained by being photographed by the camera 40 becomes difficult to analyze because the object becomes difficult to identify and the detailed structure of the object becomes difficult to understand as the amount of information is reduced. Therefore, in the present embodiment, the amount of information reduction in the important part of the image is relatively small, and the amount of information reduction in the non-important part is relatively large.

Since the important object is determined for each scene in the present embodiment, the control unit 20 extracts the important object in which the specific object is in the specific state for each scene of the image indicated by the image information 30b. If the scene in the image is a specific lane congestion scene, the important object is a vehicle that is in a specific lane. Therefore, when the scene of the image is a specific lane congestion scene, the control unit 20 identifies an image of a vehicle existing in a specific lane among the surrounding vehicles as an important object, and positions the important object in the image. To identify. In this way, if a vehicle existing in a specific lane is regarded as an important object, it is possible to perform an analysis focusing on a vehicle in a specific lane that is congested at the image transmission destination.

When the scene in the image is an obstacle presence scene, the important object is an obstacle on the road on which the vehicle travels. Therefore, when the scene of the image is a scene with an obstacle, the control unit 20 specifies the image of the portion determined to be an obstacle as an important object when the scene is specified. In addition, the control unit 20 specifies the position of the important object in the image. In this way, if an obstacle on the road on which the vehicle travels is regarded as an important object, it is possible to perform analysis focusing on the obstacle at the image transmission destination.

When the image scene is a map non-recording scene, the important object is a traffic sign that is not recorded in the map information 30a. Therefore, when the scene of the image is a map non-recording scene, the control unit 20 identifies the image of the traffic sign recognized when specifying the scene as an important object. In addition, the control unit 20 specifies the position of the important object in the image. As described above, if a traffic sign not recorded in the map information 30a is regarded as an important object, the image transmission destination should analyze the traffic sign not recorded in the map information 30a and add it to the map information 30a. Information indicating a traffic sign can be generated.

When the position of the important object is specified, the control unit 20 performs information reduction processing so that the important object has a smaller amount of information reduction than the non-important object. Information reduction may be carried out by various methods, but in the present embodiment, it is carried out by compression processing. The compression process may be various methods, and in the present embodiment, compression is performed in the JPEG format. That is, the control unit 20 adjusts the compression ratio of the rectangular region including the important object so that the compression ratio is lower than that of the rectangular region not including the important object, and JPEG compresses the image information 30b. The rectangle may be specified by various methods. For example, a rectangle having a plurality of sizes is defined in advance, and a rectangle having the smallest size including each important object is selected. Further, there is a configuration in which rectangles having a default size are defined in advance and the minimum number of rectangles including important objects are selected. Further, there is a configuration in which the image in the bounding box, which will be described later, has a low compression rate, and the image not included in the bounding box has a high compression rate.

According to the above processing, the amount of information reduced for the important part of the image is suppressed, while the amount of information is further reduced for the non-important part. Therefore, when the analysis or the like is performed based on the image information 30b after the information reduction, the possibility that the amount of information of the important object is insufficient is reduced.

The image transmission unit 21d is a program module that causes the control unit 20 to execute a function of transmitting an image after information reduction processing to an external device. That is, the control unit 20 controls the communication unit 45 by the function of the image transmission unit 21d, and transmits the compressed image information 30b to the server 50. The image information 30b is associated with scene identification information. Therefore, the server 50 can specify which scene the image information 30b is based on the identification information.

In the present embodiment, since the important object is determined for each scene, it can be considered that the scene identification information indicates the important object included in the image. Therefore, the scene identification information is information about important objects. Further, in the present embodiment, as will be described later, information indicating the position of the important object in the image is associated with the image. Information indicating the position of the important object in the image can also be said to be information about the important object. Of course, various other information is assumed as the information indicating the position of the important object in the image. In any case, the image after the information reduction is associated with the scene identification information and the information about the important object, and is transmitted to the server 50. According to the above configuration, it is possible to transmit an image in which the amount of information of important objects is likely to be sufficient and the amount of information of non-important objects is unlikely to be excessive.

The server 50 includes a control unit including a CPU, ROM, RAM, etc. (not shown) and a recording medium. The control unit of the server 50 can execute various programs recorded on the recording medium to execute various processes. In the present embodiment, the control unit functions as an image receiving unit 50a and an image recording unit 50b by a program (not shown).

The image receiving unit 50a causes the control unit to execute a function of receiving an image. That is, when an image is transmitted from the navigation system 10, the control unit receives the image via a communication unit (not shown) by the function of the image receiving unit 50a. The image recording unit 50b causes the control unit to execute a function of associating an image with information about an important object in the image and recording the information on a recording medium. That is, when the image is received, the control unit saves the image in the recording medium by the function of the image recording unit 50b.

The image is associated with scene identification information and information about the important object indicating the position of the important object in the image. Therefore, the server 50 records the information about the important object included in the image on the recording medium in association with the image. When the image associated with the information about the important object is recorded on the recording medium in this way, the server 50 identifies the important object, the type of the scene, etc. in the image based on the information about the important object, and is important. It becomes possible to perform image analysis and the like related to objects.

(2) Image transmission processing:
Next, the image transmission process executed by the control unit 20 will be described with reference to FIG. When the control unit 20 takes a picture with the camera 40 by the function of the image taking unit 21a, the control unit 20 executes the image transmission process shown in FIG. In the image transmission process, the control unit 20 executes the object recognition process by the function of the scene determination unit 21b (step S100). Specifically, the control unit 20 acquires the image information 30b and performs distortion correction or the like by the lens. Further, the control unit 20 executes image recognition processing for surrounding vehicles and traffic signs by using YOLO (You Only Look Once), pattern matching, and the like. As a result, the control unit 20 detects the image of the surrounding vehicle and the image of the traffic sign included in the image information 30b.

In the present embodiment, when the control unit 20 recognizes the recognized peripheral vehicle or traffic sign, the control unit 20 identifies the bounding box surrounding these objects in the image. The size and position of the bounding box indicate the size of the object's image and the position of the object in the image. FIG. 3A is a diagram showing an example of an image I taken by the camera 40 and after distortion correction has been performed. This example is an example in which a peripheral vehicle is recognized by the object recognition process. As shown in FIG. 3A, in the present embodiment, the bounding box B is a rectangular area surrounding the peripheral vehicles detected from the image I.

The size and position of the bounding box B are represented by, for example, the coordinates of the upper left vertex and the coordinates of the lower right vertex of the bounding box B. The control unit 20 acquires the height h (number of pixels) of the bounding box B and the representative coordinates Bo (x, y) of the bounding box B from the coordinates of the two diagonal vertices of the bounding box B. The representative coordinates Bo are, for example, the center coordinates of the bounding box B (midpoints in the width direction and the height direction) and the like. The control unit 20 specifies the relative orientation of the surrounding vehicles as seen from the vehicle based on the position of the representative coordinate Bo of the bounding box B. Further, the control unit 20 specifies the distance from the vehicle to the peripheral vehicle based on the height h of the bounding box B and the type of the peripheral vehicle.

Specifically, each coordinate in the image I is associated with the relative orientation of the object reflected in the coordinates when the vehicle is used as a reference, and information indicating the correspondence is stored in the recording medium 30. There is. Based on this correspondence, the control unit 20 acquires the relative orientation of the surrounding vehicles reflected in the representative coordinates Bo. In the present embodiment, the control unit 20 defines a vehicle coordinate system based on the vehicle. The vehicle coordinate system is a coordinate system defined by a vehicle width axis (X axis shown in FIG. 3B) and a vehicle length axis (Y axis shown in FIG. 3B) that are orthogonal to each other.

FIG. 3B shows a plane including the vehicle width axis and the vehicle length axis. In the figure, the point O is the origin of the vehicle coordinate system in the vehicle. In the example of FIG. 3B, the vehicle length axis is parallel to the link indicating the road section on which the vehicle is traveling. The relative orientation is represented by, for example, the angle (θ) formed by the straight line SL connecting the origin O of the vehicle coordinate system and the point corresponding to the representative coordinate Bo and the vehicle length axis (for example, when θ is a negative value, the vehicle travels. It indicates that it is on the left side of the vehicle length axis when facing forward in the direction, and on the right side if it is a positive value).

Further, the control unit 20 identifies the type of peripheral vehicle in the bounding box B by the object recognition process. The type of the peripheral vehicle may be any type indicating the size of the vehicle body, and may be classified into, for example, a freight vehicle, a passenger car, a two-wheeled vehicle, and the like. Further, in the present embodiment, a typical vehicle height (for example, 1.5 [m] in the case of a passenger car) is specified for each type of peripheral vehicle. Further, the linear distance between the vehicle and the peripheral vehicle and the height h of the bounding box B when the peripheral vehicle is photographed by the camera 40 are measured in advance. Information indicating the correspondence between the height h of the bounding box B and the linear distance with respect to the origin of the vehicle coordinate system is stored in the recording medium 30 for each type of vehicle.

For example, if the height of the bounding box surrounding a passenger car having a typical actual vehicle height of 1.5 [m] is h1 pixels, the linear distance is D1 [m], and if it is h2 pixels, the linear distance is D2. It is associated with [m]. Information indicating the correspondence between other types such as freight vehicles and two-wheeled vehicles is stored in the recording medium 30. The control unit 20 calculates a linear distance D (see FIG. 3B) corresponding to the height h of the bounding box B based on this correspondence. As described above, the control unit 20 acquires the relative orientation θ of the surrounding vehicles included in the image and the linear distance D from the vehicle based on the image taken by the camera 40.

In the present embodiment, images are taken every shooting cycle by the camera 40, and the processing after step S100 is performed for each image. Therefore, the same peripheral vehicle can be recognized over the shooting process of several frames. Therefore, in the present embodiment, the control unit 20 gives the same identification information to the peripheral vehicle while the same peripheral vehicle is being photographed. Therefore, the control unit 20 identifies the characteristics (for example, colors, patterns in the bounding box B, etc.) of the images of the peripheral vehicles for which the relative orientation θ and the linear distance D are specified, and the identification information corresponding to the characteristics. (For example, a number or the like) is associated with information indicating a relative direction θ, a straight line distance D, and a type of a peripheral vehicle, and recorded on the recording medium 30.

In order to give the same identification information to the same peripheral vehicle each time an image is taken, the control unit 20 refers to the recording medium 30 and corresponds to the peripheral vehicle recognized by the immediately preceding image and the latest image. It is determined whether or not it matches the characteristics of the attached image. If they match, the control unit 20 also gives the identification information given to the peripheral vehicle in the immediately preceding image to the peripheral vehicle recognized in the latest image. As a result, the same identification information is given to the peripheral vehicles that are continuously photographed by the camera 40.

In step S100, in addition to the recognition of surrounding vehicles as described above, the recognition of traffic signs is also performed. In the present embodiment, when the control unit 20 recognizes a traffic sign based on the characteristics of each traffic sign, a bounding box indicating the traffic sign is specified. FIG. 6B shows an example of image information 30b including a traffic sign. When the image information 30b includes a traffic sign, the control unit 20 recognizes the traffic sign and identifies a bounding box B indicating the presence of the traffic sign and its position in the image. Then, the control unit 20 associates the identification information of the traffic sign with the information indicating the position of the bounding box B and records it on the recording medium 30.

Next, the control unit 20 executes the white line recognition process by the function of the scene determination unit 21b (step S105). The white line recognition process may be carried out by various methods. For example, when the control unit 20 executes a straight line detection process using a Hough transform or the like, the color of the area sandwiched by the detected straight lines is white, and the width of the white area is within a predetermined distance, it is set as a white line. Examples include recognition processing. In the present embodiment, since it is assumed that a white line existing at the end in the width direction of the lane and extending in the vehicle traveling direction (white solid line indicating the lane boundary line, white broken line, etc.) is recognized, the white line disappears. Conditions such as extending toward a point may be added.

Next, the control unit 20 executes the area recognition process by the function of the scene determination unit 21b (step S110). In the present embodiment, the area recognition process is a process of recognizing a continuous area in the image indicated by the image information 30b. That is, among the objects in the image, there is a high possibility that the road surface, the sky, etc. form a substantially uniform continuous region.

Therefore, based on the image information 30b, the control unit 20 specifies a region in which pixels in which a color change (for example, a color difference specified by brightness or saturation) is within a predetermined range are continuous. Of course, as a condition for area recognition, various other conditions are assumed, such as an area in which pixels within a specific color range (for example, a preset range as a road surface color) are continuous. May be specified. When a continuous region is specified, the control unit 20 specifies a continuous region below the image including the lower end of the image as an image region of the road surface on which the vehicle travels. Then, the information indicating the area of the image of the road surface is recorded on the recording medium 30.

Next, the control unit 20 determines whether or not any of the object, the white line, and the area is recognized by the function of the scene determination unit 21b (step S120). That is, if the image taken by the camera 40 does not include surrounding vehicles and traffic signs, the control unit 20 does not determine that the object has been recognized. Further, if the image taken by the camera 40 does not include a white line existing at the end in the width direction of the lane and extending in the vehicle traveling direction, the control unit 20 does not determine that the white line has been recognized. Further, the control unit 20 does not determine that the area is recognized when the image taken by the camera 40 does not include an image of the road surface on which the vehicle travels.

If no object, white line, or area is recognized, the control unit 20 ends the image transmission process. That is, if none of the surrounding vehicle, the traffic sign, the white line which is the boundary line of the lane, and the road surface on which the vehicle travels is recognized in the image taken by the camera 40, the image is not targeted for transmission.

On the other hand, when any of the object, the white line, and the area is recognized in step S120, the control unit 20 executes the scene determination process by the function of the scene determination unit 21b (step S125). The scene determination process is a process for determining whether the image information 30b corresponds to any of a specific lane congestion scene, an obstacle existence scene, and a map non-recording scene, or does not correspond to any of these. The details of the scene determination process will be described later. When the scene determination process is executed, the image information 30b is associated with the identification information indicating the scene.

Next, the control unit 20 determines whether or not the image information 30b includes an important object by the function of the information reduction processing unit 21c (step S130). In the present embodiment, in the scene determination process, the scene is determined based on the fact that the specific object is in a specific state. Then, the object in the specific state is an important object. Therefore, when the control unit 20 includes identification information indicating that the image information 30b is any one of a specific lane congestion scene, an obstacle existence scene, and a map non-recording scene, the control unit 20 includes an important object. judge. Of course, a process of detecting an important object from the image information 30b may be performed.

If it is not determined in step S130 that an important object is included, the control unit 20 performs uniform compression by the function of the information reduction processing unit 21c (step S140). In uniform compression, the image is compressed so that the compression ratio is common over the entire area of the image information 30b.

When it is determined in step S130 that an important object is included, the control unit 20 performs non-uniform compression by the function of the information reduction processing unit 21c (step S140), that is, the control unit 20 is in the bounding box B. The image or the image in a rectangle of a predetermined size including the bounding box B is compressed at a low compression rate. On the other hand, the control unit 20 compresses the image other than the portion having the low compression rate at a higher compression rate. The compression rate may be specified by various methods, or may be selected according to the content of the image and the communication situation.

When the compression process is performed in step S135 or step S140, the control unit 20 transmits the image information 30b to the server 50 by the function of the image transmission unit 21d (step S145). That is, the control unit 20 controls the communication unit 45 and transmits the compressed image information 30b to the server 50 in a state in which the scene identification information is associated with the image information 30b. In the present embodiment, the image information 30b includes various other information such as a specific lane in which a traffic jam is occurring, identification information corresponding to the characteristics of a peripheral vehicle, a relative orientation θ of the peripheral vehicle, and a linear distance D. Information indicating the types of surrounding vehicles, information indicating traffic signs, information indicating the position of the bounding box, information indicating the position of white lines and areas, the current location of the vehicle at the time of image capture, etc. are also transmitted in association with each other.

(2-1) Scene judgment processing (specific lane congestion scene):
FIG. 4A is a scene determination process for determining whether or not the scene of the image information 30b is a specific lane congestion scene. The scene determination process is executed when a peripheral vehicle is recognized in the object recognition process in step S100. When the scene determination process is started, the control unit 20 acquires the relative orientation of the surrounding vehicles, the linear distance, and the identification information corresponding to the feature for a certain period before the present (step S200). That is, the control unit 20 acquires the identification information corresponding to the relative direction θ, the linear distance D, and the features of each peripheral vehicle recognized in the object recognition process in step S100. Since this processing is performed every time the image information 30b is acquired, the control unit 20 is based on the result of the object recognition processing performed on the image information 30b captured within the fixed period before the present, for the fixed period. The relative azimuth θ, the linear distance D, and the identification information corresponding to the feature are acquired in chronological order.

Next, the control unit 20 acquires the current location of the vehicle (step S205). That is, the control unit 20 acquires the current location of the vehicle (the current location when the image information 30b is taken by the camera 40) based on the output signals of the GNSS receiving unit 41, the vehicle speed sensor 42, and the gyro sensor 43.

Next, the control unit 20 acquires the speed of the surrounding vehicle (step S210). That is, since the relative direction θ and the linear distance D from the vehicle are obtained for each peripheral vehicle by the process of step S100, the control unit 20 has the vehicle based on the data of the peripheral vehicles for several frames before the present. The relative moving speed (dd shown in FIG. 3B) of the surrounding vehicles as seen from the above is specified. Then, the control unit 20 identifies the current speed of the vehicle (for example, the detected value of the vehicle speed sensor 42), and based on the current speed and the relative moving speed of each peripheral vehicle, the speed of each peripheral vehicle (speed with respect to the road). ) To get.

Next, the control unit 20 identifies the lane in which the surrounding vehicle is traveling (step S215). That is, the control unit 20 refers to the map information 30a and specifies the lane configuration of the road section in which the current location of the vehicle acquired in step S205 exists. Further, the control unit 20 identifies which lane the vehicle is traveling in is the lane shown in the map information 30a based on the lane recognized in step S105. When the lane is not recognized in step S105, the control unit 20 performs processing such as assuming that the surrounding vehicle and the vehicle are traveling in the same lane.

4B to 4D are diagrams schematically showing the vehicle C and the peripheral vehicles 1 to 5 existing around the vehicle C. In this example, it is assumed that the time advances in the order of FIG. 4B, FIG. 4C, and FIG. 4D. In these examples, the number of lanes in the road section in which the vehicle is traveling is three. Therefore, in the leftmost lane, the left boundary line is a solid white line and the right boundary line is a broken line white line. In the center lane, the left and right borders are broken white lines. In the rightmost lane, the right border is the solid white line and the left border is the dashed white line.

In the example of FIGS. 4B to 4D, the control unit 20 identifies that the vehicle is traveling in the central lane based on the fact that the closest white lines existing on the left and right of the vehicle are broken lines. Then, the control unit 20 specifies the relative position of the peripheral vehicle as seen from the vehicle based on the relative orientation θ of the peripheral vehicle as seen from the vehicle and the linear distance D. Further, the control unit 20 specifies the range of each lane existing around the vehicle based on the number of lanes and the width of the lane indicated by the map information 30a, and which position of each peripheral vehicle exists at each relative position. By specifying whether it corresponds to the lane, the lane in which the surrounding vehicle is traveling is specified. Of course, the lane in which the surrounding vehicle is traveling may be identified based on the image.

By the above processing, as shown in FIGS. 4B to 4D, the current location of the vehicle C and the positions of the peripheral vehicles in the vicinity thereof are specified, and the lane in which the peripheral vehicles are traveling is also specified. Become.

Next, the control unit 20 determines whether or not there is a lane in a specific lane (step S220). That is, when there are a predetermined number or more (for example, three or more) of peripheral vehicles on the same lane, the control unit 20 considers that these peripheral vehicles form a convoy. Then, when there is a lane in which a lane is formed, the control unit 20 determines that the lane is a specific lane and determines that the lane exists in the specific lane. If it is not determined in step S220 that a lane exists in a specific lane, the control unit 20 skips steps S225 and S230.

On the other hand, when it is determined in step S220 that a lane exists in a specific lane, the control unit 20 determines whether or not the vehicle speed is equal to or less than the threshold value (step S225). That is, the control unit 20 acquires the speeds of the peripheral vehicles acquired in step S210, and whether or not the speeds (for example, average speeds) of the peripheral vehicles forming the convoy are equal to or less than a predetermined threshold value. To judge. The threshold value is a small speed (for example, 20 km / h or the like) for determining whether or not there is a traffic jam.

If it is not determined in step S225 that the vehicle speed is equal to or less than the threshold value, the control unit 20 skips step S230. When it is determined in step S225 that the vehicle speed is equal to or less than the threshold value, the control unit 20 determines that the image information 30b is a specific lane congestion scene (step S230). That is, the control unit 20 associates the image information 30b with information indicating that the scene is a specific lane congestion scene.

(2-2) Scene judgment processing (obstacle existence scene):
FIG. 5A is a scene determination process for determining whether or not the scene of the image information 30b is an obstacle existence scene. When the scene determination process is executed, the control unit 20 acquires an image area of the road surface (step S300). That is, the control unit 20 acquires an image region of the road surface from the regions recognized as continuous by the region recognition process in step S110. FIG. 5B shows an example in which the image information 30b is an image I in which an obstacle Ob exists on the road surface. When step S300 is executed in this example, the portion of the road surface shown in gray is acquired as the region Zr of the image of the road surface.

Next, the control unit 20 identifies a non-road area within the area of the image of the road surface (step S305). That is, the control unit 20 extracts a non-road surface portion surrounded by the road surface image in the road surface image. At this time, the control unit 20 excludes the white line recognized by the white line recognition process. As a result, the control unit 20 considers the remaining portion as a non-road area. In the example shown in FIG. 5B, the obstacle Ob remains and is regarded as a non-road area.

Next, the control unit 20 determines whether or not the area of the non-road area is equal to or greater than the threshold value (step S310). That is, when the non-road area on the road surface is larger than a certain size, the control unit 20 considers that the non-road area is likely to be an image of an obstacle. The threshold value may be defined in advance as a value for determining whether or not it is an obstacle.

If it is not determined in step S310 that the area of the non-road area is equal to or greater than the threshold value, the control unit 20 skips step S315. When it is determined in step S310 that the area of the non-road area is equal to or larger than the threshold value, the control unit 20 determines that the image information 30b is an obstacle presence scene (step S315). That is, the control unit 20 associates the image information 30b with information indicating that the scene is an obstacle-existing scene. Since the example shown in FIG. 5B is an example in which an obstacle Ob exists on the road surface, in this example, it is determined that the area of the non-road area is equal to or larger than the threshold value, and the image I is an obstacle existence scene. It is judged. The determination as to whether or not an obstacle exists may be performed by various other methods. For example, when the presence or absence of an obstacle is recognized by object recognition and the presence or absence of an obstacle is recognized, the obstacle is recognized. A configuration or the like that determines that the scene is an object existence may be adopted.

(2-3) Scene judgment processing (map non-recording scene):
FIG. 6A is a scene determination process for determining whether or not the scene of the image information 30b is a map non-recording scene. When the scene determination process is executed, the control unit 20 acquires the current location of the vehicle (step S400). That is, the control unit 20 acquires the current location of the vehicle based on the output signals of the GNSS receiving unit 41, the vehicle speed sensor 42, and the gyro sensor 43.

Next, the control unit 20 acquires a traffic sign around the current location based on the map information (step S405). That is, the control unit 20 refers to the map information 30a and acquires the position and identification information of the traffic sign existing within the predetermined distance from the current location acquired in step S400. Next, the control unit 20 extracts the difference from the recognition result (step S410). That is, the control unit 20 refers to the recording medium 30 and acquires the identification information of the traffic sign detected from the image information 30b by the object recognition process in step S100 as the recognition result.

FIG. 6B shows an example of image information 30b including a traffic sign prohibiting turning. In this example, the identification information of the traffic sign is recorded on the recording medium 30 as a recognition result. Therefore, the control unit 20 acquires the identification information indicating the traffic sign prohibiting turning as a recognition result. Further, the control unit 20 compares the traffic sign identification information obtained as a recognition result with the traffic sign identification information acquired in step S405. Then, when there is a traffic sign that exists as a recognition result but does not exist in the map information 30a, the control unit 20 extracts the identification information of the traffic sign as a difference.

Next, the control unit 20 determines whether or not there is a difference (step S415). That is, when the difference is extracted by the process of step S410, the control unit 20 determines that there is a difference. If it is not determined in step S415 that there is a difference, the control unit 20 skips step S420.

If it is determined in step S415 that there is a difference, the control unit 20 determines that the image information 30b is a map non-recording scene (step S420). That is, the control unit 20 associates the image information 30b with information indicating that the scene is a map non-recording scene. As in the example shown in FIG. 6B, if the traffic sign prohibiting turning is recognized, but it is not recorded in the map information 30a, it is determined in step S415 that there is a difference, so that the image I is not recorded in the map. It is judged to be a scene.

(3) Other embodiments:
The above embodiment is an example for carrying out the present invention, and various other embodiments can be adopted. For example, the image transmission system may be a device mounted on a vehicle or the like, a device realized by a portable terminal, or a plurality of devices (for example, a control unit in a navigation system). It may be a system realized by a control unit or the like in the camera 40).

At least a part of the image capturing unit 21a, the scene determination unit 21b, the information reduction processing unit 21c, and the image transmission unit 21d constituting the image transmission system may be divided into a plurality of devices and exist. Of course, some configurations of the above-described embodiments may be omitted, and the order of processing may be changed or omitted. For example, the number of scenes to be determined by the scene determination unit 21b may be larger or smaller. Further, the judgment order of the scenes may be changed.

It suffices if the image capturing unit is mounted on the vehicle and can capture an image of the road on which the vehicle is traveling. That is, it suffices if the surroundings of the vehicle can be imaged while the vehicle is traveling on the road. Various devices can be assumed as the devices for acquiring images. For example, a camera mounted on a vehicle, a camera mounted on a terminal used in a vehicle interior, or the like can be assumed. The image may be an image of the road on which the vehicle is located, and may include an image of the surroundings thereof. In addition, the scenery in front of the vehicle may be photographed, and the scenery in the rear or side may be photographed.

When the object existing in the image corresponds to the important object in a specific state, the information reduction processing unit reduces the amount of information reduction per unit area of the important object per unit area of the non-important object other than the important object. It suffices if information reduction processing can be performed on the image so that the amount is less than the amount. That is, it is sufficient that the information reduction processing unit can adjust the amount of information for each area of the image and can perform the information reduction processing so that the amount of information reduction of the important object is relatively smaller than that of other objects. ..

The amount of information per unit area is represented by, for example, the number of bytes of data required to represent an image in the unit area, but the amount of information can be adjusted by performing image processing on the amount of information. be. The smaller the amount of information reduction from the captured image, the closer the image is to the object represented by the captured image, and the higher the possibility that image processing such as image recognition processing can be performed accurately.

The information reduction process for reducing the amount of information per unit area is typically a compression process, but it may be a trimming process, or a trimming process and a compression process may be used in combination. For example, information reduction processing may be performed to leave important objects and trim other objects. In addition, a portion not used for analysis, such as an empty portion, may be trimmed from the captured image. Further, the area where the amount of information is adjusted may be an area surrounded by the outline of the important object, or an area having a predetermined shape including or including the important object (for example, a rectangular area). good.

The important object may be any object that can be noticed in the image, and may be determined in advance. The important object is not limited to the object specified to be in a specific state according to the judgment of the scene. Therefore, the scene may not be determined, and the presence or absence of important objects may be determined by applying pattern matching, feature extraction, YOLO, or the like from each image.

Important objects are objects that want to suppress excessive reduction of the amount of information. Then, it is recognized as an important object when it is a specific object that can be photographed in the image and the specific object is in a predetermined specific state. The important object may be a predetermined specific state, and can be an important object when various objects are in various states other than the above examples.

An important object is an object whose importance changes depending on the position in the image, the situation of surrounding objects, the relationship with map information, etc., even if the objects are of the same type. That is, even if the object is a vehicle, whether or not it is an important object may change depending on the traffic situation, such as an object in a lane that is not congested is regarded as a non-important object. Further, even if the object is an obstacle, the object existing on the road surface on which the vehicle travels is an important object, but the object existing on the sidewalk is a non-important object, and the object may change depending on the situation.

Furthermore, even if the object is a traffic sign, if the traffic sign is recorded in the map information, it is regarded as a non-important object, and whether or not it is an important object in relation to the map information changes. Can be done. The non-important object may be any object other than the important object, and any subject outside the outline of the important object can be a non-important object. Therefore, various objects recognized as individual entities such as sidewalks and vehicles can be non-important objects, and objects that can exist continuously in an image such as the sky and sidewalks can also be non-important objects. The amount of information reduction may differ between the important object and the non-important object, and as described above, the amount of information reduction may be adjusted for each rectangular area, and inside and outside the outline of the important object. The amount of information reduction may change.

The image transmission unit only needs to be able to transmit the image after the information reduction processing to an external device. That is, when an image is transmitted and used in the destination device, if the amount of information is large, the communication speed decreases and the communication cost increases. Therefore, when it is assumed that an image is transmitted, it is preferable that the amount of information is small. On the other hand, in an image transmission system, whether or not it is an important object can change depending on whether or not it is an object of interest in an image. Therefore, by adjusting the amount of information reduction according to the importance, it is possible to reduce the amount of information to be communicated while making it possible to realize analysis according to the purpose of the image. Therefore, it is possible to efficiently reduce information.

When the information reduction process is a compression process, the compression format is not limited to the JPEG format as in the above-described embodiment, and may be various formats. For example, various compression formats such as JPEG, PNG, and GIF can be adopted. Further, the image is not limited to a still image and may be a moving image. In this case, various video compression formats can be adopted. Further, a plurality of compression formats may be used in combination, and different compression formats may be adopted for each region.

The scene determination unit only needs to be able to determine which of the predetermined scenes the image is. That is, the process for acquiring information for determining whether or not the image corresponds to the default scene and the determination criteria are predetermined, and the scene determination unit determines that the image is based on at least the image. It suffices if it can be determined whether or not it corresponds to the default scene. The scene may be at least one of the three types included in the above-described embodiment, or may include other scenes. Other scenes include, for example, the presence or absence of traffic congestion that is not limited to a specific lane, accident occurrence scenes, scenes with relatively many blind spots, scenes with relatively many road changes (curve sections, etc.). Can be mentioned.

When it is determined whether or not a specific lane on the road is congested and there is no congestion in a lane other than the specific lane, the specific lane may be any lane when viewed from the vehicle. Therefore, it is not limited to the scene in which the leftmost lane of the left-hand traffic is congested and the other lanes are not congested as in the above-described embodiment. For example, an arbitrary lane on a road section traveling in the same traveling direction as the traveling direction of the vehicle may be a specific lane. Further, any lane on the road section in which the vehicle travels in the direction opposite to the traveling direction of the vehicle may be a specific lane. When a specific lane is congested and the vehicle in the specific lane is an important object, the vehicle at the end of these important objects (the rearmost vehicle in the vehicle traveling direction) The amount of information reduction per unit area of the vehicle may be less than the amount of information reduction per unit area of the vehicle other than the end. With this configuration, when a traffic jam occurs in a specific lane, it becomes easy to analyze the vehicle at the end of the traffic jam.

The obstacle may be a feature that hinders the running of the vehicle by being stationary on the road. Therefore, the type of obstacle is not limited. Further, the process for detecting an obstacle is not limited to the above-mentioned process. For example, a feature having a specific feature may be specified as an obstacle, or an obstacle may be detected by a machine learning model in which machine learning is performed based on images of various obstacles. There may be.

When it is determined whether or not the presence of the feature included in the image is a scene not shown in the map information, the feature is not limited to the traffic sign. That is, any feature whose existence can be recorded in the map information can be targeted. For example, it may be determined whether various facilities are included in the image and not included in the map information, or whether the structure of the road (new road, etc.) is different from the structure shown in the map information. May be good.

Further, in the above-described embodiment, it is assumed that the compression ratio of the important objects is the same even when a plurality of important objects exist, but the compression ratio may be changed according to the importance. Further, the compression ratio may change depending on the position of the important object. That is, the control unit 20 may adjust the compression rate so that the higher the importance of the important object, the smaller the amount of information reduction per unit area. The importance may be determined by various factors, for example, important objects in different scenes may have different compression ratios from each other. Further, the analysis target in the important object, for example, the central portion of the traffic sign may have a lower compression ratio than the other portions.

Furthermore, the method of processing so that the amount of information reduction of important objects is smaller than the amount of information reduction of non-important objects, as in the present invention, can also be applied as a method or a program to be executed by a computer. In addition, the above systems, programs, and methods may be realized as a single device or may be realized by using parts shared with each part provided in the vehicle, including various aspects. It is a program. In addition, some of them are software and some of them are hardware, which can be changed as appropriate. Further, the invention is also established as a recording medium for a program that controls a system. Of course, the recording medium of the program may be a magnetic recording medium or a semiconductor memory, and any recording medium to be developed in the future can be considered in exactly the same way.

10 ... Navigation system, 20 ... Control unit, 21 ... Image transmission program, 21a ... Image capture unit, 21b ... Scene determination unit, 21c ... Information reduction processing unit, 21d ... Image transmission unit, 30 ... Recording medium, 30a ... Map information , 30b ... image information, 40 ... camera, 41 ... GNSS receiver, 42 ... vehicle speed sensor, 43 ... gyro sensor, 44 ... user I / F unit, 45 ... communication unit, 50 ... server

Claims

An image capturing unit that is mounted on a vehicle and captures an image of the road on which the vehicle is traveling.
For important objects in a specific state among the objects existing in the image, the amount of information reduction per unit area of the important objects is smaller than the amount of information reduction per unit area of non-important objects other than the important objects. In addition, the information reduction processing unit that performs information reduction processing on the image and
An image transmitter that transmits the image after information reduction processing to an external device,
An image transmission system equipped with.
Further, a scene determination unit for determining which of the predetermined scenes the image is based on the image is provided.
The information reduction processing unit identifies the important object according to the scene.
The image transmission system according to claim 1.
The scene determination unit
When a specific lane on the road is congested and there is no congestion in a lane other than the specific lane, it is determined that the scene is a specific lane congestion scene.
The information reduction processing unit
When the scene of the image is the specific lane congestion scene, the image of the vehicle existing on the specific lane of the image is specified as the important object.
The image transmission system according to claim 2.
The scene determination unit
If there is a stationary area surrounded by roads that is larger than the specified size, it is judged to be an obstacle presence scene, and it is determined.
The information reduction processing unit
When the scene of the image is the obstacle presence scene, the image of the region is specified as the important object.
The image transmission system according to claim 2 or 3.
The scene determination unit
If the feature included in the image does not exist in the map information, it is determined that the scene is not recorded on the map, and the scene is determined to be unrecorded.
The information reduction processing unit
When the scene of the image is the map non-recording scene, the image of the feature is specified as the important object.
The image transmission system according to any one of claims 2 to 4.