JP2009206702A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for generating an image useful to a user at the position of his own vehicle, and to provide an image processing method. <P>SOLUTION: The image processor mounted on a vehicle is provided with: an image acquiring part for acquiring photographed images photographed by a plurality of cameras; a position recognizing part for recognizing a vehicle position being information about the position of the vehicle; a position classification recognizing part for recognizing a position classification being a classification of the vehicle position recognized by the position recognizing part; a visual point determining part for determining a visual point on the basis of the position classification recognized by the position classification recognizing part; and an image generating part for generating an image seen from the visual point determined by the visual point determining part as an output image on the basis of the photographed image acquired by the image acquiring part. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for performing image processing on an image obtained by photographing the periphery of a vehicle.

Conventionally, in order to supplement the driver's viewpoint, there is a technique for displaying an image from a vehicle-mounted camera according to the operation of the vehicle. For example, an image processing device that changes the image displayed to the driver according to the vehicle running state, the running speed, the steering angle of the steering wheel, the detection result of the object detection sensor, and provides safety confirmation support when driving the vehicle (For example, refer to Patent Document 1). There is also a method for detecting a moving object from a camera mounted on a vehicle (see, for example, Patent Document 2).
Japanese Patent No. 3300334 JP-A-6-333049

  However, there are cases where a useful image cannot be provided to the driver even if the vehicle operation, the running state, or information from the object detection sensor is used. For example, information useful to the driver when driving slowly to make a right or left turn on a narrow T-junction without a signal is different from information useful to the driver when driving slowly in a parking lot. The former is an image in which the left and right in front of the own vehicle can be seen, and the latter is an image in which the own vehicle is viewed from directly above. However, the two cannot be distinguished from the vehicle operation information. In other words, there may be a case where an image appropriate for the driver cannot be generated even using the vehicle traveling state, traveling speed, steering angle, and object detection result.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus and an image processing method for generating an image useful for a user at a vehicle position.

  In order to solve the above-described problem, an aspect of the present invention is an image processing device mounted on a vehicle, an image acquisition unit that acquires captured images captured by a plurality of cameras, and information on the position of the vehicle A position recognition unit that recognizes a vehicle position, a position type recognition unit that recognizes a position type that is a type of vehicle position recognized by the position recognition unit, and a position type recognized by the position type recognition unit. A viewpoint determination unit that determines a viewpoint, and an image generation unit that generates, as an output image, an image viewed from the viewpoint determined by the viewpoint determination unit based on the captured image acquired by the image acquisition unit. Prepare. According to such an image processing device, an image useful for the user can be generated at the vehicle position.

  Further, in the image processing device according to the present invention, the image generation unit projects the captured image onto a predetermined shape as a projection image, and generates an image obtained by viewing the projection image from the viewpoint as the output image. Also good. According to such an image processing device, an output image from an arbitrary viewpoint can be generated based on a plurality of captured images.

  In the image processing device according to the present invention, the viewpoint determination unit acquires the association between the position type and the viewpoint, and based on the position type recognized by the position type recognition unit and the association, The viewpoint of the output image may be determined. According to such an image processing apparatus, a viewpoint suitable for the position type can be selected.

  In the image processing apparatus according to the present invention, the position type may include a registered position. According to such an image processing apparatus, the position type can be registered.

  In the image processing device according to the present invention, the position type recognition unit may acquire map information and recognize the position type based on the map information and the vehicle position. According to such an image processing apparatus, the position type can be recognized from the own vehicle position.

  Another embodiment of the present invention is an image processing method for performing image processing of an image obtained by photographing the periphery of a vehicle, and is a vehicle that obtains captured images captured by a plurality of cameras and is information regarding the position of the vehicle. Recognize the position, recognize the position type that is the recognized vehicle position type, determine the viewpoint based on the recognized position type, and view from the determined viewpoint based on the acquired captured image. The generated image is generated as an output image. According to such an image processing method, an image useful for the user can be generated at the vehicle position.

  Moreover, what applied the component of this invention, or arbitrary combinations of a component to a method, an apparatus, a system, a recording medium, a data structure, etc. is also contained in this invention.

  According to the disclosed image processing apparatus and image processing method, it is possible to generate an image useful to the user at the vehicle position.

  Embodiments of the present invention will be described below with reference to the drawings.

  First, the configuration of the vehicle image processing apparatus according to the present embodiment will be described.

  FIG. 1 is a block diagram showing an example of the configuration of the vehicle image processing apparatus according to the present embodiment. The vehicle image processing apparatus includes an image photographing unit 1, a vehicle state detection / recognition unit 2, an image processing unit 3, and an image display unit 4.

  The image capturing unit 1 includes N cameras 11 attached to the vehicle, a camera parameter table 1a that stores position information (camera parameters) of each camera 11 in advance, and images (captured images) captured by the cameras 11. An image data buffer 1b (image acquisition unit) for temporary storage is provided. In the present embodiment, the number N of cameras 11 is four.

  FIG. 2 is a table showing an example of the camera parameter table according to the present embodiment. The camera parameter table 1a stores information about the camera number, position coordinates (X, Y, Z) in the host vehicle coordinate system, azimuth angle, elevation angle, angle around the optical axis, and angle of view for each camera 11. The definition of each value uses the definition of patent document 1, for example.

  FIG. 3 is a layout view of an example of the camera layout according to the present embodiment as viewed from the left side of the vehicle. FIG. 4 is a layout view of an example of the camera layout according to the present embodiment as viewed from above the vehicle. FIG. 5 is a layout view of an example of the camera layout according to the present embodiment as viewed from the front of the vehicle. The four cameras 11 are assigned camera numbers 1, 2, 3, and 4, respectively, and respectively photograph the vehicle front, the vehicle right side, the vehicle rear, and the vehicle left side.

  FIG. 6 is a diagram illustrating an example of a captured image according to the present embodiment. In this figure, the upper left image is an image taken by the rear camera. Similarly, the upper right image is an image taken by the front camera. Similarly, the lower left image is an image taken by the right camera. Similarly, the lower right image is an image taken by the left camera.

  FIG. 7 is a diagram illustrating an example of an output image generated by the image processing unit according to the present embodiment. This output image displays the result of image processing of the captured image and a preset shape of the vehicle.

  In the present embodiment, the host vehicle coordinate system is used. FIG. 8 is a diagram illustrating an example of the host vehicle coordinate system according to the present embodiment. The own vehicle coordinate system is a three-dimensional orthogonal coordinate system in which an arbitrary point of the own vehicle is an origin O. In the present embodiment, the traveling direction is the X axis, the axis parallel to the ground and orthogonal to the right direction with respect to the traveling direction is the Y axis, and the upward direction orthogonal to the XY plane is the Z axis. The origin is the center of gravity of the coordinates of the four cameras 11. The camera parameter table 1a described above stores information on the position coordinates X, Y, and Z, the azimuth angle, the elevation angle, and the optical axis angle measured in the vehicle coordinate system when the camera 11 is installed.

  Images taken by the cameras 11 are temporarily stored in the image data buffer 1b. In the present embodiment, in order for the approaching object recognition unit 25 of the vehicle condition detection / recognition unit 2 to be described later to detect an approaching object to the host vehicle, the image data buffer 10 displays the image of each camera 11 in the time axis direction. Accumulate more than the specimen. For example, if the camera 11 takes 33 milliseconds to capture one frame (image), the image data buffer 10 holds the frame acquired at the present time and the frame acquired 33 milliseconds ago. Since there are four cameras, the image data buffer 1b always stores eight images.

  The vehicle state detection / recognition unit 2 includes a position / traveling direction detection unit 21 (position recognition unit), a map information DB (database) 22, a position type recognition unit 23, a driving state detection unit 24 (driving information acquisition unit), and an approaching object. A recognition unit 25 (object detection unit) is provided.

  The position / traveling direction detection unit 21 acquires own vehicle position information (latitude, longitude, traveling direction) (vehicle position). Here, when there is a request from the position type recognition unit 23 or the image processing unit 3, the position / traveling direction detection unit 21 returns the vehicle position information to the request source. The position / traveling direction detection unit 21 is mounted using a GPS (Global Positioning Systems) position measuring device and an electronic compass device.

  The map information DB 22 uses a database such as a car navigation system. The map information DB 22 is mounted using a hard disk. FIG. 9 is a table showing an example of the map information DB according to the present embodiment. For each node (point), the latitude, longitude, number of links (number of roads connected to that node), connection node number (number of other nodes connected to that node via roads), and location type are stored. Has been.

  The position type recognition unit 23 recognizes a position type that is the type of the current position of the vehicle. In the present embodiment, the position type is “intersection (other than T-shaped intersection, crossroad, etc.)” “parking lot” “T-shaped intersection” “other (other than the above four types)”. The position type recognition unit 23 refers to the map information DB 22 and acquires a position type corresponding to the latitude and longitude obtained from the position / traveling direction detection unit 21. The recognition result is sent to the image processing unit 3. In this embodiment, four types of positions are used, but the number of types of positions may be increased by combining road widths (widths of driving lanes) and the like.

  The driving state detection unit 24 detects the driving state from the vehicle operation state and the driving state correspondence table, and sends the detection result to the image processing unit 3. In the present embodiment, there are four types of vehicle operation states: “steering angle of steering wheel”, “direction indicator”, “traveling speed”, and “select lever range”. Further, in the present embodiment, the driving state takes eight types of values: “straight ahead”, “left turn”, “right turn”, “left lane change”, “right lane change”, “slow travel”, “reverse”, and “others”.

  FIG. 10 is a table showing an example of the operation state correspondence table according to the present embodiment. “---” in the figure means that there is no condition. For example, when the “select lever range” is “R”, the operation state of the seventh stage in the figure is “reverse” regardless of three types of states such as “steering angle of steering wheel” and “direction indicator”. " In the present embodiment, the vehicle operation state has four items, but other items may be added. For example, there are “depressed amount of accelerator pedal” and “depressed amount of brake pedal”. Regarding the driving state, instead of “left turn” and “right turn”, “turning left”, “waiting left turn”, “turning right”, “waiting right turn”, etc. may be used.

  The approaching object recognition unit 25 checks for the presence of an object around the host vehicle. The image accumulated in the image data buffer 1b is acquired via the image processing unit 3, and the presence or absence of an object approaching the host vehicle is detected from the accumulated eight images. When an approaching object exists, the position (X, Y, Z) of each object in the vehicle coordinate system is also detected. As a method for detecting the presence or absence of a moving object, for example, the method described in Patent Document 2 is used.

  The image generation unit 3 includes an image generation unit 31 (viewpoint determination unit and image generation unit) and an image parameter DB 32.

  The image generation unit 31 projects a captured image onto a projection shape to generate a projection image, determines a virtual viewpoint based on information obtained from each unit, and generates an image obtained by viewing the projection image from the virtual viewpoint as an output image. . In the present embodiment, the image generation unit 31 obtains the image captured by each camera 11 and information such as the position and angle of each camera 11 from the image capturing unit 1. Further, the image generation unit 31 obtains the latitude, longitude, travel information, the driving state of the vehicle, the number of approaching objects to the host vehicle, and each object position coordinate from the vehicle state detection / recognition unit 2. The image generation unit 31 obtains image parameters necessary for image generation from the image parameter DB 32.

  The image parameter DB 32 is a database in which parameters necessary for image generation are determined in advance. FIG. 11 is a table showing an example of the image parameter DB according to the present embodiment. The image parameter DB 32 in the present embodiment includes a registered position DB 321, a position type DB 322, a driving state DB 333, an approaching object number DB 324, and an image generation setting DB 325.

  FIG. 12 is a table showing an example of the registration position DB according to the present embodiment. The registered position DB 321 defines a registered position (registered name, latitude, longitude) that is a position registered by the user, and settings (image generation settings) used to generate an output image to be displayed at the position. For example, when the host vehicle exists within a radius of 20 meters centering on the latitude and longitude registered as the home parking lot, the image generation unit 31 generates an output image overlooking the host vehicle using the image generation setting PD. To do. The image generation setting PD represents an output image setting in an image generation setting DB 325 described later.

  FIG. 13 is a table showing an example of the position type DB according to the present embodiment. The position type DB 322 defines a position type and an image generation setting used for the position type. In the present embodiment, the image generation setting when the position type is “intersection” is not defined. In a position type for which no image generation setting is defined, the image generation setting is determined according to the driving state of the vehicle and the like as will be described later.

  FIG. 14 is a table showing an example of the operation state DB according to the present embodiment. The operation state DB 323 defines an operation state and an image generation setting used for the operation state. In the present embodiment, the image generation setting when the driving state is “straight ahead” is not defined. In the driving state where the image generation setting is not defined, the image generation setting is determined based on the number of approaching objects and the like as will be described later.

  FIG. 15 is a table showing an example of the approaching object number DB according to the present embodiment. The approaching object number DB 324 defines the approaching object number and the image generation setting used for the approaching object number. In the present embodiment, the image generation setting when the number of approaching objects is 0 is not defined.

  FIG. 16 is a table showing an example of the image generation setting DB according to the present embodiment. The image generation setting DB 325 stores a projection shape parameter for determining a projection shape and a virtual viewpoint parameter for determining a virtual viewpoint as parameters necessary for the output image to be generated. In the present embodiment, the projection shape is a spherical surface. FIG. 17 is an expression showing an example of a projected shape according to the present embodiment. The projection shape parameter is the radius R of the sphere in this equation. X, Y, and Z are the host vehicle coordinate systems.

  In the present embodiment, image generation settings PA, PB, PC, PD, PE, and PF are defined as image generation settings in the image generation setting DB 325. Virtual viewpoint parameters are set in advance for the image generation settings PA, PB, PC, and PD.

  The image generation setting PA is a setting for generating an output image through which the left rear of the host vehicle can be seen. The image generation setting PB is a setting for generating an output image through which the right rear of the host vehicle can be seen. The image generation setting PC is a setting for generating an output image through which the back of the vehicle can be seen. The image generation setting PD is a setting for generating an output image overlooking the host vehicle. The image generation setting PE is a setting for generating an output image in which an approaching object is reflected, and a virtual viewpoint parameter that allows an approaching object to be reflected in the output image is determined based on a preset virtual viewpoint parameter. The image generation setting PF is a setting for generating an output image in which the left and right in front of the host vehicle can be seen, and an image captured by the front camera is used as an output image as it is.

  The image generation unit 31 generates a projection image by projecting a captured image captured by each camera 11 onto a projection shape. FIG. 18 is a diagram showing an example of a projected image according to the present embodiment. The projection image is generated using the camera parameters in the camera parameter table 1a and the captured images obtained by the cameras 11. Here, the own vehicle is arranged below the projected shape.

  Further, the image generation unit 31 projects the projection image on a two-dimensional plane using the value of the virtual viewpoint parameter to obtain an output image. A method for projecting captured images obtained by a plurality of cameras into a three-dimensional shape based on camera positions and directions, and a method for generating an image from a virtual viewpoint using the projection results are described in, for example, Patent Document 1 Is used.

  The image display unit 4 displays the output image generated by the image processing unit 3.

  Next, a method for determining the virtual viewpoint of the image generation setting PE will be described.

  The approaching object detection unit 25 detects an approaching object using the captured image stored in the image data buffer 1b. In the present embodiment, the detection of the approaching object uses only the captured images of the left, right, and rear cameras, and does not use the captured images of the front camera. In addition, virtual viewpoints a, b, c, d, and e are set in advance as virtual viewpoints useful to the driver when an approaching object is detected. As virtual viewpoint parameters for determining the virtual viewpoint, the coordinates and direction of the virtual viewpoint with respect to the projection image are set.

  FIG. 19 is a diagram illustrating an example of the virtual viewpoint a according to the present embodiment. FIG. 20 is a diagram illustrating an example of the virtual viewpoint b according to the present embodiment. FIG. 21 is a diagram illustrating an example of the virtual viewpoint c according to the present embodiment. FIG. 22 is a diagram illustrating an example of the virtual viewpoint d according to the present embodiment. FIG. 23 is a diagram illustrating an example of the virtual viewpoint e according to the present embodiment. The virtual viewpoint a is a viewpoint in which the left rear side of the own vehicle is viewed from the right side of the own vehicle. The virtual viewpoint b is a viewpoint for viewing the right rear side of the own vehicle from the left side of the own vehicle. The virtual viewpoint c is a viewpoint in which the rear side of the own vehicle is viewed from the right side of the own vehicle. The virtual viewpoint d is a viewpoint in which the rear side of the own vehicle is viewed from the left side of the own vehicle. The virtual viewpoint e is a viewpoint in which the rear of the host vehicle is viewed from the front of the host vehicle, and is a virtual viewpoint in which almost all captured images of the left, right, and rear cameras are shown.

  When the approaching object is detected, the image generation unit 31 detects the coordinates (X_A, Y_A) of the center point A of the circumscribed rectangle of the approaching object in the output image using each of the virtual viewpoints a, b, c, and d. The coordinates of the origin B (X_B, Y_B) of the vehicle coordinate system and the coordinates of the midpoint C (X_C, Y_C) of the point A and the point B are obtained.

  FIG. 24 is a diagram illustrating an example of an output image of the virtual viewpoint b when there is one approaching object according to the present embodiment. Here, the lower left of the output image is the origin O ′, the horizontal axis is the X ′ axis, and the vertical axis is the Y ′ axis. Further, among the coordinates of the vertices of the circumscribed rectangle of the approaching object, the X ′ coordinate at the left end is X_left, the Y ′ coordinate at the upper end is Y_top, the X ′ coordinate at the right end is X_right, and the Y ′ coordinate at the lower end is Y_bottom. Also, let D be the center point of the output image. FIG. 25 is an expression showing the condition of the position of the circumscribed rectangle of the approaching object according to the present embodiment. Here, W and H are the width and height of the output image, respectively. The image generation unit 31 uses a virtual viewpoint in which X_left, Y_top, X_right, and Y_bottom satisfy this condition, and does not use a virtual viewpoint that does not satisfy this condition.

  Further, the image generation unit 31 selects a virtual viewpoint that minimizes the distance of the line segment CD from among the virtual viewpoints that satisfy the condition of the position of the circumscribed rectangle of the approaching object. Further, when all of the virtual viewpoints a, b, c, and d do not satisfy the condition of the position of the circumscribed rectangle of the approaching object, the image generation unit 31 uses the virtual viewpoint e.

  Note that the image generation unit 31 may change the virtual viewpoint parameter in the vicinity of the initial value in obtaining the distance between the line segments CD. For example, when the X coordinate of the virtual viewpoint is set to 10, the image generation unit 31 obtains the distance between the line segments CD when the X coordinate is changed to 5 or 15 and the distance is the minimum. The virtual viewpoint parameter to be selected is selected. Similarly, the image generation unit 31 may change virtual viewpoint parameters other than the X coordinate. The number of virtual viewpoint parameter combinations to be changed may be changed according to the processing performance of the image generation unit 31.

  Next, a case where there are two or more approaching objects will be described. The center of the circumscribed rectangle of each approaching object is A1, A2,. And the center of gravity of these points is point A ′. Also, let C be the midpoint between the point A 'and the origin B of the host vehicle coordinate system, and D be the center of the image coordinate system. At this time, the image generation unit 31 selects a virtual viewpoint that minimizes the distance of the line segment CD among the virtual viewpoints that satisfy the condition of the position of the circumscribed rectangle of the approaching object.

  FIG. 26 is a diagram illustrating an example of an output image of the virtual viewpoint b when there are two approaching objects according to the present embodiment. In this case, the center of gravity (middle point) of the line connecting A1 and A2 is A '.

  In addition, when there is no virtual viewpoint that satisfies the condition of the position of the circumscribed rectangle of the approaching object and a plurality of detected approaching objects appear in the output image at the same time, the image generation unit 31 A virtual viewpoint to be displayed in the output image with priority given only to an approaching object having a high approach speed to the host vehicle is selected.

  When the same approaching object is captured by a plurality of different cameras 11, the image generation unit 31 needs to recognize that the plurality of captured images show the same object. For example, when the same approaching object (such as a car) is shown in the left camera and the rear camera, it is necessary to recognize that they are the same so that the number of approaching objects is not counted repeatedly.

  When the same object is reflected in the images captured by the plurality of cameras 11, the object should be reflected at the end of each captured image. For example, when the same object is shown in the left camera and the rear camera, the left end of the image captured by the left camera and the right end of the image captured by the rear camera are detected as the approaching object region. Therefore, when an approaching object is detected in the predetermined pixel region at the left end of the captured image of the left camera and the predetermined pixel region at the right end of the captured image of the rear camera, the image generation unit 31 sets them as the same object. recognize.

  With the above-described virtual viewpoint determination method, at least a part of the own vehicle and the approaching object can be reflected in the generated image. Conventionally, there has been a technique for displaying the center of an approaching object at the center of the output image. However, in this case, the vehicle may not be reflected, and in this case, the user cannot easily understand the distance to the approaching object. On the other hand, the output image according to the present embodiment can display the sense of distance between them in an easy-to-understand manner.

  Next, a viewpoint image indicating a virtual viewpoint will be described.

  The viewpoint image is an image that clearly shows the virtual viewpoint of the output image to the driver, and is displayed together with or in the output image.

  When the image generation setting is switched, the user may not easily grasp the virtual viewpoint instantaneously. By displaying the viewpoint image showing the virtual viewpoint along with the output image,

  A plurality of viewpoint areas defined in advance in the range (viewpoint area) of the azimuth angle α of the virtual viewpoint (the angle formed with the y-axis of the host vehicle coordinate system) are set. In addition, an image of the own vehicle (own vehicle image) viewed from the viewpoint area is stored in advance for each viewpoint area. FIG. 27 is a diagram showing an example of the viewpoint area according to the present embodiment. In the present embodiment, six viewpoint regions T1 to T6 are defined with respect to the azimuth angle of the virtual viewpoint.

  FIG. 28 is a table showing an example of the correspondence between the viewpoint area and the vehicle image according to the present embodiment. As the association between the viewpoint area and the own vehicle image, the viewpoint area number and the own vehicle image are defined for each range of α. The image generation unit 31 selects the vehicle image used for the viewpoint image according to the association and the value of α.

  FIG. 29 is a diagram showing an example of the own vehicle image according to the present embodiment. Va is an image of the host vehicle viewed from the right side of the host vehicle, and is used when the virtual viewpoint exists on the right side of the host vehicle. Vb is an image of the host vehicle viewed from the front of the host vehicle, and is used when the virtual viewpoint exists in front of the host vehicle. Vc is an image of the vehicle viewed from the back of the vehicle, and is used when the virtual viewpoint exists behind the vehicle.

  Furthermore, the image generation unit 31 generates a viewpoint image by placing a camera figure (camera mark) representing a virtual viewpoint on the selected vehicle image. The host vehicle image is divided in advance in a grid pattern, and the grid in the host vehicle image is associated with the coordinates of the virtual viewpoint.

  FIG. 30 is a diagram showing an example of the arrangement of camera marks in the own vehicle image according to the present embodiment. In this example, Va is selected as the vehicle image. The image generation unit 31 divides the region in Va into lattice-like regions M1, M2,..., And places camera marks on the lattice in Va corresponding to the x coordinate and the z coordinate of the virtual viewpoint. For example, when the x coordinate of the virtual viewpoint is −20 to −10 and the z coordinate is 30 or more, the camera mark is determined to be arranged in the region M1. Next, the image generation unit 31 determines the orientation of the camera mark using the elevation angle value of the virtual viewpoint parameter.

  FIG. 31 is a diagram illustrating an example of an output image to which a viewpoint image according to the present embodiment is added. In the present embodiment, the image generation unit 31 arranges the viewpoint image in the lower right part of the output image.

  Further, when there is no virtual viewpoint parameter, that is, in the case of the image generation setting PF, the image generation unit 31 generates a viewpoint image using the camera parameter.

  The image generation unit 31 generates a virtual viewpoint obtained by interpolating from the virtual viewpoint before switching to the virtual viewpoint after switching when switching the virtual viewpoint, and switches from the output image before switching using the interpolated virtual viewpoint. An output image obtained by continuously changing up to the subsequent output image may be generated.

  The vehicular image processing apparatus according to the present embodiment may be realized by a CPU and a storage device. At this time, the position / traveling direction detection unit 21, the map information DB (database) 22, the position type recognition unit 23, the driving state detection unit 24, the approaching object recognition unit 25, and the image generation unit 3 are programs executed by the CPU. It is stored in a storage device. The vehicle image processing apparatus according to the present embodiment can be applied to a car navigation system.

  Next, the operation of the vehicle image processing apparatus according to this embodiment will be described.

  FIG. 32 is a flowchart illustrating an example of the operation of the vehicle image processing device according to the present embodiment. The captured image captured by each camera 11 of the image capturing unit 1 is temporarily stored in the image data buffer 1a.

  First, the position / traveling direction detection unit 21 acquires the vehicle position (latitude, longitude, traveling direction) and sends it to the image processing unit 3 (S1). The image processing unit 3 that has acquired the own vehicle position refers to the registered position DB 321 and determines whether the own vehicle position corresponds to one of the registered positions (S2). Here, the image processing unit 3 determines that the own vehicle position corresponds to the registered position if the own vehicle position is within a radius of 20 meters of the registered position.

  When the own vehicle position corresponds to the registered position (S2, Y), the image processing unit 3 determines the image generation setting corresponding to the own vehicle position by the registered position DB 321 and generates an output image according to the image generation setting (S9). ) And displayed on the image display unit 4 (S10). Next, the image processing unit 3 determines whether or not the driving of the host vehicle has ended (S11). If it has ended (S11, Y), this flow ends, and if it has not ended. (S11, N), the process returns to S1. Whether or not the driving of the host vehicle has ended may be identified by whether or not the engine of the host vehicle is running.

  When the own vehicle position does not correspond to the registered position (S2, N), the position type recognition unit 23 acquires the own vehicle position from the position / traveling direction detection unit 21, and recognizes the position type of the own vehicle position from the map information DB 22. The recognition result is sent to the image processing unit 3 (S3). Next, the image processing unit 3 refers to the position type DB 322 and determines whether there is an image generation setting corresponding to the acquired position type (S4).

  When there is an image generation setting corresponding to the acquired position type (S4, Y), the image processing unit 3 determines the image generation setting corresponding to the acquired position type using the position type DB 322, and proceeds to the process S9.

  When there is no image generation setting corresponding to the acquired position type (S4, N), the driving state detection unit 24 detects the driving state from the operation state of the vehicle, and sends the detection result to the image processing unit 3 (S5). Next, the image processing unit 3 refers to the driving state DB 323 and determines whether there is an image generation setting corresponding to the acquired driving state (S6).

  When the image generation setting corresponding to the acquired driving state exists (S6, Y), the image processing unit 3 determines the image generation setting corresponding to the acquired driving state by the driving state DB 323, and proceeds to the process S9.

  When there is no image generation setting corresponding to the acquired driving state (S6, N), the approaching object detection unit 25 detects an approaching object to the host vehicle and sends the detection result to the image processing unit 3 (S7). Next, the image processing unit 3 refers to the approaching object number DB 324 and determines whether there is an image generation setting corresponding to the acquired approaching object number (S8).

  When the image generation setting corresponding to the acquired approaching object number exists (S8, Y), the image processing unit 3 determines the image generation setting corresponding to the acquired approaching object number by the approaching object number DB 324, and proceeds to the process S9. Transition.

  When there is no image generation setting corresponding to the acquired number of approaching objects (S8, N), the process proceeds to S11.

  According to the present embodiment, it is possible to recognize the position of the current position of the vehicle, the driving state, the number of approaching objects, and the like as the vehicle situation. Thereby, an appropriate output image can be set for various vehicle situations, and a useful image can be displayed.

  Furthermore, since the setting of the output image can be changed corresponding to the position registered in advance by the user, for example, a vehicle status detection / recognition unit such as “home parking lot” or “a specific place on the road from home to work place” 2 makes it possible to display useful images even in situations where it is difficult to identify.

  The present invention can be implemented in various other forms without departing from the spirit or main features thereof. Therefore, the above-described embodiment is merely an example in all respects and should not be interpreted in a limited manner. The scope of the present invention is shown by the scope of claims, and is not restricted by the text of the specification. Further, all modifications, various improvements, alternatives and modifications belonging to the equivalent scope of the claims are all within the scope of the present invention.

  Regarding the above embodiment, the following additional notes are disclosed.

(Appendix 1) An image processing apparatus mounted on a vehicle,
An image acquisition unit for acquiring captured images captured by a plurality of cameras;
A position recognition unit for recognizing a vehicle position, which is information relating to the position of the vehicle;
A position type recognition unit for recognizing a position type that is a type of vehicle position recognized by the position recognition unit;
A viewpoint determination unit that determines a viewpoint based on the position type recognized by the position type recognition unit;
An image generation unit that generates, as an output image, an image viewed from the viewpoint determined by the viewpoint determination unit based on the captured image acquired by the image acquisition unit;
An image processing apparatus comprising:
(Supplementary Note 2) In the image processing apparatus according to Supplementary Note 1,
The image generation unit is an image processing apparatus that projects the captured image into a predetermined shape to form a projection image, and generates an image obtained by viewing the projection image from the viewpoint as the output image.
(Supplementary Note 3) In the image processing apparatus according to Supplementary Note 1,
The viewpoint determination unit acquires an association between the position type and the viewpoint, and determines the viewpoint of the output image based on the position type recognized by the position type recognition unit and the association. .
(Supplementary Note 4) In the image processing apparatus according to Supplementary Note 1,
The position type is an image processing apparatus including a registered position.
(Supplementary Note 5) In the image processing apparatus according to Supplementary Note 1,
The position type recognition unit is an image processing apparatus that acquires map information and recognizes a position type based on the map information and the vehicle position.
(Supplementary note 6) In the image processing apparatus according to supplementary note 1,
And a driving information acquisition unit that acquires driving information that is information related to driving of the vehicle,
The viewpoint determination unit is an image processing device that determines a viewpoint based on driving information acquired by the driving information acquisition unit when there is no association between the position type recognized by the position type recognition unit and the viewpoint.
(Supplementary note 7) In the image processing apparatus according to supplementary note 6,
The viewpoint determination unit acquires an association between the driving information and the viewpoint, and determines the viewpoint of the output image based on the driving information acquired by the driving information acquisition unit and the association .
(Supplementary note 8) In the image processing apparatus according to supplementary note 1,
Furthermore, an object detection unit that detects the position of an object around the vehicle,
The viewpoint determination unit is an image processing apparatus that determines a viewpoint based on a position of an object detected by the object detection unit when there is no correspondence between the position type recognized by the position type recognition unit and the viewpoint.
(Supplementary note 9) In the image processing apparatus according to supplementary note 8,
The viewpoint determination unit is an image processing apparatus that determines a viewpoint of the output image so that the position of the object detected by the object detection unit and the position of the vehicle are reflected in the output image.
(Supplementary Note 10) In the image processing apparatus according to Supplementary Note 1,
Furthermore, an image processing apparatus that displays an output image generated by the image generation unit.
(Supplementary note 11) In the image processing apparatus according to supplementary note 1,
The image generation device further includes a viewpoint image representing the viewpoint determined by the viewpoint determination unit in the output image.
(Supplementary Note 12) In the image processing apparatus according to Supplementary Note 11,
The said viewpoint image is an image processing apparatus which is an image of the said vehicle seen from the viewpoint determined by the said viewpoint determination part.
(Supplementary note 13) An image processing method for performing image processing of an image obtained by photographing the periphery of a vehicle,
Acquire images taken by multiple cameras,
Recognizing the vehicle position, which is information about the position of the vehicle,
Recognize the position type, which is the recognized vehicle position type,
Based on the recognized location type, determine the viewpoint,
An image processing method for generating, as an output image, an image viewed from a determined viewpoint based on an acquired captured image.
(Supplementary note 14) In the image processing method according to supplementary note 13,
An image processing method for projecting the photographed image into a predetermined shape to form a projected image, and generating an image obtained by viewing the projected image from the viewpoint as the output image.
(Supplementary note 15) In the image processing method according to supplementary note 13,
An image processing method for acquiring an association between the position type and the viewpoint and determining a viewpoint of the output image based on the recognized position type and the association.
(Supplementary note 16) In the image processing method according to supplementary note 13,
The position type is an image processing method including a registered position.
(Supplementary note 17) In the image processing method according to supplementary note 13,
An image processing method for acquiring map information and recognizing a position type based on the map information and the vehicle position.
(Supplementary note 18) In the image processing method according to supplementary note 13,
Furthermore, driving information that is information related to driving of the vehicle is acquired,
An image processing method for determining a viewpoint based on acquired driving information when there is no correspondence between a position type recognized by the position type recognition unit and a viewpoint.
(Supplementary note 19) In the image processing method according to supplementary note 18,
An image processing method for acquiring association between the driving information and the viewpoint and determining a viewpoint of the output image based on the driving information acquired by the driving information acquisition unit and the association.
(Supplementary note 20) In the image processing method according to supplementary note 13,
Furthermore, the position of the object around the vehicle is detected,
An image processing method for determining a viewpoint based on a detected position of an object when there is no correspondence between the position type recognized by the position type recognition unit and the viewpoint.

It is a block diagram which shows an example of a structure of the image processing apparatus for vehicles which concerns on this Embodiment. It is a table | surface which shows an example of the camera parameter table which concerns on this Embodiment. It is the layout which looked at the example of arrangement | positioning of the camera which concerns on this Embodiment from the vehicle left side. It is the layout which looked at the example of arrangement | positioning of the camera which concerns on this Embodiment from the vehicle upper direction. It is the layout which looked at the example of arrangement | positioning of the camera which concerns on this Embodiment from the vehicle front. It is a figure which shows an example of the picked-up image which concerns on this Embodiment. It is a figure which shows an example of the output image produced | generated by the image process part which concerns on this Embodiment. It is a figure which shows an example of the own vehicle coordinate system which concerns on this Embodiment. It is a table | surface which shows an example of map information DB which concerns on this Embodiment. It is a table | surface which shows an example of the driving | operation state corresponding | compatible table which concerns on this Embodiment. It is a table | surface which shows an example of image parameter DB which concerns on this Embodiment. It is a table | surface which shows an example of registration position DB which concerns on this Embodiment. It is a table | surface which shows an example of position classification DB which concerns on this Embodiment. It is a table | surface which shows an example of driving | running state DB which concerns on this Embodiment. It is a table | surface which shows an example of approaching object number DB which concerns on this Embodiment. It is a table | surface which shows an example of image generation setting DB which concerns on this Embodiment. It is a formula which shows an example of the projection shape concerning this embodiment. It is a figure which shows an example of the projection image which concerns on this Embodiment. It is a figure which shows an example of the virtual viewpoint a which concerns on this Embodiment. It is a figure which shows an example of the virtual viewpoint b which concerns on this Embodiment. It is a figure which shows an example of the virtual viewpoint c which concerns on this Embodiment. It is a figure which shows an example of the virtual viewpoint d which concerns on this Embodiment. It is a figure which shows an example of the virtual viewpoint e which concerns on this Embodiment. It is a figure which shows an example of the output image of the virtual viewpoint b in case the number of the approaching objects which concern on this Embodiment is one. It is a type | formula which shows the conditions of the position of the circumscribed rectangle of the approaching object which concerns on this Embodiment. It is a figure which shows an example of the output image of the virtual viewpoint b in case the two approaching objects which concern on this Embodiment are two. It is a figure which shows an example of the viewpoint area | region which concerns on this Embodiment. It is a table | surface which shows an example of matching with the viewpoint area | region and own vehicle image which concern on this Embodiment. It is a figure which shows an example of the own vehicle image which concerns on this Embodiment. It is a figure which shows an example of arrangement | positioning of the camera mark in the own vehicle image which concerns on this Embodiment. It is a figure which shows an example of the output image to which the viewpoint image which concerns on this Embodiment was added. It is a flowchart which shows an example of operation | movement of the vehicle image processing apparatus which concerns on this Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image photographing part, 2 Vehicle condition detection / recognition part, 3 Image processing part, 4 Image display part, 11 Camera, 1a Camera parameter table, 1b Image data buffer, 21 Position / traveling direction detection part, 22 Map information DB, 23 Position type recognition unit, 24 driving state detection unit, 25 approaching object recognition unit, 31 image generation unit, 32 image parameter DB, 321 registration position DB, 322 position type DB, 333 driving state DB, 324 approaching object number DB, 325 image Generation setting DB.

Claims (6)

An image processing apparatus mounted on a vehicle,
An image acquisition unit for acquiring captured images captured by a plurality of cameras;
A position recognition unit for recognizing a vehicle position, which is information relating to the position of the vehicle;
A position type recognition unit for recognizing a position type that is a type of vehicle position recognized by the position recognition unit;
A viewpoint determination unit that determines a viewpoint based on the position type recognized by the position type recognition unit;
An image generation unit that generates, as an output image, an image viewed from the viewpoint determined by the viewpoint determination unit based on the captured image acquired by the image acquisition unit;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The image generation unit is an image processing apparatus that projects the captured image into a predetermined shape to form a projection image, and generates an image obtained by viewing the projection image from the viewpoint as the output image. The image processing apparatus according to claim 1 or 2,
The viewpoint determination unit acquires an association between the position type and the viewpoint, and determines the viewpoint of the output image based on the position type recognized by the position type recognition unit and the association. . The image processing apparatus according to any one of claims 1 to 3,
The position type is an image processing apparatus including a registered position. The image processing apparatus according to any one of claims 1 to 4,
The position type recognition unit is an image processing apparatus that acquires map information and recognizes a position type based on the map information and the vehicle position. An image processing method for performing image processing of an image obtained by photographing a periphery of a vehicle,
Acquire images taken by multiple cameras,
Recognizing the vehicle position, which is information about the position of the vehicle,
Recognize the position type, which is the recognized vehicle position type,
Based on the recognized location type, determine the viewpoint,
An image processing method for generating, as an output image, an image viewed from a determined viewpoint based on an acquired captured image.

Images