JP2007334426A - Image processing program, image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing program, an image processor and an image processing method, using an appropriate image among externally photographed and transmitted images. <P>SOLUTION: This image processing method makes a computer execute: a reception step for receiving photographic data comprising photographic image data as an image photographed by an external camera and photographic meta data as information relating to a state that the photographic image data is photographed; a state acquisition step for acquiring state data as information relating to the state of a traveling object; a decision step for deciding whether to use the photographic data received in the reception step based on preliminarily set conditions and the photographic meta data received in the reception step and the state data acquired in the state acquisition step; and an arithmetic step for executing an arithmetic operation by using the photographic data decided to be used in the decision step. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing program, an image processing apparatus, and an image processing method that use an image received from the outside in a moving body.

  At an intersection, an accident occurs (a right-handed accident) in which a right-turn vehicle that is a vehicle that is about to turn right and an opposite-straight vehicle that is a vehicle that is going straight in the opposite lane contact each other. FIG. 23 is a plan view showing an example of the positional relationship between the own vehicle and the oncoming vehicle when turning right. FIG. 24 is an image showing an example of a visual recognition result viewed from the own vehicle when making a right turn. As shown in these drawings, the driver of the host vehicle 3 cannot visually recognize because the oncoming straight vehicle 4x enters the blind spot of the oncoming vehicle 4y. In order to prevent such an accident caused by a blind spot, there is an attempt to use an in-vehicle camera provided in the own vehicle for the purpose of recognizing the danger of a blind spot.

  For example, as a first conventional technique, there is an attempt to install a camera that captures an image of an intersection on the road side as an infrastructure and use the image on the vehicle side (for example, AHS: Advanced cruise-assist Highway Systems). This is a method in which an image of a fixed camera installed on a road is transmitted to a predetermined area, and a vehicle passing through the area receives and uses the image. ID or camera position.

  As a second conventional technique, there is an attempt to perform danger recognition using an image of an in-vehicle camera installed in another vehicle (see, for example, Non-Patent Document 1). This is because the vehicle running in front captures the front image, acquires the position and direction of the following vehicle, converts the captured image into an image viewed from the camera of the following vehicle, and transmits it to the following vehicle through inter-vehicle communication. It is.

As a related art related to the present invention, there is an inter-vehicle communication information processing apparatus that creates an inter-vehicle network, receives a camera video from a camera holding vehicle, and displays it (for example, see Patent Document 1).
JP 2004-145672 A "Examination of position / orientation information acquisition of in-vehicle camera in ITS-Application to HIR system-", Masato Koyamaishi, Hiroshi Sakai, Toshiaki Fujii, Masayuki Tanimoto (Nagoya Univ.), IEICE Tech.

  However, in the first prior art described above, since the roadside camera does not transmit fine camera parameters (view angle, elevation angle, focal length, etc.), the vehicle side tries to process and use the received image. There is a problem that the exact shooting space is not known and cannot be used.

  In the second prior art described above, when a vehicle at the center of an intersection transmits an image to a plurality of opponents, the accurate position and direction of the in-vehicle cameras of all the opponents are acquired and viewed from those cameras. Therefore, there is a problem that the calculation cost (processing load and processing time) is very high.

  The present invention has been made to solve the above-described problems, and provides an image processing program, an image processing apparatus, and an image processing method that use an appropriate image out of images captured and transmitted externally. Objective.

  In order to solve the above-described problem, the present invention is an image processing program for causing a computer provided in a moving body to perform calculation of data transmitted from the outside, and is an image captured by an external camera. A receiving step for receiving image data including image data and shooting metadata that is information on a state in which the captured image data was shot; a state acquisition step for acquiring state data that is information on the state of the moving body; Judgment as to whether or not to use the shooting data received in the reception step based on preset conditions, the shooting metadata received in the reception step and the state data acquired in the state acquisition step. A determination step to be performed, and an operation to perform an operation using the shooting data determined to be used in the determination step. To perform the steps on a computer.

  According to this image processing program, appropriate shooting data can be used by receiving shooting data consisting of shooting image data and shooting metadata, and determining using the state of the moving object and preset conditions. Can do.

  In the image processing program according to the present invention, the state data is information obtained by communication with a sensor provided in the moving body or the outside, and the moving state of the moving body and the state of the environment of the moving body It is characterized by including at least one of these.

  According to this image processing program, appropriate shooting data can be used according to the moving state of the moving body and the environmental state.

  In the image processing program according to the present invention, the condition includes a precondition that is a condition of the state data and a photographing condition that is a condition of the photographing metadata, and the determination step includes the precondition that the state data is the premise. Judging whether or not a condition is satisfied, and when the state data satisfies the precondition, performing an operation using the shooting metadata satisfying the shooting condition and the shot image data corresponding to the shooting metadata It is characterized by.

  According to this image processing program, when the state of the moving body becomes an appropriate state set in advance, it is possible to use only photographing data appropriate for the state.

  Moreover, what applied the component of this invention, or arbitrary combinations of a component to a method, an apparatus, a system, a recording medium, a data structure, etc. is also contained in this invention.

  Further, the present invention is an image processing method for performing an operation using data transmitted from a camera in a moving body, which is captured image data that is an image captured by the camera and information regarding the state of the capturing A receiving step for receiving shooting data comprising metadata, a state acquiring step for acquiring state data that is information relating to the state of the mobile object, a preset condition, and the shooting metadata received by the receiving step; A determination step for determining whether to use the imaging data received in the reception step based on the state data acquired in the state acquisition step, and a determination that the mobile body uses the shooting data in the determination step And a calculation step of performing a calculation using the captured image data.

  According to this image processing method, when the camera transmits the captured image data and the captured metadata as the captured data, it is possible to select and use appropriate captured data in the moving body.

  According to the present invention, it is possible to use an appropriate image among images captured and transmitted externally.

  Embodiments of the present invention will be described below with reference to the drawings.

  First, the configuration of the image processing apparatus according to the present embodiment will be described.

  FIG. 1 is a block diagram showing an example of the configuration of the image processing apparatus according to the present embodiment. The image processing apparatus 1 is mounted on a vehicle, and includes a wireless communication unit 11, a shooting data storage unit 12, a condition data storage unit 13, a state data storage unit 14, a precondition determination unit 15, a shooting condition determination unit 16, and a calculation unit 17. The notification unit 18 is provided. A wireless camera 2 exists around the image processing apparatus 1. Note that the notification unit 18 may be provided outside the image processing apparatus 1. The image processing apparatus 1 is connected to a sensor 19 mounted on the same vehicle.

  The receiving unit corresponds to the wireless communication unit 11 in the embodiment. The state acquisition unit corresponds to the state data storage unit 14 in the embodiment. The determination unit corresponds to the condition data storage unit 13, the state data storage unit 14, the precondition determination unit 15, and the shooting condition determination unit 16 in the embodiment.

  The wireless camera 2 captures an image and transmits captured image data, which is a captured moving image, and imaging metadata representing the imaging space to the surroundings as imaging data. The wireless camera 2 may be a fixed camera installed at an intersection or the like, may be a moving camera installed in another vehicle, or may be a camera installed in the own vehicle. If the wireless camera 2 is a camera installed in the host vehicle, the transmission of the captured image data and the captured metadata is changed to a configuration that transmits to the captured data storage unit 12 not by wireless but by wire. It doesn't matter.

  The wireless communication unit 11 wirelessly receives shooting data from the nearby wireless camera 2 (S41). Note that the wireless communication unit 11 may not only simply receive the photographic data, but also perform a process of decoding the scrambled photographic data to limit the transmission destination if necessary. In this case, the wireless camera 2 transmits a scramble that can be released only to a specific user over the photographing data. That is, the wireless camera 2 can transmit shooting data that can be used only by a specific user in order to protect privacy or charge a service. Further, in a system using the wireless camera 2 and the image processing apparatus 1 mounted on the own vehicle, the wireless camera 2 is scrambled so that only the own vehicle can be used when it is not desired to use the captured data for other vehicles. It can be performed. In addition, in consideration of the installation cost of the wireless camera 2 on the road, when providing shooting data as a paid specific member service, the wireless camera 2 can be scrambled so that only the member can use it.

  The shooting data storage unit 12 stores shooting data received by the wireless communication unit 11. The shooting metadata includes, for example, the three-dimensional position (including height) of the wireless camera 2 for specifying the shooting space, the shooting direction (shooting azimuth angle (camera central axis direction), pan angle, tilt angle, etc.), camera Including the angle of view, focal length (field angle), imaging resolution (imaging element size, etc.) The shooting metadata is used to define a shooting space, for example, a three-dimensional position, a geodetic system (intersection local coordinate system, global coordinate system, new world coordinate system, old Japan coordinate system) and relative coordinates. The reference point information or the like may be attached. In addition to the information representing the shooting space, the shooting metadata includes shooting time, camera ID for distinguishing image frames from the same camera from image frames of other cameras, information on the camera installer and installation location, shooting It may include data such as codec type and frame rate of image data, type information such as whether it is a visible image or an infrared image, information indicating that the wireless camera 2 is mounted on the own vehicle, and the like.

  According to this shooting metadata, the vehicle that uses the shot image data acquires information that identifies the shooting space, unlike the conventional technology that transmits the position of the shot camera and the traveling direction of the car on which the shot camera is mounted. It can be used for image processing.

  The state data storage unit 14 stores state data, which is data obtained by collecting information relating to various current states that can be acquired, including running information of the vehicle (S46). Specifically, the state data is obtained by the sensor 19, and includes position information indicating the position and posture of the vehicle, vehicle control operation information such as a steering wheel operation and an accelerator brake operation by the driver, information on various in-vehicle sensors, current time Etc. Here, the position information may be acquired from GPS, or may be information in which the traveling direction, speed, position, and the like are calculated by image recognition of the in-vehicle camera.

  The state data may also be information obtained from an external database by communication of the wireless communication unit 11 such as weather around the vehicle, temperature, illuminance, map, road condition, traffic regulation, other vehicle information, etc. good. In addition, information on the result of image processing performed by the calculation unit 17 in the past, for example, alert information by image recognition, may be held as state data for feedback.

  The state data is mainly used in the precondition determining unit 15 to determine whether or not the current state is a state in which each image processing is performed. Further, the state data may be used for the image processing of the received photographing data in the calculation unit 17.

  According to the state data storage unit 14, the right to determine the change of the own vehicle state lies in the driver's explicit vehicle operation due to problems such as accident responsibilities, and the driving environment and By utilizing the fact that image processing is often used as a means for assisting whether the vehicle operation is appropriate, it is possible to detect the change of the own vehicle operation and control the processing timing of the image processing using the received image data. For example, when a vehicle equipped with image processing dedicated to right-hand accident detection goes straight through an intersection, it is not recommended to receive and process all moving image data from the wireless camera group installed at each intersection as infrastructure. Efficient. For this reason, the image processing apparatus according to the present embodiment limits wireless reception itself when turning right. When the wireless reception cannot be controlled, the image processing apparatus controls the sending of the wirelessly received shooting data to the calculation unit 17 as if the wireless acquisition of the shooting data itself was not possible. As described above, the image processing apparatus according to the present embodiment can efficiently perform image processing by using the operation state of the own vehicle for image processing.

  The condition data storage unit 13 holds condition data that is a definition of conditions for performing control of input of photographing data to the calculation unit 17 and the like. Specifically, there are preconditions and shooting conditions.

  The precondition is a condition for determining whether or not the current state is a state suitable for performing image processing. Based on the determination result of the precondition, it is determined whether to control the wireless communication unit 11 or to perform determination based on the imaging condition. Specifically, the current vehicle state, environmental state, previous image processing results, and the like.

  The shooting conditions are determined based on the preconditions that the current image processing is being performed, and then it is determined whether or not all of the shooting data actually received wirelessly is used for image processing or a part thereof. It is a condition for. Specifically, it is a condition that limits the information described in the shooting metadata.

  The condition data storage unit 13 is a database that registers and holds preconditions and shooting conditions. The precondition determination unit 15 and the imaging condition determination unit 16 refer to the condition data storage unit 13 and perform determination. If necessary, the calculation unit 17 can also refer to the condition data storage unit 13. Here, since it is highly possible that the precondition and the photographing condition are closely related to each other, one database is used. However, different databases may be used, and each determination unit individually holds these contents. Thus, the configuration of the independent condition data storage unit 13 is not necessary.

  Here, a specific example of the condition data will be described. FIG. 2 is an XML notation showing an example of the condition data according to the present embodiment. In this example, the condition data is a condition for selecting data necessary for a right turn, and is described using XML. The condition data may be in the form of a table, or other description data may be used as the description unit, name, value unit, etc. of the condition data.

  In this example, first, a coordinate system specification (GeoCoordinate) necessary for specifying an imaging condition is described (first line). Here, a coordinate system (MyCarCoordinate1) is designated in which the center of gravity of the vehicle is the origin, the traveling direction is the X-axis direction, the right-hand direction of the traveling direction is the Y-axis direction, and the unit is m. If the default coordinate system is determined, it is not necessary to specify it.

  Next, the precondition and the photographing condition are collectively described in a unit of service (Service) defined for each type of image processing performed by the calculation unit 17 (line 4). In this example, the number of services (Num) is one, and the service is defined with the name “confirm right turn” (line 5).

  As the definition of the service, first, in order to determine whether or not the corresponding image processing is necessary, preconditions (PreConditions) that are preconditions for starting the service are listed as a condition group (two conditions). Here, it is only necessary to satisfy either of the two conditions, and either judgment type (judge) is described as “alternative” (line 6). The content of Condition is described using the vehicle state. The first condition is a case where the right steering wheel of the vehicle is turned by 0.1 (relative value to a predetermined steering angle) or more (expressed as over + equal), and is described in a coordinate system in which the right steering wheel is a positive value. (7th line). The second condition is a case where the right winker of the own vehicle is operated once or more, and is described in a coordinate system in which the right winker is a positive value (11th line). The two conditions are described with a value related to the operation of the own vehicle as a unit of car operation (CarManipulate).

  Note that the precondition in this example is only the state of the vehicle. For example, if it is night now, the environmental state is measured by another measuring device, or if there is a complex intersection nearby. The environmental state etc. referring to other databases such as “” may be described. Further, if the service is to be always started as in forward monitoring, the precondition may be omitted.

  Next, a clear operation (Do) to be executed when the precondition is satisfied is listed. In this example, activation (start of wireless reception) of the wireless communication unit 11 (DataInputByAir) is described as a car operation (CarManipulate) (17th line). Actually, the wireless communication unit 11 may be continuously activated and may be omitted. Also, other operations may be described.

  Next, shooting conditions are conditions of shooting data to be used, and are described as an input filter group (InputFilters) (line 20). The determination based on the photographing condition is also an operation as a result of satisfying the precondition in a broad sense, but here, it is described separately from the operation (Do) in order to clearly indicate that the determination is performed.

  The input filter for right turn confirmation in this example consists of one filter (Filter), and the determination is made when the shooting space in the shooting data is at least partially within the designated area (enter) (line 21). Eye). The designated area is designated by a rectangle (Rectangle) using the host vehicle coordinate system, and a vehicle front area mainly indicating the right turn direction from the vehicle center axis is designated (line 22). The designated area may be designated as an ellipse. In addition, the input filter may be, for example, “a region having a width of 5 m on the right side with respect to an estimated route expected to travel in 20 seconds after the precondition is satisfied” or “a region having a radius of 10 m centering on a nearby intersection”. It may be described as a complex condition.

  Next, specific examples of other preconditions will be described.

  The more state data that can be used for determination, the more complex condition data can be defined using them. For example, it is possible to control the photographing data to be used by using preconditions such as “dark but heavy traffic” and “having many accidents in the past”.

  First, the definition of the precondition indicating “dark but heavy traffic” will be described. Here, “dark” means that the illuminance value measured by the own vehicle is below a predetermined threshold, and “a lot of traffic” means that the location of the own vehicle is a commercial area according to the use area designation in the map information Represents the case. FIG. 3 is an XML notation showing a first example of a complicated precondition according to the present embodiment. In this figure, the first line represents that it is determined that this precondition is satisfied when both of the two conditions are OK (all). The second line represents a case where the illuminance in the vehicle is equal to or less than the next Data value as the first condition. The third line indicates that the luminance is 30 lx (lux) as Data. The fifth line represents the case where the surrounding area type is the next Data value as the second condition. The sixth line indicates that, as Data, zoning (use area) is Commercial Zone (commercial area).

  Next, a definition of a precondition indicating “a number of accidents in the past” will be described. Here, information on the frequency of occurrence of traffic accidents investigated by the Ministry of Land, Infrastructure, Transport and Tourism or each police headquarters is included in the map information in advance. FIG. 4 is an XML notation showing a second example of a complicated precondition according to the present embodiment. The first line represents that it is determined that the precondition is satisfied when the next condition is OK. The second line represents, as Condition, that the frequency of traffic accidents in the map information is equal to or greater than the next Data value. The third line indicates that TAFR (Traffic Accident Frequency Rate) is 170 [cases / square km] as Data. In this example, the frequency of occurrence of traffic accidents is used, but detailed data such as child accidents, elderly accidents, motorcycle accidents, and the like exist, and these may be used.

  Next, specific examples of other imaging conditions will be described.

  First, a case where the codec type of the photographed image data is used for the photographing metadata and the photographing conditions will be described. FIG. 5 is an XML notation showing an example of shooting conditions using the codec type of the shot image data according to the present embodiment. In accordance with this shooting condition, the shooting condition determination unit 16 determines whether or not the codec type is “video / mpeg4”. The codec type is described using MIME-TYPE. When the shooting metadata satisfies this shooting condition, the shot image data can be decoded, and thus becomes an input to the calculation unit 17. On the other hand, if the shooting metadata does not satisfy this shooting condition, the shot image data cannot be decoded and is not used as an input to the calculation unit 17.

  In this XML notation, the second line represents a case where it matches the value of the next Data as a Filter. The third line indicates that codec is video / mpeg4 as Data.

  MIME-TYPE has a high affinity with XML and can represent various data types. However, since MIME-TYPE is processed as text, when it is necessary to compress wireless transmission / reception data, MIME-TYPE is compressed with a predefined binary encoded code, WBXML (Wireless Binary XML), or the like. You may use things.

  Next, a case where the frame rate of the photographed image data is used for photographing metadata and photographing conditions will be described. At this time, the calculation unit 17 can obtain the difference in the photographing time between frames from the frame rate. For example, when the calculation unit 17 acquires the difference between the moving speed of the wireless camera 2 and the shooting time, the difference (moving distance) of the shooting position of the moving wireless camera 2 can be calculated. The computing unit 17 can realize a stereo camera using the moving distance and two images before and after the moving.

  Next, a case where the image type of the photographed image data is used for the photographing metadata and photographing conditions will be described. FIG. 6 is an XML notation showing an example of shooting conditions using the image type of the shot image data according to the present embodiment. In accordance with the imaging conditions, the imaging condition determination unit 16 determines whether the image type is near infrared light (450 to 800 nm) or visible light. When the shooting metadata satisfies this shooting condition, the calculation unit 17 inputs. On the other hand, when the shooting metadata does not satisfy this shooting condition (such as a far-infrared image), it is not used as an input to the calculation unit 17.

  In this XML notation, the first line indicates that it is determined that this imaging condition is satisfied when at least one of the two filters is OK (alternative). The second line indicates that the first filter matches the next Data value. The third line indicates that the type of light (Ray) is infrared light (IR) as Data. The fourth line represents that the shortest wavelength (WLMin) is 450 nm as Data. The fifth line represents that the longest wavelength (WLMax) is 800 nm as Data. The eighth line indicates that the type of light (Ray) is visible light (Visible) as Data. By using such shooting conditions, the shooting condition determination unit 16 similarly uses or selects a normal visible light camera and an infrared light camera used at night as the wireless camera 2 to be used. It becomes possible.

  Next, a case will be described in which information indicating that the wireless camera 2 is mounted on the own vehicle is used as shooting metadata and shooting conditions. The plurality of wireless cameras 2 are mounted on the host vehicle and include information indicating that the camera is the host vehicle in shooting metadata for each frame of the shot image data. Therefore, the image processing apparatus 1 can use image data from the wireless camera 2 in the own vehicle, other vehicles, and the environment for image processing unified in the same framework.

  The shooting conditions in the condition data storage unit 13 are, for example, that the shooting data of the own vehicle camera is given higher priority for image processing than the shooting data of other cameras. By omitting this and always using it for image processing, it is possible to easily distinguish the shooting data of the own vehicle camera from the others and treat them specially. Therefore, it is possible to realize a system configuration that takes into account the importance of the shooting data of the own vehicle camera, which often captures the situation around the own vehicle. Thus, if the wireless cameras in the own vehicle, other vehicles, and the environment can be handled in a unified manner, the vehicle purchaser can later freely purchase or customize the wireless camera 2 of the own vehicle and customize it. Thus, it becomes possible to acquire more image processing materials more freely.

  Further, by leaving the selection of shooting data to the calculation unit 17, only preconditions for controlling only the timing of transferring the shooting data to the calculation unit 17 are described, and operations and inputs after the preconditions are satisfied. The filter group may not be described. In addition, as a complicated precondition specification, after the precondition is satisfied, the sensor installed in the vehicle is activated to grasp the detailed state, and the precondition determination is performed again using the data obtained from the sensor. As described above, the precondition and the subsequent operation may be nested, and may be described as a structure including the operation as a part of the precondition.

  The precondition determination unit 15 collates state data including the traveling information of the own vehicle with the condition data, and determines whether or not the current state matches the precondition (S42). The precondition determination unit 15 controls the wireless communication unit 11 according to the determination result, or controls other devices in the vehicle, for example, activation of the vehicle sensor necessary for more detailed current state determination. Or do.

  The shooting condition determining unit 16 compares shooting metadata included in the received shooting data with the condition data, and selects shooting data to be used for image processing (S43). For example, in the case of the condition data in FIG. 2, the imaging condition determination unit 16 checks whether there is an operation to turn the steering wheel to the right or a turn signal right turn instruction in the current state of the host vehicle included in the state data, and assumes a right turn confirmation service. It is assumed that the condition (PreConditions) is satisfied, and the position area (area) that is likely to be related to the right turn described as the shooting condition (InputFilters) is obtained using the current position and traveling speed of the host vehicle.

  Next, the shooting condition determination unit 16 aligns the coordinate system so that the shooting space in the shooting metadata wirelessly received by the wireless communication unit 11 can be compared with the position area (area), and the shooting space is within the position area. By checking whether or not the data is included, it is determined whether or not the photographing data satisfies the photographing condition, and only the data determined to satisfy the photographing data is sent to the calculation unit 17.

  Next, detailed operations of the precondition determination unit 15 and the imaging condition determination unit 16 will be described.

  FIG. 7 is a flowchart illustrating an example of detailed operations of the precondition determination unit 15 and the imaging condition determination unit 16 according to the present embodiment. Here, description will be made using the use condition data of FIG.

  First, the precondition determining unit 15 determines whether or not there is the next ServiceData in the condition data (S11). If there is no next ServiceData (S11, NO), this flow ends. On the other hand, when there is the next ServiceData (S11, YES), the precondition determining unit 15 determines whether or not PreCondition (precondition) is present in the ServiceData (S12). When there is no PreCondition (S12, NO), the process proceeds to S14. On the other hand, when there is PreCondition (S12, YES), the precondition determining unit 15 determines whether or not PreCondition is satisfied (S13). In the case of this example of condition data, the precondition determining unit 15 satisfies one of the condition groups in PreCondition (the right steering wheel has an operation angle of 0.1 operation unit or more and the right winker is operated). It is determined whether or not (alternative).

  When PreCondition is not satisfied (S13, NO), the process proceeds to S11. On the other hand, when PreCondition is satisfied (S13, YES), the precondition determining unit 15 determines whether there is Do (execution item) (S14). If there is no Do (S14, NO), the process proceeds to S21. On the other hand, when there is Do (S14, YES), the precondition determination unit 15 executes the contents of Do (activates the wireless communication unit of the vehicle) (S15).

  Next, the shooting condition determination unit 16 determines whether the wireless communication unit 11 is in operation and there is a next frame of the received shot image data (S21). When the wireless communication unit 11 is not in operation or there is no next frame (S21, NO), the process proceeds to step S11. On the other hand, when the wireless communication unit 11 is in operation and there is a next frame (S21, YES), the imaging condition determination unit 16 acquires the next frame (S22), and whether there is an InputFilter (imaging condition). Is determined (S23).

  When there is no InputFilters (S23, NO), the process proceeds to S27. On the other hand, when InputFilters are present (S23, YES), the shooting condition determination unit 16 takes a shooting camera ID representing the wireless camera 2 that shot the corresponding frame (when the shooting metadata is included in the shooting metadata of the corresponding frame). It is determined whether or not the shooting conditions have already been determined (S24). When it is not determined or when the photographing camera ID is unknown (S24, NO), the process proceeds to S26. On the other hand, if it is determined (S24, YES), the imaging condition determination unit 16 determines whether or not the determination result is OK (the imaging condition is satisfied) (S25).

  Here, the wireless camera 2 includes its own camera ID in the shooting metadata for each frame of the shot image data. By attaching a camera ID indicating that the image is captured by the same wireless camera 2 as shooting metadata for each frame, the shooting condition determination unit 16 can omit data from a fixed camera that is obviously not used. it can. Further, when receiving photographing data from a fixed camera in which the position of the own vehicle does not change (such as when stopped) and the photographing space does not change, the photographing condition determination unit 16 only needs to make a determination using the photographing space once. Thereafter, calculation can be omitted by reusing the determination result. Therefore, it is possible to reduce the cost of determination and calculation by including the camera ID in the shooting metadata.

  If it is NG (S25, NO), the process proceeds to S21. On the other hand, if the process is OK (S25, YES), the process proceeds to process S27. The shooting condition determination unit 16 determines whether or not the corresponding frame satisfies InputFilters (S26). In the case of this example of condition data, a filter group in InputFilters (a part of the shooting space of the corresponding frame is within an area of 30 m × 20 m on the front right side on the own vehicle coordinate system MyCarCoord1 having the current vehicle center of gravity position as the origin. It is determined whether all (all) are satisfied. At the same time, if the determination result is clearly applicable to the shooting data of the same camera, the camera ID is stored as determined.

  When InputFilters are not satisfied (S26, NO), the process proceeds to S21. On the other hand, when the InputFilters are satisfied (S26, YES), the imaging condition determination unit 16 passes the corresponding frame to the calculation unit 17 (S27), and proceeds to processing S11.

  Next, an example of specific operations of the precondition determination unit 15 and the shooting condition determination unit 16 will be described.

  FIG. 8 is a plan view showing a first example of an intersection according to the present embodiment. This figure shows a case where wireless cameras 2a and 2b are installed at each intersection, photographing the photographing spaces 22a and 22b, respectively, and wirelessly transmitting them as photographing data.

  First, the precondition determination unit 15 detects the right turn motion of the host vehicle from the steering wheel operation or the winker operation, confirms that the preconditions of FIG. 2 are satisfied, and detects the start of the right turn confirmation service. Next, the shooting condition determination unit 16 calculates a rectangular area having the detected position as the origin, that is, the shooting condition area 21 shown in FIG. 8, which is the shooting condition of the shooting data for right turn confirmation from the detected current position. Here, it is assumed that the wireless communication unit 11 receives the respective shooting data from the wireless cameras 2a and 2b. The shooting condition determination unit 16 geometrically calculates whether or not the shooting spaces 22a and 22b included in the shooting metadata enter the shooting condition area 21, respectively, determines that only the shooting space 2b by the wireless camera 2b is applicable, and wirelessly. The photographing data from the camera 2b is passed to the calculation unit 17.

  Next, the calculation unit 17 will be described.

  The calculation unit 17 performs a predetermined calculation using the shooting data passed from the shooting condition determination unit 16 (S44), passes the calculation result to the notification unit 18, and notifies the driver (S45). The notification unit 18 notifies the driver by displaying the display image passed from the calculation unit 17 or reproducing a warning sound. Here, an example of image processing will be described.

  FIG. 9 is a plan view showing a second example of the intersection according to the present embodiment. The vehicle 3 in this figure is going to make a right turn at the intersection. In addition, there are other vehicles 4 a and 4 b as oncoming vehicles of the host vehicle 3. It is assumed that the other vehicle 4a exists in the blind spot of the other vehicle 4b for the own vehicle 3 and cannot be directly recognized by the driver of the own vehicle 3. In addition, a wireless camera 2c facing directly downward and a wireless camera 2d facing diagonally downward were installed above the intersection (the position of the traffic light, etc.) shown in this figure, and obtained by photographing the photographing spaces 22c and 22d, respectively. Each frame 23c, 23d is wirelessly transmitted as shooting data. FIG. 10 is an image showing an example of the frame 23c according to the present embodiment. FIG. 11 is an image showing an example of the frame 23d according to the present embodiment.

  The calculation unit 17 converts the frame 23d photographed from obliquely above into a frame 24d photographed from directly above. First, the calculation unit 17 performs geometric transformation (affine transformation) on the frame 23d using the frames of the captured image data received from the wireless cameras 2c and 2d and the captured metadata (for example, the elevation angle of the camera) to obtain the target view. A frame 24d that is an overhead image from right above the intersection is generated as (an image viewed from a predetermined position and direction). FIG. 12 is an image showing an example of the frame 24d according to the present embodiment. Since the frame 23c photographed by the wireless camera 2c is already an overhead image from directly above, only the frame 23d photographed by the wireless camera 2d is converted. In addition, the calculating part 17 may perform still object removal (it uses the existing methods, such as taking the difference between several frames, taking a background difference) of the frames 23c and 24d.

  Next, the computing unit 17 performs size conversion and rotation of the frames 23c and 24d, superimposes them on the map screen in an appropriate order, generates a display screen, and causes the notification unit 18 to display the display screen. FIG. 13 is an image showing an example of the map screen. Here, the arrows represent the position and direction of the vehicle 3. Here, the calculation unit 17 superimposes on the map screen in order from the frame in which the shooting spaces 22c and 22d are large (the zoom magnification of the wireless cameras 2c and 2d is low), and the distance from the own vehicle 3 to the shooting spaces 22c and 22d is short. Processing such as superimposing on the map screen in order from the frame is performed. FIG. 14 is an image showing an example of a display screen according to the present embodiment. In this figure, the calculation unit 17 superimposes the frames 23c and 24d on the map screen together with the background, but performs the above-mentioned stationary object removal, superimposes only the moving object, superimposes the frames 23c and 24d as a transmission image, etc. Processing may be performed.

  Next, overhead image conversion, which is processing for converting an image viewed obliquely from above into an image viewed from directly above (processing for converting frame 23d to frame 24d), will be described.

  FIG. 15 is a diagram illustrating an example of the overhead image conversion according to the present embodiment. FIG. 16 is an equation showing an example of the overhead image conversion according to the present embodiment. The calculation unit 17 indicates that the point (x, y) on the image looking down at the horizontal depression angle θ from the wireless camera 2g (view angle θv and focal length f) at the height H in FIG. And a point at the position (X, Y) of the road surface as shown in the fifth equation. FIG. 17 is an image showing an example of a pre-conversion image of the overhead view image conversion according to the present embodiment. FIG. 18 is an image showing an example of the converted image of the overhead view image conversion according to the present embodiment. All the pixel points in the frame photographed from diagonally above as shown in FIG. 17 can be converted into points in a trapezoidal area which is an image seen from directly above as shown in FIG.

  As described above, the calculation unit 17 processes and displays the shooting data of the wireless camera 2 existing around the host vehicle based on the shooting space and the like, so that the driver can Can be recognized quickly.

  In the present embodiment, the calculation unit 17 superimposes a plurality of images, but only one image may be used as a display image. In addition, the calculation unit 17 may perform image processing other than affine transformation. In addition, when multiple pieces of shooting data shot in the same shooting space are received at the same time, shooting that selects appropriate shooting data, such as using shot image data that shows a fine part based on shooting metadata for image processing Conditions may be defined.

  Moreover, the calculating part 17 may perform a danger detection process using the imaging | photography data passed from the imaging condition determination part 16, and may notify a driver | operator. Here, an example of the danger detection process will be described.

  FIG. 19 is a plan view showing a third example of the intersection according to the present embodiment. The vehicle 3 in this figure is going to make a right turn at the intersection. In addition, there are other vehicles 4a, 4b, 4e as oncoming vehicles of the host vehicle 3. It is assumed that the other vehicle 4a exists in the blind spot of the other vehicle 4b for the own vehicle 3 and cannot be directly recognized by the driver of the own vehicle 3. A triangle extending from the host vehicle 3 indicates a viewing area 25 by the driver of the host vehicle 3 and a blind spot 26 in the viewing area 25. FIG. 20 is an image showing an example of the visual recognition result 27 of the visual recognition area 25 according to the present embodiment. In this visual recognition result 27, only the other vehicle 4b is visually recognized, and the other vehicle 4a is not visually recognized. In addition, the other vehicle 4e is equipped with the wireless camera 2e, and is present at a position where the wireless camera 2e can photograph the other vehicles 4a and 4b. Moreover, the triangle extended from the other vehicle 4e shows the imaging | photography space 22e image | photographed with the wireless camera 2e. FIG. 21 is an image showing an example of the frame 23e obtained from the imaging space 22e according to the present embodiment. In this frame 23e, both the other vehicles 4a and 4b are photographed.

  The image processing apparatus 1 of the own vehicle 3 receives the shooting data transmitted by the wireless camera 2e. The computing unit 17 first detects an object in the received frame 23e. Here, the calculation unit 17 recognizes an image using, for example, characteristics of a tire of a car in the frame 23e. For example, when using a plurality of continuous frames from the same wireless camera 2, the calculation unit 17 compares the image flows and detects a moving object.

  As a result, the other vehicles 4a and 4b are detected in the frame 23e. Using this detection result and shooting metadata from the wireless camera 2e (camera direction and angle of view of the wireless camera 2e), the calculation unit 17 can detect an object existence possibility area that is an area where an object may exist. To decide. FIG. 22 is a plan view showing an example of the object existence possibility area according to the present embodiment. Object existence possibility areas 28a and 28b are determined in the imaging space 22e. In addition, the screen position 29e of the frame 23e is shown in the shooting space 22e. The object existence possibility areas 28a and 28b are areas projected onto the positions of the other vehicles 4a and 4b at the screen position 29e. Here, the depth (range of distance from the wireless camera 2e) of the object existence possibility regions 28a and 28b is set to infinity, but may be a finite value in consideration of the actual effective resolution and the subject depth. Further, the calculation unit 17 sets an area where the visual recognition area 25 of the host vehicle 3 and the object existence possibility area 28a overlap as an intersection area 30a, and an area where the visual recognition area 25 of the host vehicle 3 overlaps with the object existence possibility area 28b as an intersection area. 30b.

  Next, the calculation unit 17 determines whether or not the intersection areas 30 a and 30 b are in a dangerous range in the travel of the host vehicle 3. For example, the calculation unit 17 sets conditions such that the distance from the own vehicle 3 to the intersection areas 30a and 30b is closer than a predetermined threshold, and a part of the intersection areas 30a and 30b overlaps with an area related to the progress of the own vehicle 3. If so, notify the driver of the danger.

  Conventionally, an image taken by a camera other than the own vehicle has been used only for display on the own vehicle. Moreover, the image for detecting the danger was only an image taken with the own vehicle. According to this danger detection process, it is possible to use shooting data from the wireless camera 2 installed in the own vehicle, another vehicle, the environment, etc., so that the accuracy of detecting the danger can be improved.

  The wireless camera 2 may control the shooting data to be transmitted using the information that can be acquired and preset conditions. The information that can be acquired includes, for example, information from sensors and GPS installed in the wireless camera 2, traveling state such as the position, posture, traveling direction, and speed of the wireless camera 2 by image recognition, road conditions obtained by communication, It includes the surrounding brightness, weather, surrounding map, applicable regulations, surrounding vehicle status, and wireless communication status of other cameras. Further, the wireless camera 2 determines the presence / absence of photographing, the photographing range, the frame rate, the photographing data to be transmitted, the moving image codec, the presence / absence of wireless transmission, the wireless transmission strength, and the like based on the information that can be acquired.

  According to the wireless camera 2, for example, a wireless camera that captures and transmits wirelessly only at night (detects night from the time or surrounding luminance information), or a wireless camera that transmits a fine video only when a surrounding vehicle is detected (wireless It is possible to detect the surrounding vehicle by a sensor installed in the camera, or obtain the surrounding vehicle information from a database collecting the position reports from the surrounding vehicle itself to detect the surrounding vehicle). Therefore, the wireless camera 2 can efficiently transmit shooting data.

  Note that all the components of the image processing apparatus 1 may not be present in one vehicle. Although the immediacy is lost, the calculation unit 17 may be in an external device via the network, and the image processing apparatus may transmit the shooting data to the calculation unit 17 via the network and receive the result via the network. . In addition, the status data may be stored in a separate location divided into a plurality of data for each fine item, or stored as a database by inquiring data values from an external sensor or the like when an image processing apparatus is required. It is not necessary. Further, the image processing apparatus may be mounted on a moving body other than the vehicle.

  In the prior art, when trying to receive and use an image taken by a surrounding road camera (fixed surveillance, car, etc.) for the purpose of avoiding danger, the exact shooting space is not known and cannot be processed or used. There has been a problem that the processing cost (processing load, processing time) for the transmission side to provide the processed image is too high.

  In the present embodiment, a camera that wirelessly transmits captured image data and captured metadata as captured data, and received captured data transmitted from the vicinity, and the camera and image are captured using the captured metadata and its own state. An image processing apparatus for selective use is provided. Accordingly, since the transmission side transmits the photographing metadata together with the photographing image data, it is possible to freely process and use the image on the receiving side. Further, since processing is performed on the receiving side, processing cost on the transmitting side is reduced, and load distribution can be realized.

  Furthermore, since the transmitting side can concentrate on providing materials called moving images, it is possible to develop a transmission camera on the road as infrastructure without considering the responsibility for accidents caused by errors in the processing algorithm. Increase the feasibility of installation. In addition, by applying the present invention to the wireless camera installed in the vehicle and the service in the vehicle, the number of the on-vehicle cameras in the vehicle, the installation position, and the shooting direction can be freely customized by the individual. Service can be realized.

  Note that the transmission step corresponds to the operation of the wireless camera 2 in the embodiment. The reception step corresponds to step S41 in the embodiment. The state acquisition step corresponds to step S46 in the embodiment. The determination step corresponds to the processes S42 and S43 in the embodiment. The calculation step corresponds to step S44 in the embodiment.

  Further, the image processing apparatus according to the present embodiment can be easily applied to the information processing apparatus, and can further improve the performance of the information processing apparatus. Here, the information processing apparatus may include, for example, a car navigation system, an ECU (Electric Control Unit), and the like.

  Furthermore, it is possible to provide a program that causes a computer constituting the image processing apparatus to execute the above steps as an image processing program. By storing the above-described program in a computer-readable recording medium, the computer constituting the image processing apparatus can be executed. Here, examples of the recording medium readable by the computer include an internal storage device such as a ROM and a RAM, a portable storage such as a CD-ROM, a flexible disk, a DVD disk, a magneto-optical disk, and an IC card. It includes a medium, a database holding a computer program, another computer and its database, and a transmission medium on a line.

(Supplementary Note 1) An image processing program for causing a computer provided in a moving body to execute calculation of data transmitted from the outside,
A reception step of receiving shooting data consisting of shooting image data that is an image shot by an external camera and shooting metadata that is information about a state in which the shooting image data was shot;
A state acquisition step of acquiring state data which is information relating to the state of the mobile body;
Judgment as to whether or not to use the shooting data received in the reception step based on preset conditions, the shooting metadata received in the reception step and the state data acquired in the state acquisition step. A determination step to be performed;
An image processing program that causes a computer to execute an operation step that performs an operation using imaging data determined to be used in the determination step.
(Appendix 2) In the image processing program described in Appendix 1,
The image processing program characterized in that the arithmetic unit converts photographed image data in the photographed data based on the photographed metadata in the photographed data.
(Appendix 3) In the image processing program described in Appendix 1 or Appendix 2,
The image processing program, wherein the calculation unit detects a danger based on the photographing data.
(Supplementary Note 4) In the image processing program according to any one of Supplementary Notes 1 to 3,
The state data is information obtained by communication with a sensor provided in the moving body or the outside, and includes at least one of a moving state of the moving body and an environmental state of the moving body. Image processing program.
(Supplementary Note 5) In the image processing program according to any one of Supplementary Notes 1 to 4,
The condition includes a precondition that is a condition of the state data and a shooting condition that is a condition of the shooting metadata.
The determination step determines whether or not the state data satisfies the precondition, and if the state data satisfies the precondition, corresponds to the shooting metadata and the shooting metadata that satisfy the shooting condition. An image processing program for performing calculation using photographed image data.
(Appendix 6) In the image processing program described in Appendix 5,
The shooting metadata includes shooting space information that is information about a space shot by the camera as the shot image data.
The image processing program characterized in that the shooting condition includes a condition of the shooting space information in shooting data to be used.
(Appendix 7) In the image processing program described in Appendix 5 or Appendix 6,
The state data includes information on user operations on the mobile object,
The image processing program characterized in that the precondition includes a predetermined operation.
(Supplementary note 8) In the image processing program described in supplementary note 5,
The image processing program characterized in that the shooting condition includes that an ID of a camera included in the shooting metadata is an ID of a camera provided in the moving body.
(Supplementary note 9) In the image processing program according to any one of supplementary notes 5 to 8,
The shooting metadata includes an image type,
The image processing program characterized in that the photographing condition includes a condition of a type of image to be used.
(Additional remark 10) It is an image processing apparatus which is provided in a mobile body and performs the calculation of the data transmitted from the outside,
A receiving unit that receives shooting data including shooting image data that is an image shot by an external camera and shooting metadata that is information about a state in which the shooting image data is shot;
A state acquisition unit that acquires state data that is information relating to the state of the mobile body;
Judgment as to whether or not to use shooting data received by the receiving unit based on preset conditions, shooting metadata received by the receiving unit and state data acquired by the state acquiring unit A determination unit to perform;
An image processing apparatus comprising: an arithmetic unit that performs an operation using photographing data determined to be used by the determination unit.
(Supplementary note 11) In the image processing apparatus according to supplementary note 10,
The image processing apparatus, wherein the arithmetic unit converts photographed image data in the photographing data based on the photographing metadata in the photographing data.
(Supplementary Note 12) In the image processing apparatus according to Supplementary Note 10 or Supplementary Note 11,
The image processing apparatus, wherein the calculation unit detects a danger based on the photographing data.
(Supplementary note 13) In the image processing device according to any one of supplementary notes 10 to 12,
The state data is information obtained by communication with a sensor provided on the moving body or the outside, and includes at least one of a moving state of the moving body and an environmental state of the moving body. Image processing device.
(Supplementary Note 14) In the image processing apparatus according to any one of Supplementary Notes 10 to 13,
The condition includes a precondition that is a condition of the state data and a shooting condition that is a condition of the shooting metadata.
The determination unit determines whether or not the state data satisfies the precondition, and if the state data satisfies the precondition, corresponds to the shooting metadata and the shooting metadata that satisfy the shooting condition. An image processing apparatus that performs a calculation using photographed image data.
(Supplementary Note 15) In the image processing apparatus according to Supplementary Note 14,
The shooting metadata includes shooting space information that is information about a space shot by the camera as the shot image data.
The image processing apparatus characterized in that the shooting condition includes a condition of the shooting space information in shooting data to be used.
(Supplementary Note 16) An image processing method for performing a calculation using data transmitted from a camera in a moving body,
A receiving step of receiving shooting data including shooting image data that is an image shot by the camera and shooting metadata that is information relating to the shooting state;
A state acquisition step of acquiring state data which is information relating to the state of the mobile body;
Judgment as to whether or not to use the imaging data received in the reception step based on preset conditions, the imaging metadata received in the reception step and the status data acquired in the status acquisition step. A determination step to be performed;
An image processing method for executing, in the moving body, a calculation step of performing calculation using photographing data determined to be used in the determination step.
(Supplementary note 17) In the image processing method according to supplementary note 16,
The received photographing data is data obtained by controlling the photographing of the photographing image data or the transmission of the photographing data based on information obtained by communication with a sensor installed in the camera or the outside. An image processing method comprising:

It is a block diagram which shows an example of a structure of the image processing apparatus which concerns on this Embodiment. It is XML description which shows an example of the condition data which concern on this Embodiment. It is XML description which shows the 1st example of the complicated precondition which concerns on this Embodiment. It is XML description which shows the 2nd example of the complicated precondition which concerns on this Embodiment. It is XML description which shows an example of the imaging | photography conditions using the codec classification of the captured image data which concerns on this Embodiment. It is XML description which shows an example of the imaging conditions using the image classification of the captured image data which concerns on this Embodiment. It is a flowchart which shows an example of detailed operation | movement of the precondition determination part 15 and the imaging condition determination part 16 which concern on this Embodiment. It is a top view which shows the 1st example of the intersection which concerns on this Embodiment. It is a top view which shows the 2nd example of the intersection which concerns on this Embodiment. It is an image which shows an example of the flame | frame 23c which concerns on this Embodiment. It is an image which shows an example of the frame 23d which concerns on this Embodiment. It is an image which shows an example of the flame | frame 24d which concerns on this Embodiment. It is an image which shows an example of a map screen. It is an image which shows an example of the display screen which concerns on this Embodiment. It is a figure which shows an example of the bird's-eye view image conversion which concerns on this Embodiment. It is a formula which shows an example of bird's-eye view image conversion concerning this embodiment. It is an image which shows an example of the image before conversion of the bird's-eye view image conversion which concerns on this Embodiment. It is an image which shows an example of the image after conversion of the overhead image conversion which concerns on this Embodiment. It is a top view which shows the 3rd example of the intersection which concerns on this Embodiment. It is an image which shows an example of the visual recognition result 27 of the visual recognition area | region 25 which concerns on this Embodiment. It is an image which shows an example of the flame | frame 23e obtained from the imaging | photography space 22e which concerns on this Embodiment. It is a top view which shows the example of the object presence possibility area | region which concerns on this Embodiment. It is a top view which shows an example of a right-handed accident. It is an image which shows an example of the visual recognition result seen from the own vehicle at the time of a right turn.

Explanation of symbols

1 image processing device, 2, 2a, 2b, 2c, 2d, 2e, 2g wireless camera, 3 vehicle, 4a, 4b, 4c, 4d, 4e other vehicle, 11 wireless communication unit, 12 photographing data storage unit, 13 conditions Data storage unit, 14 state data storage unit, 15 precondition determination unit, 16 shooting condition determination unit, 17 calculation unit, 18 notification unit, 19 sensor, 21 shooting condition area, 22a, 22b, 22c, 22d, 22e shooting space, 23c, 23d, 23e frame, 25 viewing area, 26 blind spot, 27 viewing result, 28a, 28b object existence possibility area, 29e screen position, 30a, 30b intersecting area.

Claims (5)

An image processing program for causing a computer provided in a moving body to execute calculation of data transmitted from the outside,
A reception step of receiving shooting data consisting of shooting image data that is an image shot by an external camera and shooting metadata that is information about a state in which the shooting image data was shot;
A state acquisition step of acquiring state data which is information relating to the state of the mobile body;
Judgment as to whether or not to use the shooting data received in the reception step based on preset conditions, the shooting metadata received in the reception step and the state data acquired in the state acquisition step. A determination step to be performed;
An image processing program that causes a computer to execute an operation step that performs an operation using imaging data determined to be used in the determination step. In the image processing program according to claim 1,
The state data is information obtained by communication with a sensor provided in the moving body or the outside, and includes at least one of a moving state of the moving body and an environmental state of the moving body. Image processing program. The image processing program according to claim 1 or 2,
The condition includes a precondition that is a condition of the state data and a shooting condition that is a condition of the shooting metadata.
The determination step determines whether or not the state data satisfies the precondition, and if the state data satisfies the precondition, corresponds to the shooting metadata and the shooting metadata that satisfy the shooting condition. An image processing program for performing calculation using photographed image data. An image processing apparatus that is provided in a moving body and performs calculation of data transmitted from the outside,
A receiving unit that receives shooting data including shooting image data that is an image shot by an external camera and shooting metadata that is information about a state in which the shooting image data is shot;
A state acquisition unit that acquires state data that is information relating to the state of the mobile body;
Judgment as to whether or not to use shooting data received by the receiving unit based on preset conditions, shooting metadata received by the receiving unit and state data acquired by the state acquiring unit A determination unit to perform;
An image processing apparatus comprising: an arithmetic unit that performs an operation using photographing data determined to be used by the determination unit. An image processing method for performing an operation using data transmitted from a camera in a moving body,
In the camera, a transmission step of shooting and transmitting shooting data including shooting image data that is an image obtained by the shooting and shooting metadata that is information relating to the shooting state;
In the mobile body, a reception step of receiving the imaging data transmitted in the transmission step;
In the moving body, a state acquisition step of acquiring state data that is information on the state of the moving body;
Whether to use the imaging data received in the reception step based on preset conditions, imaging metadata received in the reception step, and status data acquired in the status acquisition step in the mobile body A determination step for determining whether or not,
An image processing method for executing, in the moving body, an operation step of performing an operation using imaging data determined to be used in the determination step.

Images