JP2020184170A - Image analyzer, electronic controller, and image analysis method - Google Patents

Abstract

To enable specifying time when image data is photographed, from the image data even though a frame defect is present during transmission of information.SOLUTION: An image analyzer 100 comprises a transmission unit 101 for transmitting a light emission command of a light emission pattern, and light emission time of the light emission pattern to an electronic controller installed in a moving body; an acquisition unit 106 for acquiring image data of the moving body including the light emission pattern from an imaging apparatus that photographs the moving body; a pattern detection unit 107 for detecting the light emission pattern from the image data; and a photographing time specification unit 108 for setting the light emission time to time when the image data is photographed.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image analysis device, an electronic control device, and an image analysis method executed by these devices.

In recent years, monitoring and management systems using network cameras and the like have been introduced for various purposes. For traffic purposes, it is used for the purpose of monitoring such as signal ignoring monitoring and speed violation monitoring, for the purpose of managing road operation management, toll collection management, etc., as well as for the purpose of acquiring information such as traffic congestion information acquisition and vehicle identification information acquisition. Has been done. Furthermore, the images obtained by the camera are expected to be used for driving support of autonomous vehicles. In order to strengthen the ability to respond to disasters and accidents, promote visualization of road conditions to improve safety and security, and realize autonomous driving, it is required to expand the visible range and manage the entire road network. ..

Patent Document 1 describes a network camera system that realizes time synchronization between video information. A network camera system includes a plurality of camera devices and lighting devices connected to each other by a network. Specifically, the illumination device irradiates the irradiation light including the coded information associated with the reference time information, and a plurality of camera devices capture the reflected light of the illumination light to take a picture of the monitored area. It is disclosed that the obtained video information is analyzed to extract the reference time information, and that the video information is encoded in association with the reference time information.

Japanese Unexamined Patent Publication No. 2012-124574

By the way, the present inventor has found the following problems.
In the video information time synchronization method of Patent Document 1, since the reflected light itself contains the reference time information, the reference time information cannot be restored when a frame is lost or the like. Further, the video information time synchronization method of Patent Document 1 cannot be used with a camera that cannot capture infrared rays.

Therefore, an object of the present invention is an image analysis device, an electronic control device, and an image analysis method executed by these, which make it possible to specify the time when the image data is taken even if there is a frame defect at the time of information transmission. Is to provide.

In order to solve the above-mentioned problems, the image analysis device (100) of the present disclosure transmits a light emission command of a light emission pattern and a light emission time of the light emission pattern to an electronic control device (201) mounted on a moving body (200). The transmission unit (101), the acquisition unit (106) that acquires the image data of the moving body including the light emitting pattern from the photographing apparatus that photographed the moving body, and the pattern that detects the light emitting pattern from the image data. It has a detection unit (107) and a shooting time specifying unit (108) in which the light emission time is the time when the image data is shot.

Further, the electronic control device (201) mounted on the mobile body (200) of the present disclosure includes a receiving unit (204) that receives a light emitting command of a light emitting pattern and a light emitting time of the light emitting pattern from an image analysis device, and the above. It has a light emitting instruction unit (207) that instructs a light source (206) mounted on the moving body to emit light using the light emitting pattern at a light emitting time.

The numbers in parentheses attached to the scope of claims and the constituent requirements of the invention described in this section indicate the correspondence between the present invention and the embodiments described later, and are not intended to limit the present invention. ..

According to the image analysis device, the electronic control device, and the image analysis method executed by these, it is possible to specify the time when the image data was taken even if there is a frame defect at the time of information transmission.

The figure explaining the outline of the image analysis system common to each embodiment of this invention. A sequence diagram illustrating the operation of the image analysis system common to each embodiment of the present invention. A block diagram illustrating the configuration of the image analysis apparatus according to the first embodiment of the present invention. The figure explaining the light emission pattern generated by the image analysis apparatus of Embodiment 1 of this invention, and the light emission time of the light emission pattern. The figure explaining the image data acquired by the image analysis apparatus of Embodiment 1 of this invention. The figure explaining the light emission pattern detected by the image analysis apparatus of Embodiment 1 of this invention. The figure explaining the method of specifying the time at which image data was taken and the delay time in the image analysis apparatus of Embodiment 1 of this invention. A flowchart illustrating the operation of the image analysis apparatus according to the first embodiment of the present invention. A flowchart illustrating the operation of the image analysis apparatus of the first modification of the first embodiment of the present invention. A flowchart illustrating the operation of the image analysis apparatus according to the second embodiment of the present invention. The figure explaining the light emission pattern generated by the image analysis apparatus of Embodiment 3 of this invention. The figure explaining another light emission pattern generated by the image analysis apparatus of Embodiment 3 of this invention. A block diagram illustrating the configuration of the image analysis apparatus according to the fourth embodiment of the present invention. A flowchart illustrating the operation of the image analysis apparatus according to the fourth embodiment of the present invention. A block diagram illustrating the configuration of the electronic control device according to the fifth embodiment of the present invention. A flowchart illustrating the operation of the electronic control device according to the fifth embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

The present invention means the invention described in the section of the scope of claims or means for solving the problem, and is not limited to the following embodiments. In addition, at least the words in brackets mean the words and phrases described in the section of claims or means for solving the problem, and are not limited to the following embodiments.

The configuration and method described in the dependent clause of the claims, the configuration and method of the embodiment corresponding to the configuration and method described in the dependent clause, and the configuration and method described only in the embodiment without description in the claims. Is an arbitrary configuration and method in the present invention. In the present invention, any configuration and method may be used in the sense that the configuration and method described in the embodiment when the claims are broader than the description of the embodiment are also examples of the configuration and method of the present invention. is there. In either case, the description in the independent clause of the claims provides an essential configuration and method of the present invention.

The effects described in the embodiments are effects in the case of having the configuration of the embodiment as an example of the present invention, and are not necessarily the effects of the present invention.

When there are a plurality of embodiments, the configuration disclosed in each embodiment is not closed only in each embodiment, but can be combined across the embodiments. For example, the configuration disclosed in one embodiment may be combined with another embodiment. Further, the disclosed configurations may be collected and combined in each of the plurality of embodiments.

The findings and issues described in the issues to be solved by the invention are not known issues, but are independently discovered by the present inventor, and are facts that affirm the inventive step of the invention together with the constitution and method of the present invention.

Each of the following embodiments will be described mainly by using an apparatus for executing an image analysis method of image data obtained by photographing a vehicle as an example, but the present invention photographs a moving body other than the vehicle unless there is a limitation within the scope of the patent claim. It also includes an apparatus for executing an image analysis method of the image data.

(Outline of image analysis system common to each embodiment)
First, the outline of the image analysis system 10 will be described with reference to FIG.

The image analysis system 10 includes an image analysis device 100 and an electronic control device 201 mounted on the vehicle 200. The image analysis system 10 may further include a camera 300.

The camera 300 photographs the vehicle 200 and transmits the image data of the vehicle 200 to the image analysis device 100 via the communication network 30.
The camera 300 may be an infrastructure camera, a security camera, a live camera attached to a road or a building around the road, or a drive recorder attached to a vehicle.
As the camera 300, in addition to a network camera to which an IP address is assigned and communication is performed by the camera itself, a camera that does not communicate by the camera itself but communicates via a device that can use a communication network is also assumed.
The camera 300 may be capable of capturing an image in which visible light is combined, as well as an image in which infrared light or ultraviolet light is combined.

The vehicle 200 is equipped with an electronic control device 201. The electronic control device 201 can communicate with the image analysis device 100 and transmit and receive necessary information via the network 20. The vehicle 200 may be a driver-driven vehicle operated by the driver or an automatically driven vehicle operated by the system according to each level.

The image analysis device 100 collects, analyzes, and manages image data from a plurality of cameras including the camera 300. For example, based on the image data, the operator gives advice to the driver of the vehicle 200, collects images of a vehicle traveling on a road at a certain point, and collects statistics. Further, the information related to the vehicle 200, such as the position information of the vehicle 200, may be saved in association with the image data.

In the communication network 20 used for communication between the image analysis device 100 and the electronic control device 201, a wireless communication method is particularly used between the electronic control device 201 and a base station device (not shown). Examples of wireless communication methods include IEEE802.11 (WiFi), IEEE802.16 (WiMAX), W-CDMA (Wideband Code Division Multiple Access), HSPA (High Speed Packet Access), LTE (Long Term Evolution), and LTE-. A (Long Term Evolution Advanced), 4G, 5G and the like can be used. A wired communication method such as a telephone line or the Internet is used between the base station device and the image analysis device 100.

A wired communication system such as a telephone line or the Internet is usually used for the communication network 30 used for communication between the image analysis device 100 and the camera 300. Of course, a wireless communication method similar to that of the communication network 20 may be used between a part of the section, for example, the camera 300 and the base station device (not shown).

In both the communication network 20 and the communication network 30, when transmitting and receiving information, transmission and reception delays occur as the traffic increases. Further, since the camera 300 compresses the image data and transmits the image data, the time required for the compression also causes a delay.

Here, the image analysis device 100 and the electronic control device 201 may be used for applications such as automatic driving and driving support, and the time is usually synchronized with high accuracy within several nanoseconds. However, the time is not always synchronized between the camera 300 and the image analysis device 100. Therefore, the shooting time associated with the image data transmitted from the camera 300 is not necessarily synchronized with the time managed by the image analysis device 100, and the reliability of the shooting time is low. Further, when the time received by the image analysis device 100 is set as the shooting time of the image data, the time shifts by the delay time.

Therefore, in the image analysis system 10 of each embodiment, a light source mounted on the vehicle 200, for example, a light source separately attached to a headlamp, a roof, or the like is set to a predetermined light emission pattern at a predetermined time according to an instruction from the image analysis device 100. It is designed to emit light. When the vehicle 200 is photographed by the camera 300, an image including this light emission pattern is photographed. Then, the image analysis device 100 identifies the shooting time and the delay time of the image by using the light emission pattern included in the image as a clue.

Next, the basic operation of the image analysis system 10 will be described with reference to FIG.

In step S11, the image analysis device 100 transmits the light emission command and the light emission time to the electronic control device 201 mounted on the vehicle 200.
In step S12, the electronic control device 201 receives the light emission command and the light emission time from the image analysis device 100.
In step S13, the electronic control device 201 instructs the light source 206 mounted on the vehicle 200 to emit light so as to emit light using the light emission pattern at the light emission time.
In step S14, the camera 300 photographs the vehicle 200 using the image sensor provided in the camera 300. At this time, the captured image also includes a light emission pattern.
In step S15, the image data including the light emission pattern is output to the image analysis device 100.
In step S16, the image analysis device 100 acquires image data of the vehicle 200 including the light emission pattern from the camera 300.
In step S17, the image analysis device 100 detects a light emission pattern from the acquired image data.
In step S18, the image analysis device 100 matches the light emission pattern targeted by the light emission command in S11 with the light emission pattern detected in S17, and if it is determined that the light emission patterns are the same, transmission is performed in S11. Let the light emission time be the shooting time of the image data. Further, the time difference between the light emission time transmitted in step S11 and the acquisition time of the image data acquired in step S16 is defined as the delay time.

With the above image analysis system 10, the shooting time of the image data can be accurately specified. Further, since the matching of the light emission pattern is used, the shooting time of the image data can be accurately specified even when the camera 300 shoots or when the image data is transmitted via the network 30 and there is a frame defect. be able to.

(Embodiment 1)
1. 1. Configuration of Image Analysis Device 100 First, the configuration of the image analysis device 100 of the present embodiment will be described with reference to FIG.

The image analysis device 100 (corresponding to the “image analysis device”) has a communication unit 101, a map DB 102, a CPU 103, an acquisition unit 106, and a memory 109.

Here, the "image analysis device" may be any name such as an information processing device, an information processing module, a microcomputer, etc., as long as it can execute calculations and controls based on the input instructions and output the result. Further, it may be a packaged semiconductor device or a configuration in which each semiconductor device is connected by wiring on a wiring board, and includes a semiconductor chip, a system board, a PC, and a personal digital assistant.

The communication unit 101 transmits / receives information to / from the electronic control device 201 via the communication network 20 of FIG. In the present embodiment, in particular, the communication unit 101 (corresponding to the “transmission unit”) is mounted on the vehicle 200 (corresponding to the “moving body”) with the light emission command of the “light emission pattern” and the light emission time for emitting the light emission pattern. It transmits to the electronic control device 201 (corresponding to the "electronic control device"). Further, the generated light emission pattern may be transmitted. Further, the communication unit 101 (corresponding to the “reception unit”) receives the position information of the vehicle 200 from the electronic control device 201. Further, the communication unit 101 may receive the travel plan information and the speed information.

Here, the "light emission pattern" includes, in addition to a blinking pattern that combines lighting and extinguishing, a dimming pattern that indicates the intensity of light without extinguishing. Further, the light emission includes infrared light and ultraviolet light regardless of the color of visible light. Further, the case where electromagnetic waves are used instead of light is also included in the light emission pattern.
Further, the “moving body” includes a case where the moving body is often stopped as long as it can move.

The map DB 102 holds information about roads, road structures, buildings, and dwellings. The map DB 102 is premised on including information on the position where the camera 300 of the present embodiment is installed and information on the shooting target area of the camera 300, but the information on the camera 300 may be stored by providing a separate database. .. The map DB 102 may be composed of a non-volatile storage device (not shown) such as an HDD or a flash memory, or may be composed of a volatile storage device such as a RAM.

The CPU 103 is connected to the communication unit 101, the map DB 102, the acquisition unit 106, and the memory 109, controls these, and performs various calculations. Further, in the present embodiment, the CPU 103 realizes a pattern generation unit 104, a light emission time determination unit 105, a pattern detection unit 107, and a shooting time identification unit 108.

The pattern generation unit 104 "generates" a light emission pattern that can be grasped as a signal using light. As the light emission pattern, various patterns can be considered, such as a case where lighting and extinguishing are combined, a case where one or a plurality of brightnesses intermediate between lighting and extinguishing are combined, and a case where the brightness changes continuously. Further, the light emission pattern may be a pattern in which a constant pattern is repeated twice or more. As a method of expressing the light emission pattern, it may be expressed as a discrete value (digital) or a continuous value (analog). In the case of discrete values, in addition to two values, three or more values may be used, and each value can be assigned to lighting, extinguishing, and intermediate brightness. Alternatively, the wavelength of light may be assigned to each value. In this case, the light emission pattern is such that a plurality of types of colors are emitted.
Note that the pattern generation unit 104 may select a specific light emission pattern from a plurality of light emission patterns held in advance, instead of generating the light emission pattern each time a light emission command is issued. Further, only one light emission pattern may be retained.

Here, the term "generate" includes not only the case of newly generating but also the case of using a previously saved one.

FIG. 4 shows an example of a light emission pattern generated by the pattern generation unit 104. In the light emission pattern of FIG. 4, the vertical axis shows the light emission intensity and the horizontal axis shows the time. In this example, the emission intensity is represented by two values, and one of the two values can be assigned to turn on and the other to turn off.
Further, FIG. 4 shows an example in which the same light emission pattern is repeated twice. In this case, it is desirable that the length of one light emission pattern is such that the start time of the first pattern and the start time of the next pattern can be specified separately.

The light emission time determination unit 105 determines the light emission time, which is the time when the light emission of the light emission pattern is started.

In the example of FIG. 4, the light emission time of the first light emission pattern among the light emission patterns is 8:00:05. In this example, only the light emission time of the first light emission pattern is specified, but the light emission time may be determined for each of the subsequent patterns.

An example of a more specific method for determining the light emission time will be given.
When it is determined that the vehicle 200 has entered the shooting target area of the camera 300 based on the position information of the vehicle 200 received by the communication unit 101, the time after a predetermined time from the current time, for example, the time after 2 seconds is emitted. Determined as the time.

The following examples are given as other determination methods.
The light emission time determination unit 105 analyzes the relative relationship between the vehicle 200 and the camera 300 based on the information of the vehicle 200 received by the communication unit 101 and the map information of the map DB 102. That is, based on the received position information of the vehicle 200, the positional relationship with the camera 300 is obtained, and based on the received travel plan information of the vehicle 200, it is determined whether or not to pass through the shooting target area of the camera 300. When it is determined that the vehicle 200 passes through the shooting target area of the camera 300, the time at which the vehicle 200 is expected to reach the shooting target area of the camera 300 is obtained based on the received speed information of the vehicle 200. Then, the time expected to arrive or a time after a certain time is determined as the light emission time. For example, when the time expected to arrive is 8:00:00, the time point at which the camera 300 is surely moving into the shooting target area is 8:00:05, which is 5 seconds later. Determined as the time.

The acquisition unit 106 (corresponding to the “acquisition unit”) “acquires” “image data” of the vehicle 200 “including the light emitting pattern” from the camera 300 that has taken the vehicle 200 via the network 30 of FIG.

Here, "including the light emission pattern" includes not only the case where the image data includes the light emission pattern but also the case where the light emission pattern is included as the data associated with the image data.
Further, the "image data" may be either a still image or a moving image.
Further, the “acquisition” includes a case of directly acquiring from a photographing device and a case of indirectly acquiring through another device.

In addition to the case where the image data is acquired via the communication network 30, the acquisition unit 106 may acquire the image data by reading the recording medium. In this case, the acquisition unit 106 can be realized by an interface such as USB or a reading device such as an optical disk drive.

5 (A) and 5 (B) show image data of the vehicle 200 including the light emission pattern. FIG. 5A shows image data in which the light source 206 of the vehicle 200 is lit, and FIG. 5B shows image data in which the light source 206 is turned off. As a method of detecting the brightness of the light source 206 in the image data, a well-known method such as image analysis can be used. Note that FIGS. 5A and 5B are expressed so that there is a difference in brightness in the entire image for the sake of explanation.

The pattern detection unit 107 (corresponding to the “pattern detection unit”) detects a light emission pattern from the acquired image data of the vehicle 200.

6 (A) and 6 (B) show a method of detecting an emission pattern from the acquired image data. As shown in FIGS. 6A and 6B, the image data including the light emission pattern is arranged in the order of the time when the light source was taken, and the image in which the light source 206 is turned on and the image in which the light source 206 is turned off are classified on the time axis. By doing so, the light emission pattern is detected from the image data.

Note that FIG. 6 shows a case where a light emission pattern having a fast blinking cycle is executed such that the light source 206 is switched between light and dark for every one or two pictures, but the blinking cycle is longer than this. It may be the light source pattern of. For example, the light emission pattern may be such that lighting and extinguishing are switched in 1 to several seconds. In such a case, it can be understood that the pictures A and B in FIG. 6 collectively represent about 30 pictures as one picture when the frame rate of the camera 30 is 30 frames per second (FPS). ..

As shown in FIG. 6B, when a part of the image data is missing or missing, the pattern detection unit 107 detects the light emission pattern by regarding the part as missing.

When infrared light or ultraviolet light is used as the light emission pattern, the light emission pattern recognized by infrared light or ultraviolet light may not be directly included in the image data depending on the wavelength range covered by the image sensor of the camera 300. There is also. In this case, it is determined whether the infrared light or ultraviolet light associated with the image data is turned on or off based on the captured data, not the image data itself.

The shooting time specifying unit 108 (corresponding to the “shooting time specifying unit”) is divided into a light emitting pattern generated by the pattern generating unit 104 and transmitted to the electronic control device 201 by the communication unit 101 and a light emitting pattern detected by the pattern detecting unit 107. Based on this, the light emission time determined by the light emission time determination unit 105 is set as the time when the image data is taken.

FIG. 7 shows an example of a method of specifying the time when the image data was taken based on the transmitted light emission pattern and the detected light emission pattern. Specifically, pattern matching is performed between the transmitted light emission pattern and the detected light emission pattern, and it is determined whether or not the waveforms of both are matched to the extent that they are considered to be the same. For example, since the detected light emission pattern P has a different shape from the transmitted light emission pattern, it can be seen that the light emission pattern is detected from video data different from the video data including the transmitted light emission pattern. On the other hand, although the detected light emission pattern Q has some defects, most of it has the same shape as the transmitted light emission pattern, so that it may be a light emission pattern detected from the video data including the transmitted light emission pattern. I understand. Then, the light emission time determination unit 105 sets the light emission time of the light emission pattern transmitted to the electronic control device 201 to the time when the image data including the detected light emission pattern Q is taken. In the example of FIG. 7, 8:00:05 is set as the time when the image data including the detected light emission pattern Q is taken.

When the same light emission pattern is repeated and a plurality of light emission times are provided for each light emission pattern, each light emission time is set as the time when the corresponding image data is taken. Thereby, it is possible to specify the accurate shooting time at the intermediate time point of the series of image data.

Further, in pattern matching, as a method of determining whether or not the waveforms match to the extent that they are considered to be the same, for example, the two waveforms are shifted to obtain the difference, and it is determined whether or not the difference exceeds a certain threshold value. Is possible. Thereby, even when a part of the light emission pattern cannot be detected due to the lack or deletion of the image data, the shooting time of the image data can be specified.

Further, the shooting time specifying unit 108 (corresponding to the “shooting time specifying unit”) may specify a delay time which is a time from the time when the image data is shot to the time when the image data is acquired.

In the case of the example of FIG. 7, it was specified that the time when the image data was taken was 8:00:05, which is the light emission time. Since the time when the image analysis device 100 acquires the image data from the camera 300 is 8:00:07, the delay time is 2 seconds, which is the time difference between them.

The memory 109 relates to position information, travel plan information, and speed information received from the vehicle 200, image data acquired from the camera 300, information indicating a frame rate, and light emission command, light emission time, light emission pattern, and light emission intensity. To save the information. The memory 109 may also be composed of a non-volatile storage device (not shown) such as an HDD or a flash memory, or may be composed of a volatile storage device such as a RAM.

2. Operation of Image Analysis Device 100 Next, the operation of the image analysis device 100 of the present embodiment will be described with reference to FIG.

Note that FIG. 8 not only shows an image analysis method using the image analysis device 100, but also shows a processing procedure of a program executed by the image analysis device. Further, the order of each step may be changed unless the relationship is such that the result of another step is used in one step. For example, the order of step S102 and step S103 can be changed. Hereinafter, the same applies to the flowchart of each embodiment.

In step S101, the communication unit 101 receives the position information from the electronic control device 201 mounted on the vehicle 200.
In step S102, the pattern generation unit 104 generates a light emission pattern.
In step S103, the light emission time determination unit 105 determines the light emission time for causing the light emission pattern to emit light based on the position information received in step S101.
In step S104, the communication unit 101 transmits a light emission command, a light emission time, and a light emission pattern to the electronic control device 201.
In step S105, the acquisition unit 106 acquires image data of the vehicle 200 including the light emission pattern from the camera 300.
In step S106, the pattern detection unit 107 detects the light emission pattern from the image data acquired in step S105.
In step S107, the shooting time specifying unit 108 determines the light emission time determined by the light emission time determination unit 105 based on the transmitted light emission pattern and the detected light emission pattern, and obtains the light emission time transmitted in step S104 by the image data acquired in step S105. It is the time when the picture was taken. Further, the shooting time specifying unit 108 specifies a delay time, which is a time from the time when the image data is shot to the time when the image data is acquired.

3. 3. Summary As described above, according to the first embodiment, since the light emission pattern is detected from the image data and the transmitted light emission pattern and the time when the image is taken are specified based on the transmitted light emission pattern, there is a frame defect at the time of information transmission. It is also possible to specify the time when the image was taken and the delay time.
Further, by being able to accurately specify the time when the image was taken, it is possible to accurately determine the temporal relationship of the images transmitted from the plurality of cameras, that is, the synchronization with other images and the shooting order. Further, since the delay time can be accurately specified, for example, when instructing or controlling the vehicle 200 in real time based on the acquired image data, it is possible to avoid using the image data having a large delay time. The above is the same for the modified examples described later.
Since the image analysis device 100 generates the light emission pattern in the present embodiment, it is not necessary to generate the light emission pattern in the vehicle 200, and the processing in the vehicle 200 can be reduced.

(Modified Example of Embodiment 1)
In the first embodiment, the image analysis device 100 generates a light emission pattern and transmits the light emission pattern to the electronic control device 201 mounted on the vehicle 200. In this modification, the electronic control device 201 generates a light emission pattern.

1. 1. Configuration of Image Analysis Device 100 Since the configuration of the image analysis device 100 of this modified example is almost the same as that of FIG. 3, FIG. 3 is incorporated.
However, in this modification, the pattern generation unit 104 is not provided. Hereinafter, only the portion having a function different from that of the first embodiment will be described with reference to FIG.

The communication unit 101 (corresponding to the “transmission unit”) transmits the light emission command of the light emission pattern and the light emission time of the light emission pattern to the electronic control device 201 of the vehicle 200, and does not transmit the light emission pattern.

The communication unit 101 (corresponding to the “reception unit”) receives a light emission pattern “generated” by the electronic control device 201 from the electronic control device 201 mounted on the vehicle 200.

Here, the term "generated" includes not only the case where it is newly generated but also the case where a previously saved one is used.

The shooting time specifying unit 108 (corresponding to the “shooting time specifying unit”) determines the light emitting time based on the light emitting pattern received from the electronic control device 201 by the communication unit 101 and the light emitting pattern detected by the pattern detecting unit 107. Let the light emission time determined in step 2 be the time when the image data was taken.

2. Operation of Image Analysis Device 100 Next, the operation of the image analysis device 100 of this modified example will be described with reference to FIG.

The operation of the image analysis device 100 of this modification does not include step S102 for generating a light emission pattern in the operation of the image analysis device 100 of the first embodiment of FIG. Instead, it has step S108 to receive the emission pattern.

In step S101, the communication unit 101 receives the position information from the electronic control device 201 mounted on the vehicle 200.
In step S103, the light emission time determination unit 105 determines the light emission time for causing the light emission pattern to emit light based on the position information received in step S101.
In step S104, the communication unit 101 transmits a light emission command and a light emission time to the electronic control device 201.
In step S108, the communication unit 101 receives the light emission pattern from the electronic control device 201.
In step S105, the acquisition unit 106 acquires image data of the vehicle 200 including the light emission pattern from the camera 300.
In step S106, the pattern detection unit 107 detects the light emission pattern from the image data acquired in the acquisition 106.
In step S107, the photographing time specifying unit 108 determines the light emitting time determined by the light emitting time determining unit 105 based on the light emitting pattern received from the electronic control device 201 and the light emitting pattern detected in step S106, and determines the light emitting time transmitted in step S104. , The time when the image data acquired in step S105 was taken. Further, the shooting time specifying unit 108 specifies a delay time, which is a time from the time when the image data is shot to the time when the image data is acquired.

3. 3. Summary As described above, according to the first modification of the first embodiment, in addition to the effects common to the first embodiment, it is not necessary for the image analysis device 100 to generate a light emission pattern each time a light emission command is given, and the image analysis device 100 does not need to generate a light emission pattern. The processing can be reduced. Further, since the unique light emission pattern can be prepared in advance in the vehicle 200, it is not necessary to generate the light emission pattern in the vehicle 200, and the processing in the vehicle can be reduced.

(Embodiment 2)
In the second embodiment, the light emission pattern to be generated is changed according to the frame rate of the camera.

1. 1. Relationship between the frame rate of the image data and the frequency of the light emission pattern The image data is composed of a moving image consisting of a set of still images taken at a specific frame rate (Frames Per Second: FPS). Still images are sometimes called pictures. Still images are sometimes called images. In addition, video is sometimes referred to as video.

The frame rate is the number of frames (number of still images, number of frames) to be processed per unit time in a moving image, and is also called a sampling frequency for a moving image. Generally, the frame rate is 60 FPS, 50 FPS, 30 FPS, 25 FPS, or 24 FPS, and the higher the frame rate, the smoother the motion of the moving image.

Since the light emission pattern can be represented by a wave having a rectangle as shown in the example of FIG. 4, the shape of the light emission pattern can be represented by a frequency. Then, as the frequency is increased, the resolution is also increased, so that the accuracy of the shooting time and the delay time of the image data can be improved. However, when the light emitting pattern is made to emit light, a light emitting element such as an LED can be blinked at a relatively high speed, but the camera 300 on the side receiving the light emitting pattern detects a light emitting pattern exceeding the frame rate of the camera 300. Can not do it. That is, in the present invention, it can be said that the theoretical limit of the frequency of the light emission pattern lies in the frame rate of the camera 300.

Here, consider the case of using the camera 300 having a variable frame rate. When the camera 300 is a camera connected to the communication network 30, the frame rate is changed according to the congestion of the communication network 30. For example, when the network 30 becomes congested, the size of the image data to be transmitted is reduced by lowering the frame rate. Correspondingly, in this embodiment, the frequency of the light emission pattern to be generated is changed according to the frame rate.

2. Configuration of Image Analysis Device 100 The configuration of the image analysis device 100 of the present embodiment is the same as that of the image analysis device 100 of the first embodiment shown in FIG. 3, so FIG. 3 is incorporated. Hereinafter, only the portion having a function different from that of the first embodiment will be described with reference to FIG.

The acquisition unit 106 (corresponding to the “acquisition unit”) further acquires information indicating the frame rate of the image data from the camera 300 in addition to the image data.

The pattern generation unit 104 (corresponding to the “pattern generation unit”) generates an emission pattern having a frequency equal to or lower than the frame rate of the image data. For example, when the frame rate is 30 FPS, an emission pattern having a frequency of 30 Hz or less, for example, an emission pattern having a frequency of 10 Hz is generated.

Further, when the frame rate is changed while the image data including the light emission pattern is being captured, the pattern generation unit 104 may change to the light emission pattern having a frequency equal to or lower than the changed frame rate. ..

For example, consider a case where the frame rate of the camera 300 is 25 FPS for 30 seconds after the vehicle 200 reaches the shooting target area, and then 15 FPS. In this case, the pattern generation unit 104 generates an emission pattern having a frequency of 25 Hz or less for the first 30 seconds. Then, after that, an emission pattern having a frequency of 15 Hz or less is generated.

3. 3. Operation of Image Analysis Device 100 Next, the operation of the image analysis device 100 of the present embodiment will be described with reference to FIG.

As for the operation of the image analysis device 100 of the present embodiment, step S201 for acquiring the frame rate of the image data is added to the operation of the image analysis device 100 of the first embodiment of FIG. Then, the function of step S102 is changed so as to generate a light emission pattern in consideration of the frame rate. Since the other steps are the same as those in FIG. 8, only the steps different from those in FIG. 8 will be described.

After step S101, in step S210, the acquisition unit 106 acquires the frame rate of the image data from the camera 300.
In step S102, the pattern generation unit 104 generates an emission pattern having a frequency equal to or lower than the frame rate of the image data.

4. Summary As described above, according to the second embodiment, even when the frame rate is variable, it is possible to generate an emission pattern having a frequency as close as possible to each frame rate. As a result, the accuracy of the shooting time and the delay time of the image data can be improved.

(Embodiment 3)
In the third embodiment, a light emission pattern obtained by synthesizing two or more light emission patterns is used.

1. 1. Waveform of light emission pattern FIG. 11 shows a case where a light emission pattern having a low frequency and a light emission pattern having a high frequency are combined to generate a combined light emission pattern. Hereinafter, among the emission patterns included in the composite emission pattern after synthesis, the emission pattern having a low frequency is referred to as a low frequency component emission pattern, and the emission pattern having a high frequency is referred to as a high frequency component emission pattern.

By synthesizing the low-frequency component emission pattern (corresponding to the "first emission pattern") and the high-frequency component emission pattern (corresponding to the "second emission pattern"), a combined emission pattern having the characteristics of both can be obtained. For example, the low frequency component emission pattern is 5 FPS, and the high frequency component emission pattern is 30 FPS. Based on this synthetic light emission pattern, the light source 206 of the vehicle 200 is made to emit light. Since the vertical axis of the synthetic light emission pattern indicates the intensity of light, the synthetic light emission pattern is emitted as a light emission pattern accompanied by the intensity of light.

The camera 300 captures image data including the composite light emission pattern and transmits it to the image analysis device 100. Here, the waveform of the synthetic light emission pattern that can be detected by the pattern detection unit 107 differs depending on the shooting environment of the camera 300. When the shooting environment is good (when the S / N ratio is high), for example, when the weather is fine or the atmosphere is clear, the characteristics of the high-frequency component emission pattern remain sufficiently as shown in FIG. 11 (A), and the high-frequency component The emission pattern can be separated and detected. However, when the shooting environment is bad (when the S / N ratio is low), for example, when it is cloudy or rainy, or when the atmosphere is polluted, the characteristics of the high-frequency component emission pattern remain sufficiently as shown in FIG. 11B. Only the low frequency component emission pattern can be detected. That is, when the shooting environment is good, the delay time can be specified with the accuracy of the frequency level of the high frequency component emission pattern. Further, when the shooting environment is bad, the accuracy of the frequency level of the high frequency component emission pattern is difficult, but the delay time can be specified by the accuracy of the frequency level of the low frequency component emission pattern.

FIG. 12 shows a case where a blinking pattern, which is a light emitting pattern combining lighting and extinguishing, and a dimming pattern indicating the intensity of light are combined to generate a synthetic light emitting pattern. Since the blinking pattern is a square wave, it can be said to be a digital signal. Further, the dimming pattern can be said to be an analog signal because the light intensity changes continuously. That is, the example of FIG. 12 can be said to be an example of synthesizing an emission pattern composed of digital waves and an emission pattern composed of analog waves. From another point of view, it can be said that analog waves have fine quantization. Therefore, in the example of FIG. 12, the blinking pattern may be interpreted as a low-frequency component emission pattern, and the dimming pattern may be interpreted as a high-frequency component emission pattern. it can.

In such a synthetic light emission pattern, when the shooting environment is good (when the S / N ratio is high), the characteristics of the dimming pattern remain sufficiently as shown in FIG. 12A, and the dimming pattern is separated. Can be detected. On the other hand, when the shooting environment is bad (when the S / N ratio is low), the characteristics of the dimming pattern do not sufficiently remain as shown in FIG. 12B, and only the blinking pattern can be detected.

2. Configuration of Image Analysis Device 100 The configuration of the image analysis device 100 of the present embodiment is the same as that of the image analysis device 100 of the first embodiment shown in FIG. 3, so FIG. 3 is incorporated. The difference from FIG. 3 is that the pattern generation unit 104 can synthesize a plurality of light emission patterns to generate a synthetic light emission pattern.

3. 3. Summary As described above, according to the third embodiment, it is possible to improve the detection accuracy of the delay time when the shooting environment is good, and to detect the minimum delay time even when the shooting environment is bad.

(Embodiment 4)
In the fourth embodiment, the emission intensity of the emission pattern is changed according to the distance between the vehicle 200 and the camera 300.

1. 1. Light emission intensity of light emission pattern When the distance between the vehicle 200 and the camera 300 is long, it is considered that the light emission pattern included in the image data cannot be clearly detected. Therefore, the emission intensity of the emission pattern is changed according to the distance between the vehicle 200 and the camera 300.

2. Configuration of Image Analysis Device 100 Next, the configuration of the image analysis device 100 of the present embodiment will be described with reference to FIG.

The configuration of the image analysis device 100 of the present embodiment is that the light emission intensity determining unit 110 is added to the image analysis device 100 of the first embodiment shown in FIG.

The light emission intensity determination unit 110 (corresponding to the “light emission intensity determination unit”) has a light emission pattern based on the position information of the vehicle 200 acquired by the communication unit 101 and the distance between the vehicle 200 and the camera 300 calculated from the position of the camera 300. Determines the emission intensity of.
In order to change the emission intensity of the emission pattern, it is necessary to change the luminous intensity, brightness, luminous flux, illuminance or radiant intensity of the light from the light source 206 mounted on the vehicle 200. Therefore, the emission intensity determination unit 110 determines any one of the luminous intensity, the brightness, the luminous flux, the illuminance, and the radiation intensity of the light from the light source indicating the emission intensity.

The communication unit 101 (corresponding to the “transmission unit”) transmits the light emission intensity determined by the light emission intensity determination unit 110 to the electronic control device 201 in addition to the light emission command, the light emission time, and the light emission pattern.

3. 3. Operation of Image Analysis Device 100 Next, the operation of the image analysis device 100 of the present embodiment will be described with reference to FIG.

The operation of the image analysis device 100 of the present embodiment adds step S401 for determining the light emission intensity to the operation of the image analysis device 100 of the first embodiment of FIG. 8, and steps S104 for transmitting the light emission intensity to the vehicle 200. Is a modification of. Since the other steps are the same as those in FIG. 8, only the steps different from those in FIG. 8 will be described.

After step S103, in step S401, the light emission intensity determination unit 110 determines the light emission intensity of the light emission pattern based on the position of the camera 300 and the position information of the vehicle 200.
In step S104, the communication unit 101 transmits the light emission intensity to the electronic control device 201 in addition to the light emission command, the light emission time, and the light emission pattern.

4. Summary As described above, according to the fourth embodiment, the light emission pattern can be detected from the image data even when the distance between the vehicle 200 and the camera 300 is long.

(Embodiment 5)
In the fifth embodiment, the configuration and operation of the electronic control device 201 mounted on the vehicle 200 will be described.

1. 1. Configuration of Electronic Control Device 201 First, the configuration of the electronic control device 201 of the present embodiment will be described with reference to FIG.

The electronic control device 201 and the light source 206 are mounted on the vehicle 200.

The electronic control device 201 includes a GPS 202, a CPU 203, a communication unit 204, and a memory 205.

The GPS 202 receives a signal from a GPS satellite and detects the current position information of the vehicle 200. For example, the GPS 202 detects longitude and latitude information as position information at regular intervals while the vehicle 200 is traveling. In addition to GPS, GPS202 may be a differential GPS or an inertial navigation system (INS).

The CPU 203 is connected to the GPS 202, the communication unit 204, the memory 205, and the light source 206, controls these, and performs various calculations. Further, the CPU 202 realizes the light emitting instruction unit 207.
The light emission instruction unit 207 instructs the light source 206 to emit light based on the light emission command, the light emission time, the light emission pattern, and the light emission intensity received by the communication unit 204.

The communication unit 204 (corresponding to the “reception unit”) receives a light emission command of the light emission pattern and a light emission time for causing the light emission pattern to be emitted from the image analysis device 100 (corresponding to the “image analysis device”).

Further, the communication unit 204 transmits the position information acquired by the GPS 202 to the image analysis device 100. In addition to this, travel plan information and speed information acquired by various other sensors (not shown) may be transmitted.
The position information, the travel plan information, and the speed information may be transmitted at regular time intervals or every time the vehicle 200 behaves.

As in the first modification of the first embodiment, the electronic control device 201 has a pattern generation unit, transmits a light emission pattern to the image analysis device 100, and the vehicle 200 and the image analysis device 100 generate a light emission pattern. You may want to share it. Further, instead of generating a light emission pattern each time a light emission command is issued, a specific light emission pattern may be selected from a plurality of light emission patterns held in advance.

The memory 205 stores the light emission pattern, the light emission time, the light emission intensity, and the like received from the image analysis device 100. The memory 205 may be composed of a non-volatile storage device (not shown) such as an HDD or a flash memory, or may be composed of a volatile storage device such as a RAM.

The light source 206 emits light by using a light emission pattern at the light emission time based on the instruction of the light emission instruction unit 207. As the light source 206, incandescent lamps, halogen bulbs, fluorescent lamps, strobe lights, lasers, measurers and the like can be used in addition to LEDs. The light emitted may be infrared light or ultraviolet light in addition to visible light, but in that case, it is necessary that the camera 300 has an imaging element capable of photographing infrared light or ultraviolet light. ..

2. Operation of Electronic Control Device 201 Next, the operation of the electronic control device 201 will be described with reference to FIG.

In step S501, the communication unit 204 transmits the position information of the vehicle 200 acquired by GPS 202 to the image analysis device 100 based on the instruction of the CPU 203.
In step S502, the communication unit 204 receives the light emission command, the light emission time, and the light emission pattern from the image analysis device 100.
In step S503, the light emitting instruction unit 207 instructs the light source 206 mounted on the vehicle 200 to emit light at the light emitting time using the light emitting pattern.

3. 3. Summary As described above, according to the fifth embodiment, by emitting light from the light source 206, the image analysis apparatus 100 can use the light emission pattern to specify the shooting time and the delay time of the image data.

(Summary)
The features of the image analysis device, the electronic control device, and the image analysis method executed by these in each embodiment of the present invention have been described above.

Since the terms used in each embodiment are examples, they may be replaced with synonymous terms or terms including synonymous functions.

The block diagram used in the description of the embodiment is a classification and arrangement of the configurations of the image analysis device, the moving body, and the like for each function. These functional blocks are realized by any combination of hardware or software. Further, since the block diagram shows the function, the block diagram can be grasped as the disclosure of the invention of the method.

The order of the processes, flows, and functional blocks that can be grasped as a method described in each embodiment may be changed as long as there are no restrictions such as using the results of other steps in one step.

The terms "first" and "second" used in each embodiment and the present invention are used to distinguish two or more configurations and methods of the same type, and do not limit the order or superiority or inferiority.

In addition, the present invention can be realized not only by the dedicated hardware having the configuration and the function described in each embodiment, but also a program for realizing the present invention recorded on a recording medium such as a memory or a hard disk, and an execution thereof. It can also be realized as a combination with a general-purpose hardware having a possible dedicated or general-purpose CPU and memory.

Programs stored in dedicated or general-purpose hardware recording media (external storage devices (hard disks, USB memory, CD / BD, etc.), internal storage devices (RAM, ROM, etc.)) can be recorded via the recording medium. It can also be provided to dedicated or general-purpose hardware from the server via a communication line without using a medium. This ensures that you always have the latest features through program upgrades.

Although the moving body of the present invention has been described mainly as a vehicle, it can be applied to all moving moving bodies such as motorcycles, bicycles with electric motors, railways, ships, and aircraft.

100 image analysis device, 101 communication unit, 104 pattern generation unit, 106 acquisition unit, 107 pattern detection unit, 108 shooting time identification unit, 110 emission intensity determination unit, 200 vehicles, 201 electronic control device, 204 communication unit, 206 light source, 207 Light emission indicator

Claims (14)

A transmission unit (101) that transmits the light emission command of the light emission pattern and the light emission time of the light emission pattern to the electronic control device (201) mounted on the moving body (200).
An acquisition unit (106) that acquires image data of the moving body including the light emitting pattern from an imaging device that has taken an image of the moving body.
A pattern detection unit (107) that detects the light emission pattern from the image data,
A shooting time specifying unit (108) having the light emission time as the time when the image data was shot,
Have,
Image analyzer (100). The image analysis device further includes a pattern generation unit (104) that generates the light emission pattern.
The transmission unit further transmits the generated light emission pattern to the electronic control device.
The shooting time specifying unit sets the light emission time as the time when the image data is shot, based on the transmitted light emission pattern and the detected light emission pattern.
The image analysis apparatus according to claim 1. The image analysis device further includes a receiving unit (101) that receives the light emission pattern generated by the electronic control device.
The shooting time specifying unit sets the light emission time as the time when the image data is shot, based on the received light emission pattern and the detected light emission pattern.
The image analysis apparatus according to claim 1. The shooting time specifying unit specifies a delay time, which is the time from the time when the image data is shot to the time when the image data is acquired.
The image analysis apparatus according to claim 1. The acquisition unit further acquires information indicating the frame rate of the image data from the photographing apparatus.
The pattern generation unit generates the light emission pattern having a frequency equal to or lower than the frequency indicated by the frame rate.
The image analysis apparatus according to claim 2. The pattern generation unit generates the light emission pattern by synthesizing a first light emission pattern composed of a low frequency component and a second light emission pattern composed of a high frequency component.
The image analysis apparatus according to claim 2. The pattern generation unit generates the light emission pattern by synthesizing a blinking pattern that combines lighting and extinguishing and a dimming pattern that indicates the intensity of light.
The image analysis apparatus according to claim 2. The image analysis device further includes a receiving unit (101) that receives position information from the electronic control device.
It has a light emitting intensity determining unit (110) that determines the light emitting intensity of the light emitting pattern based on the position of the photographing device that photographed the moving body and the position information.
The transmitting unit transmits the light emitting intensity determined by the light emitting intensity determining unit to the electronic control device.
The image analysis apparatus according to claim 1. An image analysis system including an electronic control device (201) mounted on a moving body (200) and an image analysis device (100) that communicates with the electronic control device.
The electronic control device is
A receiver (204) that receives the light emission command of the light emission pattern and the light emission time of the light emission pattern from the image analyzer, and
A light emitting instruction unit (207) that instructs a light source (206) mounted on the moving body to emit light using the light emitting pattern at the light emitting time.
Have,
The image analysis device is
A transmission unit (101) that transmits the light emission command and the light emission time to the electronic control device, and
An acquisition unit (106) that acquires image data of the moving body including the light emitting pattern from an imaging device that has taken an image of the moving body.
A pattern detection unit (107) that detects the light emission pattern from the image data,
A shooting time specifying unit (108) having the light emission time as the time when the image data was shot,
Have,
Image analysis system. An electronic control device (201) mounted on a mobile body (200).
A receiver (204) that receives the light emission command of the light emission pattern and the light emission time of the light emission pattern from the image analysis device, and
A light emitting instruction unit (207) that instructs a light source (206) mounted on the moving body to emit light using the light emitting pattern at the light emitting time.
Have,
Electronic control device. An image analysis method executed by an image analysis device (100) that communicates with an electronic control device (201) mounted on a mobile body (200).
The light emission command of the light emission pattern and the light emission time of the light emission pattern are transmitted to the electronic control device (S104).
The image data of the moving body including the light emission pattern is acquired from the photographing device in which the moving body is photographed (S105).
The emission pattern is detected from the image data (S106),
Let the light emission time be the time when the image data was taken (S107).
Image analysis method. An image analysis method executed by an image analysis system including an electronic control device (201) mounted on a moving body (200) and an image analysis device (100) that communicates with the electronic control device.
The image analysis device is
The light emission command of the light emission pattern and the light emission time of the light emission pattern are transmitted to the electronic control device (S104).
The electronic control device is
The light emission command and the light emission time are received from the image analyzer (S502),
The light source (206) mounted on the moving body was instructed to emit light using the light emission pattern at the light emission time (S503).
The image analysis device is
The image data of the moving body including the light emission pattern is acquired from the photographing device in which the moving body is photographed (S105).
The emission pattern is detected from the image data (S106),
Let the light emission time be the time when the image data was taken (S107).
Image analysis method. An image analysis program executed by an image analysis device (100) that communicates with an electronic control device (201) mounted on a mobile body (200).
The light emission command of the light emission pattern and the light emission time of the light emission pattern are transmitted to the electronic control device (S104).
The image data of the moving body including the light emission pattern is acquired from the photographing device in which the moving body is photographed (S105).
The emission pattern is detected from the image data (S106),
Let the light emission time be the time when the image data was taken (S107).
Image analysis program. A light emission pattern lighting program executed by an electronic control device (201) mounted on a moving body (200).
The light emission command of the light emission pattern and the light emission time of the light emission pattern are received from the image analyzer (S502).
The light source (206) mounted on the moving body is instructed to emit light using the light emission pattern at the light emission time (S503).
Light emission pattern lighting program.