JP2018125777A - Imaging apparatus, learning server, and imaging system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for transmitting required information, while restraining increase in traffic.SOLUTION: An imaging part 50 captures an image. A receiving part 56 receives the learning results of a neural network leant by an image including a processing object, from a learning server 40. A determination part 60 inputs the image, captured in the imaging part 50, to the neural network reflecting the learning results received in the receiving part 56, and determines whether or not a processing object is included in the image captured in the imaging part 50, based on the output from the neural network. When a determination is made in the determination part 60 that the processing object is included in the image, a transmission part 58 transmits the image captured in the imaging part 50. The receiving part 56 receives the learning results of the neural network learnt by the image transmitted from the transmission part 58 from the learning server 40.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging technique, and particularly relates to an imaging apparatus, a learning server, and an imaging system that transmit captured images.

  The drive recorder usually includes a moving image capturing unit, an acceleration sensor, a GPS (Global Positioning System), a memory unit, a communication module, and the like. This drive recorder automatically records images within a certain period of time before and after the accident, situations where there was a risk of an accident, and the like. On the other hand, it is required that road information and the like are easily transmitted automatically and arbitrarily as the automobile travels, and that the abundant road information and the like are used by users accurately. For this reason, the data center and the drive recorder of each user are connected via a communication network, and the road photographed image, the traveling speed, the traveling route, and the like from the drive recorder are accumulated in the data center and used (for example, Patent Document 1). reference).

Japanese Patent Laid-Open No. 2015-210713

  When data, particularly video, is continuously transmitted from the drive recorder to the data center, the communication amount increases. On the other hand, when the transmission timing of the video is limited, the communication amount decreases, but the necessary video may not be transmitted.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for transmitting necessary information while suppressing an increase in traffic.

  In order to solve the above-described problem, an imaging device according to an aspect of the present invention includes an imaging unit that captures an image, and a reception unit that receives a learning result of a neural network learned from an image including a processing target from a learning server. The image captured by the image capturing unit is input to the neural network that reflects the learning result received by the receiving unit, and the image captured by the image capturing unit is included in the processing based on the output from the neural network. A determination unit that determines whether or not the image is processed, and a transmission unit that transmits the image captured by the imaging unit when the determination unit determines that the processing target is included in the image. The receiving unit receives the learning result of the neural network learned from the image transmitted by the transmitting unit from the learning server.

  Another aspect of the present invention is a learning server. The learning server is a learning server that can communicate with an imaging device that captures an image, and a learning unit that learns a neural network from an image including a processing target, and a transmission that transmits a learning result in the learning unit to the imaging device. And the imaging device, the captured image is input to the neural network reflecting the learning result transmitted by the transmission unit, and the image includes a processing target based on the output from the neural network. A receiving unit that receives an image captured by the imaging device. The learning unit learns the neural network from the image received by the receiving unit.

  Yet another embodiment of the present invention is an imaging system. The imaging system includes a learning server that learns a neural network from an image including a processing target and transmits a learning result, and an imaging device that captures the image and receives the learning result from the learning server. The imaging apparatus inputs a captured image to a neural network that reflects the received learning result, and determines whether the captured image includes a processing target based on an output from the neural network. A determination unit and a transmission unit that transmits a captured image when the determination unit determines that the processing target is included in the image. The learning server learns the neural network from the image transmitted by the transmission unit.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, necessary information can be transmitted while suppressing an increase in communication traffic.

It is a figure which shows the structure of the imaging system which concerns on an Example. It is a figure which shows the data structure of the learning result transmitted from the learning server of FIG. It is a sequence diagram which shows the transmission procedure by the imaging system of FIG. 3 is a flowchart illustrating a transmission procedure by the imaging apparatus of FIG. 1.

  An outline of the present invention will be given before a specific description of embodiments of the present invention. An embodiment of the present invention relates to an imaging system including an imaging device such as a drive recorder mounted on a vehicle and a storage server connected to the imaging device via a network. In the imaging system, the imaging device transmits video to a storage server, and the storage server stores video. When the imaging device continuously transmits video, the network traffic increases, and the load on the network and the load on the storage server also increase. In addition, communication charges will increase. In order to prevent the occurrence of such a situation, there is a possibility that the necessary video cannot be transmitted when the timing of transmitting the video from the imaging device is limited. An object of the present embodiment is to transmit necessary video while reducing the amount of communication.

  The video system according to the present embodiment further includes a learning server, and the learning server learns the neural network from the image including the processing target. Here, the processing target includes objects such as vehicles, pedestrians, bicycles, and motorcycles, scenes such as encounters, rear-end collisions, and jumping out. Therefore, the learning result in the learning server is set so that an image including these processing targets can be detected. The learning server transmits the learning result to the imaging device. The imaging device updates the neural network so as to reflect the received learning result, and inputs each image included in the captured video to the neural network, thereby detecting an image including the processing target. When the imaging device detects an image including the processing target, the imaging device transmits a video of a certain period including the image to the storage server via the network. Further, the learning server repeatedly executes the above learning based on the video stored in the storage server, and sequentially transmits the learning result to the imaging device. The imaging device further updates the neural network to reflect the received learning result, and repeats the above-described processing.

  FIG. 1 illustrates a configuration of an imaging system 100 according to the embodiment. The imaging system 100 includes a vehicle 10, an imaging device 20, a storage server 30, and a learning server 40. The imaging device 20 includes an imaging unit 50, a processing unit 52, a storage unit 54, a reception unit 56, and a transmission unit 58, and the processing unit 52 includes a determination unit 60. The learning server 40 includes a receiving unit 70, a learning unit 72, and a transmitting unit 74. Here, the imaging device 20 is a drive recorder mounted on the vehicle 10.

  The learning server 40 can communicate with the imaging device 20 that captures an image composed of a combination of images. The learning unit 72 of the learning server 40 learns the neural network from the image including the processing target. The processing target is classified into a target scene (hereinafter sometimes simply referred to as “scene”) and a target object (hereinafter also referred to as “target”). As described above, the scene includes an encounter, a rear-end collision, a jump-out, and the like, and the objects include a vehicle, a pedestrian, a bicycle, a two-wheeled vehicle, and the like. That is, since it is assumed that the imaging device 20 is a drive recorder, a scene and an object in which a traffic accident has occurred or is highly likely to occur are targeted for processing.

  Learning of neural networks is performed by deep learning, which is one of machine learning techniques, and is based on a technique based on a human brain neural circuit. In neural networks, perceptrons imitating brain neurons are combined in three or more layers, and the features of data are automatically and deeply understood by learning from various directions and stages. Therefore, the neural network is suitable for image discrimination processing that cannot be performed without understanding complicated features. Since a known technique may be used for the neural network and the neural network learning, the description thereof is omitted here.

  The learning unit 72 decomposes the video for each scene such as encounter, rear-end collision, and pop-out into images for each frame, and inputs them into a neural network constructed in advance in chronological order. Thereby, the neural network learns the features of each scene in time series. In addition, the learning unit 72 improves learning accuracy by repeating tens of thousands to hundreds of thousands of such learning. Further, the learning unit 72 decomposes the video for each object such as a vehicle, a pedestrian, a bicycle, and a two-wheeled vehicle into an image for each frame, and executes a process similar to the process described above. As a result, the learning unit 72 acquires the learning result of the neural network for the scene and the object.

  The transmission unit 74 transmits the learning result in the learning unit 72 to the imaging device 20 via the network. FIG. 2 shows the data structure of the learning result transmitted from the learning server 40. This indicates a parameter included in the learning result. The optimization function includes a function name such as Adam and its parameters. The network model includes a base such as VGG, a convolution, a max pooling number, a layer such as total coupling, and BatchNormalization (yes or no). The activation function includes a function such as relu. The dropout includes parameters such as dropout and yes, and the loss calculation includes softmax_cross_entropy and the like. Returning to FIG.

  The receiving unit 56 of the imaging device 20 receives the learning result from the learning server 40. The receiving unit 56 outputs the learning result to the processing unit 52. The imaging unit 50 captures an image. The captured video is configured by arranging a plurality of images in time series. Therefore, it can be said that the imaging unit 50 captures an image. The imaging unit 50 converts captured images and images into digital data. In the following, video and images converted into digital data are also referred to as video and images. The imaging unit 50 outputs the video to the processing unit 52.

  The storage unit 54 stores the video imaged by the imaging unit 50 according to an instruction from the processing unit 52. Note that at least some of the plurality of images included in the video include information for specifying time such as a time stamp (hereinafter referred to as “time information”), and the storage unit 54 stores time information. Also remember.

  The determination unit 60 updates the neural network so that the learning result received by the reception unit 56 is reflected. The determination unit 60 inputs each image stored in the storage unit 54, that is, each image captured by the imaging unit 50, to the neural network. The determination unit 60 determines whether each image includes a processing target based on the output from the neural network. This corresponds to a learned neural network determining each image and detecting a scene and an object. That is, the determination unit 60 determines whether or not the above-described scene and object are included in each image.

  When the determination unit 60 detects an image including a scene and an object, the determination unit 60 acquires a video of a certain period including the image from the storage unit 54. For example, the certain period is defined to be from a timing before the image to a timing after the image. Note that the above-described time information is used to acquire such a certain period of video. Here, a video for a certain period is acquired when an image including a scene and an object is detected, but an image for a certain period is acquired when an image of either the scene or the object is detected. May be.

  When the determination unit 60 determines that the processing target is included in the image, the transmission unit 58 transmits video for a certain period to the storage server 30 via the network. This corresponds to transmitting an image captured by the imaging unit 50. The accumulation server 30 receives the video from the imaging device 20 and accumulates it. A learning server 40 is connected to the storage server 30 via a network. Note that the storage server 30 and the learning server 40 may be configured integrally.

  The receiving unit 70 of the learning server 40 receives video from the storage server 30. This is equivalent to receiving a video image captured by the imaging device 20 and including an image determined to be included in the imaging device 20 as a processing target. The receiving unit 70 outputs the video to the learning unit 72. The learning unit 72 performs the above-described neural network learning based on the video received by the receiving unit 70. That is, the learning unit 72 decomposes the received video into images for each frame, and inputs them into a neural network constructed in advance in chronological order. Subsequent processing is the same as heretofore, and the description is omitted here. A learning result by such processing is also transmitted from the transmission unit 74 to the imaging device 20.

  The receiving unit 56 of the imaging device 20 newly receives a learning result of the neural network learned from the video transmitted by the transmitting unit 58 from the learning server 40. The determination unit 60 updates the neural network so that the learning result newly received by the reception unit 56 is reflected. Since the subsequent processing is the same as before, the processing will be described here. Therefore, the learning of the neural network in the learning server 40 and the determination by the neural network in the imaging device 20 are repeatedly executed in a loop shape.

  Here, only one imaging device 20 mounted on one vehicle 10 is shown. However, the imaging system 100 may include a plurality of vehicles 10 on which the imaging device 20 is mounted. In such a configuration, the learning server 40 receives video captured by each of the plurality of imaging devices 20. At that time, the learning unit 72 learns the neural network from images captured by the plurality of imaging devices 20. By such processing, the number of images used when learning the neural network is increased.

  This configuration can be realized in terms of hardware by a CPU, memory, or other LSI of any computer, and in terms of software, it can be realized by a program loaded in the memory, but here it is realized by their cooperation. Draw functional blocks. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms only by hardware, or by a combination of hardware and software.

  The operation of the imaging system 100 having the above configuration will be described. FIG. 3 is a sequence diagram illustrating a transmission procedure by the imaging system 100. The learning server 40 learns the neural network (S10). The learning server 40 transmits the learning result (S12). The imaging device 20 updates the neural network (S14). The imaging device 20 captures an image (S16). The imaging device 20 detects a processing target in the image included in the video (S18). The imaging device 20 transmits video for a certain period (S20). The learning server 40 learns the neural network from the video included in the video (S22). The learning server 40 transmits the learning result (S24). The imaging device 20 updates the neural network (S26).

  FIG. 4 is a flowchart showing a transmission procedure by the imaging apparatus 20. The receiving unit 56 receives the learning result (S50). The determination unit 60 updates the neural network (S52). When the determination unit 60 detects a target scene (Y in S54) and detects a target object (Y in S56), the transmission unit 58 transmits videos before and after the detection time (S58). If the determination unit 60 does not detect the target scene (N in S54) or does not detect the target object (N in S56), the process ends.

  According to the present embodiment, the image is transmitted when the image including the processing target is detected by the neural network learned from the image including the processing target, so that the necessary video can be transmitted only once. In addition, since the necessary video is transmitted only once, necessary information can be transmitted while suppressing an increase in traffic. In addition, since a neural network learned from an image including a target scene is used, an image including the target scene can be detected. In addition, since a neural network learned from an image including an object is used, an image including the object can be detected.

  In addition, since the image is transmitted when the image including the processing target is detected by using the neural network learned by the image including the processing target, the necessary information can be obtained while suppressing the increase in the traffic. Can receive. Moreover, since the neural network is learned based on the received image, the accuracy of learning can be improved. In addition, since the neural network is learned from images captured by each of a plurality of imaging devices, the number of images used for learning can be increased. Further, since the number of images used for learning is increased, the accuracy of learning can be improved.

  The outline of one embodiment of the present invention is as follows. An imaging apparatus according to an aspect of the present invention includes an imaging unit that captures an image, a receiving unit that receives a learning result of a neural network learned from an image including a processing target from a learning server, and learning received by the receiving unit. A determination that determines whether or not a processing target is included in the image captured by the imaging unit based on the output from the neural network based on the output from the neural network by inputting the image captured by the imaging unit to the neural network that reflects the result And a transmission unit that transmits an image captured by the imaging unit when the determination unit determines that the processing target is included in the image. The receiving unit receives the learning result of the neural network learned from the image transmitted by the transmitting unit from the learning server.

  According to this aspect, since the image is transmitted when the image including the processing target is detected by the neural network learned from the image including the processing target, the necessary information is obtained while suppressing an increase in the communication amount. Can be sent.

  The receiving unit receives the learning result of the neural network learned from the image including the target scene from the learning server, and the determining unit determines whether the target scene is included in the image captured by the imaging unit. It may be determined. In this case, since a neural network learned from an image including a target scene is used, an image including the target scene can be detected.

  The receiving unit receives the learning result of the neural network learned from the image including the target object from the learning server, and the determining unit determines whether the target object is included in the image captured by the imaging unit. It may be determined. In this case, since a neural network trained by an image including the target object is used, an image including the target object can be detected.

  Another aspect of the present invention is a learning server. The learning server is a learning server that can communicate with an imaging device that captures an image, and a learning unit that learns a neural network from an image including a processing target, and a transmission that transmits a learning result in the learning unit to the imaging device. And the imaging device, the captured image is input to the neural network reflecting the learning result transmitted by the transmission unit, and the image includes a processing target based on the output from the neural network. A receiving unit that receives an image captured by the imaging device. The learning unit learns the neural network from the image received by the receiving unit.

  According to this aspect, it is necessary to suppress an increase in communication amount because an image is transmitted when an image including a processing target is detected by using a neural network learned from the image including the processing target. Can receive information.

  The receiving unit receives an image captured by each of the plurality of imaging devices, and the learning unit is configured to learn the neural network based on the image received by the receiving unit and captured by each of the plurality of imaging devices. Also good. In this case, since the neural network is learned from images captured by each of the plurality of imaging devices, the learning accuracy can be improved.

  Yet another embodiment of the present invention is an imaging system. The imaging system includes a learning server that learns a neural network from an image including a processing target and transmits a learning result, and an imaging device that captures the image and receives the learning result from the learning server. The imaging apparatus inputs a captured image to a neural network that reflects the received learning result, and determines whether the captured image includes a processing target based on an output from the neural network. A determination unit and a transmission unit that transmits a captured image when the determination unit determines that the processing target is included in the image. The learning server learns the neural network from the image transmitted by the transmission unit.

  According to this aspect, since the image is transmitted when the image including the processing target is detected by the neural network learned from the image including the processing target, the necessary information is obtained while suppressing an increase in the communication amount. Can be sent.

  In the above, this invention was demonstrated based on the Example. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to each of those constituent elements or combinations of processing processes, and such modifications are also within the scope of the present invention. .

  In this embodiment, the imaging device 20 is mounted on the vehicle 10. However, the present invention is not limited to this. For example, the imaging device 20 may be installed other than the vehicle 10 or may be used like a surveillance camera. According to this modification, the application range can be expanded.

  DESCRIPTION OF SYMBOLS 10 Vehicle, 20 Imaging device, 30 Storage server, 40 Learning server, 50 Imaging part, 52 Processing part, 54 Storage part, 56 Receiving part, 58 Transmitting part, 60 Determination part, 70 Receiving part, 72 Learning part, 74 Transmitting part , 100 imaging system.

Claims (6)

An imaging unit that captures an image;
A receiving unit that receives a learning result of a neural network learned from an image including a processing target from a learning server;
The image captured by the imaging unit is input to a neural network that reflects the learning result received by the receiving unit, and the image captured by the imaging unit includes a processing target based on the output from the neural network. A determination unit for determining whether or not
A transmission unit that transmits an image captured by the imaging unit when the determination unit determines that the processing target is included in the image;
The image receiving apparatus, wherein the receiving unit receives a learning result of a neural network learned from an image transmitted by the transmitting unit from the learning server. The receiving unit receives a learning result of a neural network learned from an image including a target scene from a learning server,
The imaging device according to claim 1, wherein the determination unit determines whether a target scene is included in an image captured by the imaging unit. The receiving unit receives the learning result of the neural network learned from the image including the target object from the learning server,
The imaging apparatus according to claim 1, wherein the determination unit determines whether or not a target object is included in an image captured by the imaging unit. A learning server capable of communicating with an imaging device that captures an image,
A learning unit for learning a neural network from an image including a processing target;
A transmission unit that transmits a learning result in the learning unit to the imaging device;
In the imaging apparatus, when a captured image is input to a neural network that reflects the learning result transmitted by the transmission unit, and the processing target is included in the image based on an output from the neural network. A reception unit that receives an image captured by the imaging device when determined,
The learning server, wherein the learning unit learns a neural network from images received by the receiving unit. The receiving unit receives an image captured by each of a plurality of imaging devices,
The learning server according to claim 4, wherein the learning unit is configured to learn a neural network from images received by the receiving unit and images captured by the plurality of imaging devices. A learning server that learns a neural network from an image including a processing target and transmits a learning result;
An imaging device that captures an image and receives a learning result from the learning server;
The imaging device
A determination unit that inputs a captured image to a neural network that reflects the received learning result, and determines whether the captured image includes a processing target based on an output from the neural network;
A transmission unit that transmits the captured image when the determination unit determines that the processing target is included in the image;
The learning system, wherein the learning server learns a neural network from an image transmitted by the transmission unit.

Images