WO2024057408A1

WO2024057408A1 - Control device, control method, and program

Info

Publication number: WO2024057408A1
Application number: PCT/JP2022/034243
Authority: WO
Inventors: 悠希中原; 健桑原; 亮太石橋
Original assignee: 日本電信電話株式会社
Priority date: 2022-09-13
Filing date: 2022-09-13
Publication date: 2024-03-21

Abstract

This control device comprises: an application proxy that receives data transmitted from a device; an application unit that receives the data retransmitted from the application proxy; and a control proxy that receives, from the application unit, a control request obtained on the basis of processing performed on the data, and transmits a control indication to the device.

Description

Control device, control method, and program

The present invention relates to technology for providing services using applications in a virtualization infrastructure.

There is a technology in which a video analysis application receives video data from a vehicle equipped with a camera, and controls automatic driving of the vehicle based on the video data.

With this technology, it is assumed that a huge number of vehicles will be controlled. Therefore, from the perspective of effective use of computational resources, the video analysis application for each vehicle is executed on a virtualization platform as needed, and the amount of resources allocated to the video analysis application is dynamically adjusted depending on the vehicle situation. It is assumed that this will be changed to

However, in the conventional technology, when changing the amount of resources allocated to a video analysis application, there was a problem in that video information was lost or service was interrupted. In other words, the conventional technology has a problem in that the amount of resources allocated to the video analysis application cannot be appropriately changed in the virtualization infrastructure.

Note that the above problem does not occur only in the field of vehicle control using video analysis applications, but can occur in any field where it is necessary to change the resource allocation amount of an application in a virtualization platform.

The present invention has been made in view of the above points, and it is an object of the present invention to provide a technology that makes it possible to appropriately change the resource allocation amount of an application in a virtualization infrastructure.

According to the disclosed technology, an application proxy that receives data transmitted from a device;
an application unit that receives the data retransmitted from the application proxy;
A control device is provided, comprising: a control proxy that receives a control request obtained based on processing of the data from the application section and transmits a control instruction to the device.

According to the disclosed technology, a technology is provided that makes it possible to appropriately change the resource allocation amount of an application in a virtualization infrastructure.

FIG. 3 is a diagram for explaining a use case. FIG. 2 is a diagram for explaining a problem. FIG. 2 is a system configuration diagram in Examples 1-1 and 1-2. FIG. 3 is a diagram for explaining the operation of a system control unit. FIG. 3 is a sequence diagram in Example 1-1. FIG. 3 is a sequence diagram in Example 1-1. FIG. 3 is a sequence diagram in Example 1-2. FIG. 3 is a sequence diagram in Example 1-2. FIG. 3 is a system configuration diagram in Example 1-3. FIG. 3 is a sequence diagram in Example 1-3. FIG. 3 is a sequence diagram in Example 1-3. FIG. 3 is a system configuration diagram in Examples 2-1 and 2-2. FIG. 3 is a sequence diagram in Example 2-1. FIG. 3 is a sequence diagram in Example 2-1. FIG. 3 is a sequence diagram in Example 2-2. FIG. 3 is a sequence diagram in Example 2-2. It is a diagram showing an example of the hardware configuration of the device.

Hereinafter, an embodiment of the present invention (this embodiment) will be described with reference to the drawings. The embodiments described below are merely examples, and embodiments to which the present invention is applied are not limited to the following embodiments.

Hereinafter, a case will be described in which the present invention is applied to control of a vehicle that performs automatic driving, but this is an example of an application of the present invention. INDUSTRIAL APPLICATION This invention is applicable to various fields not only the control of the vehicle which performs automatic driving.

(Use case assumed in the embodiment)
First, a use case assumed in this embodiment will be explained. In this embodiment, it is assumed that control such as stopping of an automatically driving vehicle is performed from a remote base. The vehicle here is not limited to a specific type, but includes, for example, a tractor, a bus, and the like.

For example, assume that the base is an edge base. The edge base is equipped with a virtualization infrastructure device. Specifically, the virtualization infrastructure device is one or more computers (which may also be called a host or a server host). The computer may be a physical machine or a virtual machine. Since the virtualization infrastructure device is a device that performs control, it may also be called a "control device."

Furthermore, in this embodiment, it is assumed that Kubernetes (registered trademark), which is a container-based orchestration tool, will be used for operational management of applications in the virtualization infrastructure device. In Kubernetes (registered trademark), applications are deployed in units called Pods. Deployed applications operate as containers within Pods. However, the application of the technology according to the present invention is not limited to environments using Kubernetes (registered trademark).

FIG. 1 shows an example of the configuration of a system related to this embodiment. FIG. 1 shows a configuration that does not include a video proxy/control proxy, which will be described later. As shown in FIG. 1, this system includes a virtualization infrastructure device 100 and a vehicle device 200. The vehicle device 200 is a device installed in a vehicle, includes a communication function, and is connected to a network via a mobile line or the like. The virtualization infrastructure device 100 is also connected to a network, and the vehicle device 200 and the virtualization infrastructure device 100 can communicate via the network. Vehicle device 200 may also be referred to as a "device."

As shown in FIG. 1, the vehicle device 200 is equipped with a camera/encoder 210 (camera and encoder) and a vehicle control system 220. The virtualization infrastructure device 100 also includes a video analysis application 10. Hereinafter, the "application" will be referred to as "APL". In this embodiment, each APL in the virtualization infrastructure device 100 operates within a Pod. In the following, it is assumed that one APL operates on one Pod, but a plurality of APLs may operate on one Pod.

Although FIG. 1 shows only one vehicle device 200 communicating with the virtualization infrastructure device 100, in reality, multiple vehicles (that is, multiple vehicle devices 200) communicate with the virtualization infrastructure device 100. communicate with.

A video analysis APL 10 in the virtualization infrastructure device 100 exists for each vehicle (vehicle device 200). Note that there may be a case where only one vehicle (vehicle device 200) communicates with the virtualization infrastructure device 100.

The camera/encoder 210 is an H. The camera video is encoded using a high compression codec that supports interframe compression such as H.264. Further, the camera/encoder 210 transmits the encoded video data to the video analysis APL 10 of the virtualization infrastructure device 100 through one-way communication such as RTP/UDP.

The video analysis APL 10 decodes the video from the video data received from the camera/encoder 210 and analyzes the video. The video analysis referred to here includes detection of obstacles (pedestrians, etc.) within the camera field of view, detection of deviations in the running position in the width direction of the road, and the like. The video analysis APL 10 transmits vehicle control data determined based on the analysis results to the vehicle control system 220. Examples of control include "emergency stop due to pedestrian detection" and "temporary change in traveling direction 5 degrees counterclockwise for 3 seconds to correct travel position shift".

The vehicle control system 220 performs actuation on the vehicle based on the vehicle control data received from the video analysis APL 10.

In the use case according to this embodiment, the following assumptions are made in particular. Note that the technology according to the present invention is not limited to the following assumptions.

・The number of vehicles accommodated is large-scale, exceeding 100,000.

・Vehicles are not in operation 24 hours a day, 365 days a year, and repeatedly start and stop running at any time.

- The video analysis APL 10 exists for each vehicle and is deployed on the virtualization infrastructure device 100. In particular, in order to effectively utilize computer resources in the virtualization infrastructure device 100 and reduce startup/termination and operation overhead, it is deployed in a container format rather than a bare metal VM.

- The amount of resources (number of CPUs, amount of memory, etc.) required by the video analysis APL 10 changes depending on the vehicle situation (driving position/surrounding situation, speed, etc.).

(Requirements)
The requirements for this use case are as follows.

・In order to effectively utilize the resources on the virtualization infrastructure device 100 side, resources for the video analysis APL 10 are secured only when the vehicle starts running/stops, and the video analysis APL 10 is deployed.

・In order to effectively utilize the resources on the virtualization infrastructure device 100 side, the resources for the video analysis APL 10 are increased or decreased at any time according to the situation of the running vehicle.

・In order to ensure safety, automatic driving and remote monitoring services will not be momentarily interrupted for more than a certain period of time even when the resources of the video analysis APL 10 for a running vehicle are increased or decreased. Specifically, the instantaneous interruption time does not exceed 1 second.

- In order to reduce management costs and power distribution costs on the virtualization infrastructure device 100 side, the number of real servers on which the video analysis APL 10 on the virtualization infrastructure device 100 is operating is minimized.

(Problems when using conventional technology for this use case)
It is conceivable to use conventional techniques to realize a system that satisfies the requirements in the use case described above. However, as explained below, there are problems with the conventional techniques, and therefore it is not possible to realize a system that satisfies the requirements of the above-mentioned use cases using only the conventional techniques.

Although the technologies disclosed in the documents referred to below are publicly known, explanations of the utilization of the technologies disclosed in the documents, explanations of technologies envisioned by the combination of technologies disclosed in the documents, and explanations of the technologies disclosed in the documents are provided. Explanations of the benefits of technology, explanations of problems, etc. are not publicly known.

Non-Patent Document 2 discloses a technique for performing live migration between VMs in a VM-based server virtualization infrastructure. Live migration is a technology in which APL copies the state (disk memory contents, etc.) of a VM in operation to a new VM and continues operation. By utilizing this technology, when the required resources of APL change, it is possible to start up a new VM, copy the state from the old VM to the new VM, and continuously provide services.

However, when using a VM-based server virtualization platform and live migration technology on the platform for this use case, there are the following two problems.

・VM has a longer waiting time when running/terminating compared to containers. Furthermore, processing overhead during operation is large due to HW emulation and the like.

・During live migration, a momentary interruption of about a few seconds occurs.

Non-Patent Document 1 discloses Kubernetes (registered trademark), which is a container-based server virtualization infrastructure. Kubernetes (registered trademark) has a function to orchestrate container-format APLs, and functions such as Topology Manager can allocate resources to deployed APLs. Additionally, in Non-Patent Document 3, for a deployment APL in Kubernetes (registered trademark) (the unit of APL deployment is called a Pod), a terminal whose destination is a representative identifier (e.g. Master Node IP address/port number) Techniques for forwarding request packets for outgoing communications are shown. By utilizing these techniques, it is possible to launch a new Pod and distribute communications addressed to the representative identifier to the new Pod when the required resources of the APL change.

By using Kubernetes (registered trademark) shown in Non-Patent Document 1, there are the following benefits.

・Compared to VM and bare metal, containers have shorter waiting time during operation/termination, and compared to VM, the processing overhead due to HW emulation during operation is smaller.

However, when Non-Patent Document 3 is used in this use case in combination with Prior Art 1, there are the following two problems.

- Since the state is not inherited between the old Pod and the new Pod, it is necessary to reconnect the gRPC session between the vehicle device and the video analysis APL, and the vehicle device side software for reconnection becomes complicated.

- Since the state is not inherited between the old Pod and the new Pod, the H. Since the key frame information of a high compression codec that supports interframe compression such as H.264 cannot be inherited, it is not possible to correctly reconstruct the video frames up to the arrival of the next key frame.

Non-Patent Document 4 discloses a proposal for a dynamic resource allocation change function during Pod operation, which is one of the KEP (Kubernetes Enhancement Proposals) in the OSS community that develops Kubernetes (registered trademark). . By utilizing this technology and the technology disclosed in Non-Patent Document 1, Pods can be allocated within the remaining resource amount of the server host (the computer on which APL is running) without stopping the operation of APL. It is possible to change the amount of resources.

However, when the technology of Non-Patent Document 4 is used in this use case in combination with the technology of Non-Patent Document 1, there are the following problems.

- Since it is not possible to relocate Pods between different server hosts, if the remaining amount of resources on the host is exhausted, the amount of allocated resources cannot be increased.

- Since it is impossible to relocate Pods between different server hosts, it is not possible to merge the used server hosts when the amount of consumed resources of the entire system decreases.

In order to change the resource allocation using conventional technology, it is conceivable to stop using a Pod (old Pod) that lacks resources and generate a new Pod (new Pod) with the desired amount of resources. In that case, there will be an interval of about 10 seconds between the disappearance (stopping of use) of the old Pod and the start of use of the new Pod, making it impossible to satisfy the required conditions.

FIG. 2 shows a summary of the above-mentioned problems that occur when the conventional technology is assumed to be used for this user case.

(Summary of embodiment)
In this embodiment, a proxy function is provided between the camera/encoder 210 and vehicle control system 220 installed in the vehicle and the video analysis APL provided in the virtualization infrastructure device 100. This hides the activation of a new Pod and the termination of the old Pod of the video analysis APL, and solves the problem caused by states such as memory contents not being inherited between the old and new Pods.

Specifically, it is possible to prevent the messaging session that is terminated by the vehicle device 200 from being disconnected when a new Pod is generated or when an old Pod is terminated. Furthermore, loss of frame information can be prevented by video switching timing control or codec conversion.

Hereinafter, the configuration and operation according to this embodiment will be explained in detail using Examples 1 and 2. Specifically, Examples 1-1 to 2-2 below will be explained in order.

(1) Example 1-1: Variation 1 (key frame securing method) of Example 1 (video one-stage proxy method)
(2) Example 1-2: Variation 2 (interframe decompression distribution method) of Example 1 (video one-stage proxy method)
(3) Example 1-3: Variation 3 (interframe decompression storage method) of Example 1 (video one-stage proxy method)
(4) Example 2-1: Variation 1 (key frame securing method) of Example 2 (video two-stage proxy method)
(5) Example 2-2: Variation 2 (interframe decompression method) of Example 2 (video two-stage proxy method)
(Example 1-1)
First, Example 1-1 will be explained. Example 1-1 is variation 1 (key frame securing method) of Example 1 (video one-stage proxy method). FIG. 3 shows an example of a system configuration in Example 1-1. A description will mainly be given of a configuration that is different from the configuration described with reference to FIG.

As shown in FIG. 3, in the system of Example 1-1, the virtualization infrastructure device 100 is equipped with one video proxy 105 and one control proxy 107 for each vehicle device 200. Note that the video proxy is an example of an "application proxy."

Further, FIG. 3 shows the video analysis APL 101 before switching and the video analysis APL 103 after switching. Note that the video analysis APL 101 before switching may be described as "old video analysis APL 101", and the video analysis APL 103 after switching may be described as "new video analysis APL 103".

Each proxy and each APL operates as a container within a Pod on a worker node (described as "worker" in the figure). In this embodiment, each proxy operates on one worker node, and each APL also operates on one worker node. However, multiple proxies and multiple APLs may exist in one worker node. A worker node may be a physical machine or a virtual machine.

Additionally, the virtualization infrastructure device 100 is equipped with a system control unit 150. Note that the system control unit 150 may be provided outside the virtualization infrastructure device 100. Here, it is assumed that the system control unit 150 is provided within the virtualization infrastructure device 100.

The system control unit 150 includes, for example, the function of a Kubernetes (registered trademark) master node, and is configured to assign a desired amount of resources to Pods (i.e., APL) can be generated.

An example of the functions of the system control unit 150 will be described with reference to FIG. 4. In the example of FIG. 4, it is assumed that the resource amount of the video analysis APL 13 operating in the Pod 12 on the worker node has become insufficient compared to the processing amount. At this time, when the user instructs the system control unit 150 to generate a new video analysis APL 16 that uses a desired amount of resources, the system control unit 150 generates a new image analysis APL 16 on the worker node 14 that has a sufficient amount of resources. Generate Pod 15 and video analysis APL 16. Note that if the worker node 11 has a sufficient amount of remaining resources, the Pod 15 may be generated on the worker node 11 as a result.

As described above, APL is generated (deployed) in units of Pods, and APLs operate on Pods; however, in the following figures and explanations, Pods may not be explicitly described for convenience of description. Note that a Pod on which an application operates may be referred to as an "application unit."

In the configuration of Example 1-1 shown in FIG. 3, a video stream continues to be transmitted from the camera/encoder 210 of the vehicle device 200 to the video proxy 105 while the vehicle is in operation. The video proxy 105 relays the video stream to the video analysis APL 101 (after switching, the video analysis APL 103).

The vehicle control system 220 in the vehicle device 200 and the control proxy 107 in the virtualization infrastructure device 100 are connected by gRPC, which is a session maintenance type communication protocol, and Server Streaming RPC, that is, the server (control proxy 107 here) is used. This form allows push communication from. The control proxy 107 and the video analysis APL 101 (after switching, the video analysis APL 103) communicate in both directions using HTTP/Rest, a protocol that establishes a connection each time the communication occurs.

The reasons for using the above protocol are as follows (1) to (3).

(1) Communication between the vehicle control system 220 and the control proxy 107 involves a high delay because it goes through the WAN, and in order to save the time required for communication, it is desirable to be able to omit connection establishment processing each time communication occurs;
(2) Low delay between control proxy 107 and video analysis APL 101/103 within the same data center;
(3) There is no need to reconnect the connection when restarting the video analysis APL 101/103.

The video analysis APL 101/103 decodes the video stream received from the video proxy 105 and performs video analysis for each frame image. The video analysis APL 101/103 notifies the control proxy 107 of the video analysis results as necessary.

Next, the operation of the system in each of the following cases (a) and (b) will be explained with reference to sequence diagrams.

(a) When the video analysis APL 101 analyzes the camera video and instructs the vehicle control system 220 to make an emergency stop when an obstacle is detected (b) Resource allocation of the video analysis APL while the vehicle is running When performing processing to increase the amount from {2 CPU, 4 GB RAM} to {4 CPU, 8 GB RAM} First, the operation in case (a) above will be described with reference to the sequence diagram of FIG. 5.

At S101, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and sends it to the video proxy 105 using RTP/UDP.

In S102, the video proxy 105 continuously rewrites the IP header of the RTP/UDP packet received from the camera/encoder 210 and retransmits it to the video analysis APL 101.

In S103, the video analysis APL 101 captures the plurality of received RTP/UDP packets into a buffer and reconstructs one frame worth of image.

In S104, the video analysis APL 101 performs obstacle detection analysis on one frame of the image. Any algorithm may be used as the obstacle detection algorithm. As an example, an intra-image rectangle detection function using off-the-shelf technology (OpenCV) can be used. Here, the video analysis APL 101 recognizes that a rectangular object exists within the image.

In S105, the video analysis APL 101 requests the control proxy 107 to issue an emergency stop instruction to the vehicle control system 220 by calling the HTTP/REST API.

In S106, the control proxy 107 instructs the vehicle control system 220 on the vehicle device 200 to perform an emergency stop using a server push using the established gRPC connection.

Next, the operation in case (b) above will be explained with reference to the sequence diagram of FIG. 6.

In S201, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and sends it to the video proxy 105 using RTP/UDP.

In S202, the video proxy 105 continuously rewrites the IP headers of the RTP/UDP packets received from the camera/encoder 210 and retransmits them to the video analysis APL 101.

In S203, the user instructs the system control unit 150 to increase the resource amount of the video analysis APL of the camera video related to a certain vehicle identifier (for example, {2 CPU, 4 GB RAM} → {4 CPU, 8 GB RAM}). .

In S204, the system control unit 150 newly generates a video analysis APL 103 with {4 CPU, 8 GB RAM} together with the Pod.

In S205, the system control unit 150 periodically views the log of the Pod of the new video analysis APL 103 and waits until normal operation of the video analysis APL 103 can be confirmed. Once you have confirmed normal operation, proceed to the next step.

In S206, the system control unit 150 notifies the video proxy 105 that the video retransmission destination will be changed when the next key frame arrives, and the video proxy 105 waits.

When the video proxy 105 receives the first key frame after S206, in S207, the video proxy 105 notifies the system control unit 150 of the arrival of the key frame, and performs video analysis of the RTP/UDP packet containing the key frame. Retransmission to APL 101 is suppressed.

As mentioned above, the video proxy 105 needs to determine whether the received video data is a key frame or not. The format of the RTP payload of an H.264 video stream is specified in IETF RFC 6184, so key frames can be identified according to this specification. Specifically, if the value of the NAL Unit Header stored at the beginning of the RTP payload is 5, the data can be determined to be a key frame.

In S208, the system control unit 150 instructs both the video proxy 105 and the control proxy 107 to switch the video analysis APL at the same time. In other words, an instruction is given to switch the connection destination from the video analysis APL 101 to the video analysis APL 103.

In S209, the video proxy 105 changes the retransmission destination of the RTP/UDP packet to the Pod of the new video analysis APL 103, and resumes retransmission.

Simultaneously with S209, in S210, the control proxy 107 changes the communication partner of the vehicle control information to the Pod of the new video analysis APL 103. That is, from now on, the control proxy 107 will not accept communication with the Pod of the old video analysis APL 101.

(Example 1-2)
Next, Example 1-2 will be explained. Example 1-2 is variation 2 (interframe decompression distribution method) of Example 1 (video one-stage proxy method). The system configuration of Example 1-2 is the same as that of Example 1-1, and is as shown in FIG. 3.

The operation of the system in each of the following cases (a) and (b) will be explained with reference to sequence diagrams.

(a) When the video analysis APL 101 analyzes the camera video and instructs the vehicle control system 210 to make an emergency stop when an obstacle is detected (b) Resource allocation of the video analysis APL while the vehicle is running When performing processing to increase the amount from {2 CPU, 4 GB RAM} to {4 CPU, 8 GB RAM} First, the operation in case (a) above will be described with reference to the sequence diagram of FIG. 7.

At S301, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and sends it to the video proxy 105 using RTP/UDP.

In S302, the video proxy continuously performs video decoding on the RTP/UDP packets received from the camera/encoder 210. Furthermore, the decoded video is re-encoded using an interframe non-compression codec (MJPEG, etc.). Furthermore, the re-encoded video data is packetized and re-transmitted to the video analysis APL 101.

The subsequent steps S303 to S306 are the same as steps S103 to S106 in Example 1-1.

Next, the operation in case (b) above will be explained with reference to the sequence diagram of FIG.

At S401, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and sends it to the video proxy 105 using RTP/UDP.

In S402, the video proxy 105 continuously decodes the RTP/UDP packets received from the camera/encoder 210. Furthermore, the decoded video is re-encoded using an interframe non-compression codec (MJPEG, etc.). Furthermore, the re-encoded video data is packetized and re-transmitted to the video analysis APL 101.

In S403, the user instructs the system control unit 150 to increase the resource amount of the video analysis APL of the camera video related to a certain vehicle identifier (for example, {2 CPU, 4 GB RAM} → {4 CPU, 8 GB RAM}). .

In S404, the system control unit 150 newly generates a Pod for the video analysis APL 103, which is {4 CPU, 8 GB RAM}.

In S405, the system control unit 150 periodically views the log of the Pod of the new video analysis APL 103 and waits until it can confirm the normal operation of the video analysis APL 103. Once you have confirmed normal operation, proceed to the next step.

In S406, the system control unit 150 instructs both the video proxy 105 and the control proxy 107 to switch the video analysis APL at the same time.

In S407, the video proxy 105 starts transmitting the next RTP/UDP packet (the first RTP/UDP packet after S406) to the new video analysis APL 103. In S407, as explained in S402, an RTP/UDP packet in which the video data re-encoded using the interframe non-compression codec is packetized is transmitted.

In S408, simultaneously with S407, the control proxy 107 changes the communication partner of the vehicle control information to the Pod of the new video analysis APL 103. That is, from now on, communication with the Pod of the old video analysis APL 101 will not be accepted.

(Example 1-3)
Next, Example 1-3 will be explained. Example 1-3 is variation 3 (interframe decompression storage method) of Example 1 (video one-stage proxy method). FIG. 9 shows an example of a system configuration in Example 1-3. As shown in FIG. 9, an image frame storage section 160 is added to the configuration shown in FIG. Image frame storage section 160 may also be referred to as a "data storage section."

While the vehicle is in operation, a video stream continues to be sent from the camera/encoder 210 to the video proxy 105. The video proxy 105 decodes the received video stream and continues to store the latest image frames in the image frame storage unit 160. At this time, the image frame identifier (for example, file name) may be incremented by 1 and saved, or the latest image may be always overwritten and saved.

As in Example 1-1, the vehicle control system 220 on the vehicle device 200 and the control proxy 107 are connected by gRPC, which is a session-maintaining communication protocol, and Server Streaming RPC, that is, the server (control proxy 107 ) allows for push communication. Bidirectional communication is performed between the control proxy 107 and the video analysis APL 101/103 using HTTP/Rest, a protocol that establishes a connection each time communication occurs.

The video analysis APL 101/103 acquires the latest image frame from the image frame storage unit 160 at a desired cycle and performs video analysis on it. The video analysis APL 101/103 notifies the control proxy 107 of the video analysis results as necessary.

(a) When the video analysis APL 101 analyzes the camera video and instructs the vehicle control system 220 to make an emergency stop when an obstacle is detected (b) Resource allocation of the video analysis APL while the vehicle is running Case of increasing the amount from {2 CPU, 4 GB RAM} to {4 CPU, 8 GB RAM} First, the operation in case (a) above will be explained with reference to the sequence diagram of FIG. 10.

At S501, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and sends it to the video proxy 105 using RTP/UDP.

In S502, the video proxy 105 continuously decodes the received video stream and stores the latest image frame in the image frame storage unit 160. For example, if the frame rate of the received video stream is 50 fps, an image frame will be saved every 20 ms.

In S503, the video analysis APL 101 acquires the latest image frame at the time from the image frame storage unit 160, for example, at a cycle of 100 ms.

In S504, the video analysis APL 101 performs obstacle detection analysis on the image acquired in S503. Any algorithm may be used as the obstacle detection algorithm. As an example, an intra-image rectangle detection function using off-the-shelf technology (OpenCV) can be used. Here, the video analysis APL 101 recognizes that a rectangular object exists within the image.

S505-S506 are the same as S105-S106.

Next, the operation in case (b) will be explained with reference to the sequence diagram in FIG.

At S601, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and sends it to the video proxy 105 using RTP/UDP.

In S602, the video proxy 105 continuously decodes the received video stream and stores the latest image frame in the image frame storage unit 160. If the frame rate of the received video stream is 50 fps, an image frame will be saved every 20 ms.

In S603, the video analysis APL 101 continuously acquires image frames from the image frame storage unit 160 at a cycle of, for example, 100 ms, and executes video analysis.

In S604, the user instructs the system control unit 150 to increase the amount of resources of the video analysis APL 101 of the camera video related to a certain vehicle identifier (for example, {2 CPU, 4 GB RAM} → {4 CPU, 8 GB RAM}). .

In S605, the system control unit 150 newly generates a video analysis APL 103 with {4 CPU, 8 GB RAM} together with the Pod.

In S606, the new video analysis APL 103 starts acquiring image frames (from the image frame storage unit 160) at a cycle of 50 ms, for example, and starts executing video analysis.

In S607, the system control unit 150 periodically views the log of the Pod of the new video analysis APL 103 and waits until the normal operation of the video analysis APL 103 can be confirmed. If you can wait, move on to the next step.

In S608, the system control unit 150 instructs the control proxy 107 to switch the video analysis APL.

In S609, the control proxy 107 changes the communication partner of the vehicle control information to the new video analysis APL 103. That is, from now on, communication with the old video analysis APL 101 will not be accepted.

(Example 2-1)
Next, Example 2-1 will be explained. Example 2-1 is variation 1 (key frame securing method) of Example 2 (video two-stage proxy method). FIG. 12 shows an example of a system configuration in Example 2-1. As shown in FIG. 12, the configuration of Example 2-1 has a configuration in which a video proxy 2a (109) and a video proxy 2b (111) are added to the configuration shown in FIG. That is, the virtualization infrastructure device 100 of the second embodiment includes three

video proxies

105, 109, and 111 and one control proxy 107.

The three video proxies may be deployed on the same host (that is, the same computer) in the virtualization infrastructure device 100, or may be deployed on different hosts (that is, two or three different computers). When three video proxies are deployed on the same host, there is an effect of reducing the communication delay between the multi-stage video proxies. Although FIG. 12 shows an example in which the video proxy is configured in two stages, the video proxy may be configured in three or more stages. Even when the video proxy is configured in three or more stages, it is possible to deploy them on the same host.

While the vehicle is in operation, a video stream continues to be sent from the camera/encoder 210 to the video proxy 1 (105). Further, the video proxy 1 (105) always continues to transmit the video stream to the video proxy 2a (109) and the video proxy 2b (111). The video proxy 2a (109) and the video proxy 2b (111) each relay the video stream to the video analysis APL that they are in charge of.

Similar to Embodiment 1-1, the vehicle control system 220 on the vehicle device 200 and the control proxy 107 are connected by gRPC, which is a session maintenance type communication protocol, and Server Streaming RPC, that is, the server (control proxy 107) is possible. Bidirectional communication is performed between the control proxy 107 and the video analysis APL 101/103 using HTTP/Rest, a protocol that establishes a connection each time communication occurs.

The video analysis APL 101/103 decodes the received video stream and performs video analysis for each frame image. The video analysis APL 101/103 notifies the control proxy 107 of the video analysis results as necessary.

(a) When the video analysis APL 101 analyzes the camera video and instructs the vehicle control system 220 to make an emergency stop when an obstacle is detected (b) Resource allocation of the video analysis APL while the vehicle is running When performing processing to increase the amount from {2 CPU, 4 GB RAM} to {4 CPU, 8 GB RAM} First, the operation in case (a) above will be described with reference to the sequence diagram of FIG. 13.

At S701, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and transmits it to the video proxy 1 (105) using RTP/UDP.

In S702, the video proxy 1 (105) continuously rewrites the IP header of the RTP/UDP packet received from the camera/encoder 210 and sends it to the video proxy 2a (109) and the video analysis proxy 2b (111). and resend.

In S703, the video proxy 2a (109) continuously rewrites the IP header of the RTP packet received from the video proxy 1 (105) and retransmits it to the video analysis APL 101. Note that at this time, the video proxy 2b (111) discards the RTP packet received from the video proxy 1 (105). S704 to S707 are the same as S103 to S106 in Example 1-1 (FIG. 5).

Next, the operation in case (b) above will be explained with reference to the sequence diagram of FIG. 14.

In S801, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and transmits it to the video proxy 1 (105) using RTP/UDP.

In S802, the video proxy 1 (105) continuously rewrites the IP header of the RTP/UDP packet received from the camera/encoder 210 and sends it to the video proxy 2a (109) and the video analysis proxy 2b (111). and resend.

In S803, the video proxy 2a (109) continuously rewrites the IP header of the RTP packet received from the video proxy 1 (105) and retransmits it to the video analysis APL 101. Note that at this time, the video proxy 2b (111) discards the RTP packet received from the video proxy 1 (105).

In S804, the user instructs the system control unit 150 to increase the resource amount of the video analysis APL of the camera video related to a certain vehicle identifier (for example, {2 CPU, 4 GB RAM} → {4 CPU, 8 GB RAM}). .

In S805, the system control unit 150 newly generates the video analysis APL 103 and Pod, which are {4 CPU, 8 GB RAM}.

In S806, the system control unit 150 periodically views the log of the newly generated Pod of the video analysis APL 103 and waits until normal operation of the video analysis APL 103 can be confirmed. Once you have confirmed normal operation, proceed to the next step.

In S807, the system control unit 150 notifies the video proxy 2b (111) that it will start retransmitting the video to the new video analysis APL 103, which is in charge of it, from the arrival of the next key frame, and Proxy 2b (111) waits.

After S807, in S808, if the video proxy 2b (111) receives the key frame first, the video proxy 2b (111) notifies the system control unit 150 of the arrival of the key frame, and also sends an RTP containing the key frame. /Starts retransmission of the UDP packet to the new video analysis APL 103.

Note that the video proxy 2b (111) needs to determine whether or not the received video data is a key frame. H. Since the format of the RTP payload of the H.264 video stream is specified in IETF RFC 6184, key frames can be determined according to the specification. Specifically, if the value of the NAL Unit Header stored at the beginning of the RTP payload is 5, it is determined that the data is a key frame.

In S809, the system control unit 150 instructs the control proxy 107 to switch the video analysis APL.

In S810, the control proxy 107 changes the communication partner of the vehicle control information to the Pod of the new video analysis APL 103. That is, from now on, communication with the Pod of the old video analysis APL 101 will not be accepted.

(Example 2-2)
Next, Example 2-2 will be explained. Example 2-2 is variation 2 (interframe decompression method) of Example 2 (video two-stage proxy method). The system configuration of Example 2-2 is the same as that of Example 2-1, and is as shown in FIG. 12.

(a) When the video analysis APL 101 analyzes the camera video and instructs the vehicle control system 220 to make an emergency stop when an obstacle is detected (b) Resource allocation of the video analysis APL while the vehicle is running When performing processing to increase the amount from {2 CPU, 4 GB RAM} to {4 CPU, 8 GB RAM} First, the operation in case (a) above will be described with reference to the sequence diagram of FIG. 15.

In S901, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and transmits it to the video proxy 1 (105) using RTP/UDP.

In S902, the video proxy 1 (105) continuously decodes the RTP/UDP packets received from the camera/encoder 210. Furthermore, the decoded video is re-encoded using an interframe non-compression codec (such as MJPEG). Furthermore, the re-encoded video data is packetized and retransmitted to the video proxy 2a (109) and the video proxy 2b (111). Subsequent steps S904 to S907 are the same as steps S703 to S707 in Example 2-1.

Next, the operation in case (b) above will be explained with reference to the sequence diagram of FIG. 16.

In S1001, the camera/encoder 210 continuously encodes video traffic using an interframe compression codec (H.264) and transmits it to the video proxy 1 (105) using RTP/UDP.

In S1002, the video proxy 1 (105) continuously decodes the RTP/UDP packets received from the camera/encoder 210. Furthermore, the decoded video is re-encoded using an interframe non-compression codec (MJPEG, etc.). Furthermore, the re-encoded video data is packetized and retransmitted to the video proxy 2a (109) and the video proxy 2b (111).

In S1003, the video proxy 2a (109) continuously rewrites the IP header of the RTP packet received from the video proxy 1 (105) and retransmits it to the video analysis APL 101. Note that at this time, the video proxy 2b (111) discards the RTP packet received from the video proxy 1 (105).

In S1004, the user instructs the system control unit 150 to increase the resource amount of the video analysis APL of the camera video related to a certain vehicle identifier (for example, {2 CPU, 4 GB RAM} → {4 CPU, 8 GB RAM}). .

In S1005, the system control unit 150 newly generates the video analysis APL 103 and Pod, which are {4 CPU, 8 GB RAM}.

In S1006, the system control unit 150 periodically views the log of the Pod of the new video analysis APL 103 and waits until normal operation of the video analysis APL 103 can be confirmed. Once you have confirmed normal operation, proceed to the next step.

In S1007, the system control unit 150 instructs the video proxy 2b (111) to start retransmitting the video to the new video analysis APL 103 that is in charge of it. At the same time, it instructs the control proxy 107 to switch to the new video analysis APL 103.

In S1008, the video proxy 2b (111) transmits the next RTP/UDP packet to the new video analysis APL 103. Note that this packet is a packetized packet of video data that has been re-encoded using an interframe non-compression codec.

Simultaneously with S1008, in S1009, the control proxy 107 changes the communication partner of the vehicle control information to the new video analysis APL 103. That is, from now on, communication with the old video analysis APL 101 will not be accepted.

(Hardware configuration example)
The virtualization infrastructure device 100 can be realized by having one or more computers execute a program. This computer may be a physical computer or a virtual machine.

That is, the virtualization infrastructure device 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the virtualization infrastructure device 100. It is. The above program can be recorded on a computer-readable recording medium (such as a portable memory) and can be stored or distributed. It is also possible to provide the above program through a network such as the Internet or e-mail.

FIG. 17 is a diagram showing an example of the hardware configuration of the computer. The computer in FIG. 17 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are connected to each other by a bus B.

A program that realizes processing on the computer is provided, for example, on a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily need to be installed from the recording medium 1001, and may be downloaded from another computer via a network. The auxiliary storage device 1002 stores installed programs as well as necessary files, data, and the like.

The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when there is an instruction to start the program. The CPU 1004 implements functions related to the virtualization infrastructure device 100 according to programs stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network or the like. A display device 1006 displays a GUI (Graphical User Interface) and the like based on a program. The input device 1007 is composed of a keyboard, a mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. An output device 1008 outputs the calculation result.

(Effects of embodiment)
As described above, this embodiment employs a synchronous traffic switching technique using a video proxy and a control proxy. As a result, while the automatic vehicle driving/remote monitoring service is being executed, the amount of resources allocated to the server side APL (e.g. video analysis APL) can be changed at any time, and at that time, service interruptions or terminal application (e.g. The need for reconnection processing of the vehicle control system can be suppressed.

As a result, a container-type server virtualization infrastructure with less processing overhead than a VM can be configured while minimizing the number of idle servers in operation. As a result, stable continuation of services can be achieved while reducing costs.

In particular, in Example 1-1 (Variation 1 (key frame securing method) of Example 1 (video one-stage proxy method)), the arrival of the beginning of the key frame in the video proxy is used as an opportunity for the video at the video retransmission destination to The analysis APL is to be changed. If this device is not used, the retransmission destination video analysis APL will switch from an intermediate packet of a key frame or a certain difference frame, and the new video analysis APL will correctly reconstruct the image frame until the next key frame arrives. The problem is that it is not possible. However, the invention in Example 1-1 has the effect of preventing this problem.

In addition, in Example 1-2 (Variation 2 (interframe decompression distribution method) of Example 1 (video one-stage proxy method)), the codec of the received video is changed in the video proxy, and We decided to send the video encoded by the codec to the video analysis APL. Without this measure, problems similar to those described above will occur. However, this problem can be prevented by this technique.

In addition, in Example 1-2, for each image frame constituting a video encoded by an interframe non-compression type codec, there is a possibility that the video analysis APL of the retransmission destination will be switched from an intermediate packet of this image frame. In that case, there is a problem that the new video analysis APL cannot reconstruct the image frame. However, this reconstruction failure event is limited to only one image frame. Further, in the embodiment 1-2, the key frame detection function in the embodiment 1-1 is not required, and there is an advantage that the implementation is simple.

In addition, in Example 1-3 (Variation 3 (inter-frame decompression storage method) of Example 1 (video one-stage proxy method)), the video proxy decodes the received video and stores the image frame in the storage unit. , the video analysis APL acquires this image frame as necessary. This idea has the effect of being able to prevent damage to one image frame, which was a constraint in Example 1-2.

Furthermore, in Example 2-1 (Variation 1 (key frame securing method) of Example 2 (two-stage video proxy method)), unlike Example 1-1, the video proxy is configured in multiple stages. Specifically, a video proxy 1 (105) that first receives video from the camera/encoder 210 on the vehicle, a video proxy 2a (109), and a video proxy 2b (109) that retransmit video to the old/new video analysis APL, respectively. 111) exists.

If the video proxy is implemented in such a way that it takes time to change the video retransmission destination, there is a problem in Embodiment 1-1 that service interruption will occur. However, in Example 2-1, the start of video retransmission to the new video analysis APL can be carried out independently of the video retransmission to the old video analysis APL, so this problem can be prevented. be.

<Additional notes>
This specification discloses at least the following control device, control method, and program.
(Additional note 1)
an application proxy that receives data sent from the device;
an application unit that receives the data retransmitted from the application proxy;
A control device comprising: a control proxy that receives a control request obtained based on processing of the data from the application unit and transmits a control instruction to the device.
(Additional note 2)
After a new application section is generated in the control device,
The control device according to appendix 1, wherein connection destinations of the application proxy and the control proxy are switched from the application unit to the new application unit in synchronization.
(Additional note 3)
The control device according to Supplementary Note 2, wherein the connection destination of the application proxy and the control proxy is switched from the application section to the new application section when the application proxy receives specific data from the device.
(Additional note 4)
The control device according to supplementary note 1, wherein the application proxy includes a plurality of proxies forming a plurality of stages.
(Additional note 5)
The data transmitted from the device is encoded data encoded using a first method, and the application proxy decodes the encoded data received from the device and converts the decoded data into a first method different from the first method. The control device according to Supplementary Note 1, wherein the control device re-encodes the data using two methods and transmits the re-encoded data to the application unit.
(Additional note 6)
The data transmitted from the device is encoded data, and the control device further includes a data storage unit,
The application proxy decodes the encoded data received from the device and stores the decoded data in the data storage unit, and the application unit acquires the data from the data storage unit and performs analysis. The control device according to item 1.
(Supplementary Note 7)
A control method executed by a control device including an application proxy, an application unit, and a control proxy,
the application proxy receiving data sent from a device;
the application unit receiving the data retransmitted from the application proxy;
A control method comprising: the control proxy receiving a control request obtained based on processing of the data from the application unit, and transmitting a control instruction to the device.
(Supplementary Note 8)
A non-temporary storage medium storing a program for causing a computer to function as the control device according to any one of Supplementary Notes 1 to 6.

The present embodiment has been described above, but the present invention is not limited to this specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims.

100

Virtualization infrastructure device

101, 103 Video analysis APL
105, 109, 111 Video proxy 107 Control proxy 150 System control section 160 Image frame storage section 200 Vehicle device 210 Camera/encoder 220 Vehicle control system 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU
1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

an application proxy that receives data sent from the device;
an application unit that receives the data retransmitted from the application proxy;
A control device comprising: a control proxy that receives a control request obtained based on processing of the data from the application unit and transmits a control instruction to the device.
After a new application section is generated in the control device,
The control device according to claim 1, wherein connection destinations of the application proxy and the control proxy are switched from the application section to the new application section in synchronization.
The control device according to claim 2, wherein the connection destinations of the application proxy and the control proxy are switched from the application unit to the new application unit when the application proxy receives specific data from the device.
The control device according to claim 1, wherein the application proxy includes a plurality of proxies forming a plurality of stages.
The data transmitted from the device is encoded data encoded using a first method, and the application proxy decodes the encoded data received from the device and converts the decoded data into a first method different from the first method. The control device according to claim 1, further comprising: re-encoding the data using two methods, and transmitting the re-encoded data to the application unit.
The data transmitted from the device is encoded data, and the control device further includes a data storage unit,
The application proxy decodes the encoded data received from the device and stores the decoded data in the data storage unit, and the application unit acquires data from the data storage unit and performs analysis. The control device according to item 1.
A control method executed by a control device including an application proxy, an application unit, and a control proxy,
the application proxy receiving data sent from a device;
the application unit receiving the data retransmitted from the application proxy;
A control method comprising: the control proxy receiving a control request obtained based on processing of the data from the application unit, and transmitting a control instruction to the device.
A program for causing a computer to function as the control device according to any one of claims 1 to 6.