JP2020027956A - Processing device - Google Patents

Abstract

To reduce, in a remote operation, a delay from when an operation signal is received to when an operated device is executed, and improve the operability.SOLUTION: A processing device of an operated device that generates and outputs a control signal corresponding to an operation signal indicating an instruction of a remote operator includes a prediction circuit that predicts the next instruction issued by the operator on the basis of peripheral information of the operated device and generates a predicted operation signal based on the prediction, and a control signal generation circuit that generates the control signal based on the operation signal or the predicted operation signal, and the control signal generation circuit outputs a first control signal generated on the basis of the predicted operation signal before receiving the operation signal when an error of the predicted operation signal with respect to the operation signal is within a first range.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a processing device for remote control.

  When performing remote control via communication, there is a problem that a delay occurs between the operating device and the operated device, and the operability deteriorates. Conventionally, this delay has been a problem of communication delay. For example, Patent Literature 1 relates to a space robot operation support system that solves a problem that delay in communication affects operability.

JP-A-1-97591

  In remote operation, not only a delay related to communication as shown in Patent Document 1, but also a delay that is a time required for an operation instruction from an operating device to control and execute a driving unit via a processing device of an operated device. There is. That is, an operation signal from the operating device reaches the operated device, generates a control signal for controlling the driving unit via the processing device in the non-operating device, and controls the driving unit by this control signal until the driving unit is controlled. It is a delay. Since this delay deteriorates the operability of remote control, a solution is desired.

  SUMMARY OF THE INVENTION It is an object of the present invention to provide a processing device that reduces the delay from the receipt of an operation signal to the execution of an operated device and improves the operability in remote operation.

  The present invention is a processing device of an operated device that generates and outputs a control signal corresponding to an operation signal indicating an instruction of a remote operator, and the operator can perform a next operation based on peripheral information of the operated device. A prediction circuit for predicting an instruction to be issued and generating a prediction operation signal based on the prediction, and a control signal generation circuit for generating a control signal based on the operation signal or the prediction operation signal, wherein an error of the prediction operation signal with respect to the operation signal is reduced. A first control signal that is generated based on the predicted operation signal before the control signal generation circuit receives the operation signal when the value is within the range of 1.

  ADVANTAGE OF THE INVENTION According to the processing apparatus of this invention, operability can be improved by improving the processing speed and efficiency of remote control compared with the former.

It is a block diagram showing a remote control system provided with a processing unit of the present invention. FIG. 2 is a detailed block diagram of an operation-side device of a remote operation system including the processing device of the present invention. FIG. 2 is a detailed block diagram of the operated device including the processing device of the present invention. It is a block diagram of a processing device of the present invention. FIG. 11 is a diagram showing a flow of a conventional process. It is a figure showing the flow of processing using the prediction circuit of the processing unit of the present invention. FIG. 1 is a block diagram of an imaging device suitable for use by a prediction device of a processing device according to the present invention. 1 is a detailed block diagram of an imaging device suitable for use by a prediction device of a processing device according to the present invention. FIG. 3 is a diagram for explaining the details of image recognition performed by an imaging device that outputs peripheral information used by a prediction circuit of the processing device of the present invention.

  Hereinafter, a processing apparatus according to an embodiment of the present invention will be described.

  FIG. 1 is a block diagram showing a remote operation system including the processing device of the present invention. The operating device 1 and the operated device 2 including the processing device of the present invention are connected via a communication network 3. An operator (not shown) of the operating device 1 inputs an instruction to the operated device 2 to the operating device 1. The operation device 1 generates an operation signal based on the input instruction content and sends the operation signal to the operated device 2 through the communication network 3. The operated device 2 processes the transmitted operation signal and executes the instruction. The operated device 2 includes a unit for acquiring peripheral information of the operated device 2, and a part of the peripheral information acquired by the unit is transmitted to the operating device 1 through the communication network 3.

  The connection between the operating device 1, the operated device 2, and the communication network 3 may be wired or wireless. The communication network 3 may be a public communication network used by an unspecified number of people, or may be a dedicated communication line. Further, the communication network 3 may be a mobile phone communication network or an Internet communication network using an Internet protocol.

  FIG. 2 shows a block diagram of the operating device 1.

  The operation device 1 includes a user interface (UI) 11 for an operator (not shown) to input an instruction to the operated device 2, and an operation signal generation circuit 12 for generating an operation signal based on the content input to the user interface 11. A communication circuit 13 for communicating with the operated device 2 through the communication network 3, a display device 14 for displaying a video signal included in the peripheral information transmitted from the operated device 2, and a peripheral device transmitted from the operated device 2. It includes a speaker 15 for converting an audio signal into audio.

  The user interface 11 is an input device for the operator to input instruction contents, and includes, for example, a joystick, a handle, a lever, a button, a numerical value input device, or a combination thereof. These input devices can be intuitively operated by an operator. The input signal input by the operator is often an analog signal such as a voltage value. The user interface 11 digitizes an analog input signal using an AD converter (not shown) or the like, and supplies the digitized input signal to the operation signal generation circuit 12.

  The operation signal generation circuit 12 generates an operation signal according to a predetermined protocol based on the input signal sent from the user interface 11. For example, a predetermined header is added to an input signal, or signals from a plurality of input devices are arranged in a predetermined order. That is, the operation signal generation circuit 12 arranges the instruction content input by the operator into a format that the operated device 2 can understand. The operation signal generation circuit 12 supplies the generated operation signal to the communication circuit 13.

  The communication circuit 13 sends an operation signal to the operated device 2 through the communication network 3 according to a predetermined communication protocol. Further, the communication circuit 13 receives a peripheral information signal of the operated device 2 from the operated device 2. The peripheral information signal includes a video signal and an audio signal, and the communication circuit 13 supplies the received peripheral information signal to a device that processes the signal for each type of signal.

  The display device 14 displays a video signal included in the peripheral information signal. The speaker 15 converts the audio signal included in the peripheral information signal into a sound.

  An operator (not shown) obtains peripheral information of the operated device 2 from the video of the display device 14 and the sound of the speaker 15, determines an instruction to the operated device 2 based on the information, and uses the user interface 11. Enter the instructions.

  The peripheral information of the operated device 2 is not limited to the video signal and the audio signal. For example, when the operated device 2 has a radar, the information may be sent to the operating device 1 and displayed on the display device 14 of the operating device 1. When the operated device 2 has a thermometer, the temperature may be displayed on the display device 14 or may be transmitted from the speaker 15 to the operator by voice.

  FIG. 3 shows a block diagram of the operated device 2.

  The operated device 2 communicates with the operating device 1 via the communication network 3, a processing device 22 that performs information processing in the operated device 2, and peripheral information that supplies a peripheral information signal to the processing device 22. An acquisition circuit 23, a memory 24 in which the processing device 22 stores information, a control circuit 25 for controlling the drive unit via a bus 250, and a memory 26 in which the control circuit 25 stores information.

  The operated device 2 acquires information from sensors such as the imaging device 300, the microphone 232, the thermometer 233, the laser radar 234, and the millimeter-wave radar 235 via the bus 230, for example. The information acquired by these sensors is supplied to the peripheral information acquisition circuit 23. The peripheral information processing circuit 23 sorts out the supplied signals and supplies them to the processing device 22 as peripheral information signals.

  The control circuit 25 controls the plurality of driving units 251 to 253 via the bus 250, for example. The number of drive units varies depending on the function of the operated device 2. The driving units 251 to 253 are, for example, electric motors, actuators, and transmissions. The control circuit 25 stores information related to control of the driving units 251 to 253 in the memory 26. The information includes information regarding the current state of the driving units 251 to 253, and includes, for example, information indicating whether each of the driving units 251 to 253 is in the ON state or the sleep state.

  FIG. 4 shows a block diagram of the processing device 22. The processing device 22 includes, for example, a processing circuit 221, a control signal generation circuit 222, and a prediction circuit 223. The terminal 224 is connected to the communication circuit 21, the terminal 225 is connected to the peripheral information acquisition circuit 23, the terminal 226 is connected to the memory 24, and the terminal 227 is connected to the control circuit 25.

  The processing circuit 221 allocates a signal to be processed among the communication circuit 21, the peripheral information acquisition circuit 23, the memory 24, the control signal generation circuit 222, the prediction circuit 223, and the control circuit 25 according to the input signal. It also performs calculations based on the signals.

  The control signal generation circuit 222 generates a control signal based on the operation signal extracted by the processing circuit 222 from the communication circuit 21 via the terminal 224. The operation signal describes, for example, an instruction that the operated device 2 moves in which direction and at what speed next. The control signal generation circuit 222 calculates specific movements of the driving units 251 to 253, such as what timing and what driving force is to be output, in order to execute such instruction contents, and To generate a control signal.

  The prediction circuit 223 predicts, based on the peripheral information signal supplied by the peripheral information acquisition circuit 23, what instruction the operator will give next. The number of prediction results is not limited to one, but may be plural. The prediction circuit 223 generates a predicted operation signal having the same format as the operation signal based on the result of the prediction. The prediction operation signal is described in the same format as the operation signal generated by the operation signal generation circuit 12. Therefore, when this prediction operation signal is supplied to the control signal generation circuit 222, the control signal generation circuit 222 generates a control signal in the same manner as in the case of the operation signal.

  The prediction circuit 223 performs prediction using, for example, a neural network. For the neural network, parameters learned in advance by deep learning are used. The data used for learning may be those of an unspecified number of operators, or data limited to a specific operator may be used. When data limited to a specific operator is used, it is possible to learn habits and the like peculiar to the operator.

  Next, the flow of the remote control process will be described. First, a case without the prediction circuit 223 of the present invention will be described with reference to FIG. In FIG. 5, time advances from left to right.

Process 40: The peripheral information acquisition circuit 23 of the operated device 2 acquires the peripheral information from the imaging device 300 and the sensors 232 to 235, and supplies a peripheral information signal to the processing device 22.
Process 41: The processing device 22 supplies a part or all of the peripheral information signal to the communication circuit 21 and sends it to the operating device 1 through the information network 3.
Process 42: When the peripheral information received by the receiving circuit 13 is transmitted to the operator through the display device 14 and the speaker 15, the operator determines the next operation. The operator inputs an instruction using the user interface 11, and the operation signal generation circuit 12 generates an operation signal.
Process 43: The operation signal is transmitted to the operated device 2 through the communication network 3.
Process 44: In the processing device 22, when the processing circuit 221 supplies an operation signal to the control signal generation circuit 222, the control signal generation circuit 222 generates a control signal.
Process 45: The processing device 22 supplies a control signal to the control circuit 25. The control circuit 25 stores the control signal in the memory 26 and executes the control.

  In the processing flow of FIG. 5, if it takes time for the control signal generation circuit 222 to generate the control signal, the execution of the control circuit 25 in the processing 44 is delayed. This delay deteriorates operability in remote operation. Therefore, it is desirable that the generation of the control signal be completed in as short a time as possible.

  Next, the flow of remote control when the prediction circuit 223 of the present invention is provided will be described with reference to FIG. As in the case of FIG. 5, it is assumed that time advances from left to right.

Process 50: The peripheral information acquisition circuit 23 of the operated device 2 acquires the peripheral information from the imaging device 300 and the sensors 232 to 235, and supplies a peripheral information signal to the processing device 22.
Process 51: The processing device 22 sends a part or all of the peripheral information signal to the operating device 1 through the communication circuit 21 and the information network 3.
Process 52: When the peripheral information is transmitted to the operator by the display device 14 and the speaker 15, the operator determines the next operation. The operator inputs an instruction using the user interface 11. The operation signal generation circuit 12 generates an operation signal.
Process 53: An operation signal is transmitted to the operated device 2 through the communication network 3.

  The processes 50 to 53 have the same contents as the processes 40 to 43 described above. While the processes 50 to 53 are performed, the processes 56 to 59 are executed in the operated device 2. These will be described.

Process 56: The processing circuit 221 supplies the peripheral information signal from the peripheral information acquisition circuit 23 to the prediction circuit 223 via the terminal 225. The prediction circuit 223 performs one or more predictions as to what instruction the operator will give next based on the supplied peripheral information signal. After that, a prediction operation signal is generated based on the prediction and supplied to the processing circuit 221. The processing circuit 221 stores the prediction operation signal in the memory 24 via the terminal 226.
Process 57: The processing circuit 221 supplies the prediction operation signal generated by the prediction circuit 223 to the control signal generation circuit 222. The control signal generation circuit 222 generates a first control signal based on the prediction operation signal, and supplies the first control signal to the processing circuit 221. The processing circuit 221 stores it in the memory 24 via the terminal 226.
Process 58: The processing circuit 221 checks the state of the driving units 251 to 253 through the control circuit 25. As described above, the control circuit 25 supplies the information on the states of the driving units 251 to 253 stored in the memory 26 to the processing circuit 221 via the terminal 227. The processing circuit 221 compares the first control signal generated based on the prediction operation signal obtained in the process 57 with information on the states of the driving units 251 to 253.
Process 59: As a result of the comparison in process 58, if there is a drive unit that cannot immediately execute the control signal generated in process 57, the process circuit 221 instructs the control circuit 25 to prepare the drive unit. put out.

  In the process 59, the state in which the control signal cannot be immediately executed includes a case where the power of the driving unit is not turned on and the driving unit is in a sleep state. When the drive unit in the sleep state is moved, it takes longer time to start the operation than in the case where the drive unit is already in the on state, and a delay is generated accordingly. Therefore, the control circuit 25 is instructed to supply power in advance and turn on the driving unit which may be used.

  The processes 56 to 59 have been completed by the time from the process 50 to the process 53 when the operation signal is transmitted from the operation device 1. Next, the flow of the processing after the processing 53 will be described.

Process 54: The processing circuit 221 compares, for example, the operation signal sent from the operation device 1 via the terminal 224 through the communication network 3 and the communication circuit 21 with the predicted operation signal stored in the memory 24, and Select the first prediction signal. When the error between the operation signal and the first prediction operation signal is, for example, within a first error range, the processing circuit 221 supplies a first control signal to the control circuit 25. When the error range is larger than the first error range but is within the second error range, the processing circuit 221 performs a correction calculation and generates a second control signal obtained by correcting the first control signal. And supplies the second control signal to the control circuit 25. When the error range is larger than the second error range, for example, the processing circuit 221 supplies the operation signal to the control signal generation circuit 222, and the control signal generation circuit 222 generates the operation signal based on the operation signal. 3 is supplied to the control circuit 25. Note that the first and second error ranges are set in advance.
Process 55: The control circuit 25 controls the driving units 251 to 253 according to any of the obtained first to third control signals.

  As described above, the processing device 22 of the present invention predicts the instruction of the operator, prepares the first control signal in advance according to the prediction, and supplies the first control signal to the control circuit 25. When the prediction is correct, the time until execution can be shortened, and the operability of remote operation can be improved.

  Further, the processing device 22 of the present invention compares the operation signal with the predicted operation signal, and when the operation signal is approximated as described above, performs a correction calculation and corrects the first control signal to obtain the second control signal. Since the signal is generated and the second control signal is supplied to the control circuit 25, the time from generation of the control signal to execution of the control signal can be shortened as compared with the case where the control signal is generated from the operation signal from the beginning. Operability can be improved.

  Further, the processing device 22 compares the first control signal generated based on the prediction with the states of the driving units 251 to 253, and prepares the driving units 251 to 253 to immediately execute the first control signal. Since the control circuit 25 is instructed to perform the operation, the time until the execution can be shortened, and the operability can be improved. Here, considering the second control signal, the second control signal is obtained by correcting the first control signal, and the execution contents are similar. Therefore, it is considered that the instruction of the preparation to the control circuit 25 performed by the processing device 22 based on the first control signal has a large effect even when the second control signal is executed.

  By the way, in order for the prediction circuit 223 to perform a prediction process correctly, accurate and appropriate peripheral information is necessary, and a method of recognizing a peripheral situation from a video is effective. It is desirable that the peripheral information recognized from the video be supplied to the prediction circuit 223 via the peripheral information acquisition circuit 23 as a recognition result. Therefore, an imaging apparatus 300 that performs image recognition as well as simply acquiring a video will be described.

  FIG. 7 shows a configuration of an imaging device 300 suitable for use by the prediction device of the processing device according to the present invention.

  The imaging device 300 includes a lens 301, an imaging element 302, a video signal processing unit 330, and an image recognition unit 340. The image sensor 302 is a CCD or a CMOS sensor, and includes an ADC (Analog to Digital Converter) (not shown). Red (R), green (G), and blue (B) pixels are arranged in a predetermined array in one frame. The digital RAW data is output in a plurality of frames. The RAW data output from the image sensor 302 is supplied to the video signal processing unit 330 and the image recognition unit 340, respectively.

  The video signal processing unit 330 is provided for the purpose of generating a video adjusted so as not to be unnatural when viewed by human eyes. The video signal processing unit 330 outputs a YUV signal to the bus 230 as a video signal for display.

  The image recognition unit 340 is provided for the purpose of obtaining information described later from the RAW data. Specifically, the image recognition unit 340 extracts a predetermined feature in the image, estimates a portion that is not displayed as an image, and estimates a state after several frames. That is, the image recognition unit 340 extracts a feature in an image in the imaging device 300 in parallel with the video signal processing by the video signal processing unit 330, recognizes information about the feature, and outputs the recognized result to the bus 230. .

  The imaging device 300 will be described in more detail with reference to FIG.

  The video signal processing unit 330 includes, for example, a white defect correction circuit 331, a shading correction circuit 332, a demosaic circuit 333, a correction circuit 334, and a color space processing circuit 335. The video signal processing unit 330 processes the RAW data supplied from the image sensor 302 in a predetermined order to generate an image signal.

  For example, when the signal level of the target pixel is much higher than the peripheral pixel level, the white defect correction circuit 331 determines that the target pixel is a white defect and interpolates the signal level of the pixel from the peripheral pixel signals. I do.

  For example, the shading correction circuit 332 determines the signal level when the signal level in the peripheral part of the image is different from the signal level near the center even when an image of an object having flat luminance is mainly photographed due to optical characteristics such as a lens. The signal level is corrected so that the entire image becomes flat.

  The demosaic circuit 333 generates, for example, an image signal in which each color signal is assigned to all pixels in one frame when the color filters are arranged according to a predetermined rule. The color filter array is, for example, generally a Bayer array using R, G, and B color filters. Further, there is a case where an array including pixels having no color filter and having sensitivity from visible light to near infrared is provided. Such a pixel may be called white as described later.

  The correction circuit 334 corrects various characteristics of the image. This correction includes, for example, white balance, contour correction, and gamma correction. The white balance is to adjust the R, G, and B signals so that they have the same magnitude in a white subject. The contour correction is to emphasize the signal of the contour part in order to improve the resolution. The gamma correction refers to, for example, performing correction in advance according to monitor characteristics so that the luminance level when displayed on a monitor such as the video display unit 200 becomes linear.

  The color space processing circuit 335 converts, for example, signals of three colors of R, G, and B into luminance (Y) and color difference signals. Thereafter, the color difference signal may be corrected to adjust the hue.

  Further, the color space processing circuit 335 may perform a noise reduction process. For example, processing such as adding noise of a plurality of frames to reduce noise is performed. When performing processing using such a plurality of frames, the color space processing circuit 335 waits until all of the plurality of frames to be used are input to the color space processing circuit 335 before performing the processing.

  Further, the color space processing circuit 335 compresses the color difference signal to generate a U signal and a V signal, and generates a YUV signal. The YUV signal obtained through such processing is output from the terminal 350 as a video signal.

  The RAW data of the image sensor 302 is also supplied to the image recognition unit 340. The image recognition unit 340 includes, for example, a luminance image generation unit 341, a static filter unit 342, luminance information memories 343 and 344, a dynamic filter unit 345, a luminance image recognition unit 346, a color image recognition unit 347, and a recognition result transmission unit 348. including.

  Among the components of the image recognition unit 340, the brightness image generation unit 341 to the brightness image recognition unit 346 process the brightness signal. Most of the processing in the image recognition unit 340 is performed based on luminance information. The color image recognition unit 347 determines a color image based on a video signal from the color space processing circuit 335. The recognition result transmission unit 348 generates a recognition result signal based on the recognition results of the luminance image recognition unit 346 and the color image recognition unit 347. The recognition result signal is output from the terminal 351.

  The luminance image generation unit 341 will be described. The luminance image generation unit 341 extracts luminance information from the RAW data. As a simplest method of generating a luminance image, for example, in the case of a Bayer array, there is a method of using only a signal of a pixel with a G color filter. Alternatively, there is a method of using a white pixel signal because a pixel without a color filter called white handles pure luminance only.

  Here, a case will be described in which the image sensor 302 has a Bayer array and uses G pixel signals. Normally, since the luminance signal in the demosaic processing is generated using not only G but also R and B pixel signals, the resolution is about 0.7 to 0.8 times the original resolution. However, the pixel signal of only G cannot use the pixel signals of R and B, and the resolution is reduced to about 0.5 times.

  In the case of an image that can be viewed by a human, such a deterioration in resolution has a serious problem because it greatly affects the image quality. However, when the purpose is to perform image recognition, resolution degradation is not always an important problem, so that the influence is small. The reason will be described later.

  As described above, if only the G pixel signals in the Bayer array are used, the R and B pixels have no signal. Therefore, the luminance image generation unit 341 simply converts the four G pixel signals around the missing pixel portion. Interpolate pixels on average.

  If there is a white flaw as described above, the white flaw correction circuit 331 corrects this. Normally, emphasis is placed on appearance, so that white defect correction is performed using not only G but also R and B pixel signals, but here, the luminance image generation unit 341 performs correction using only adjacent G pixel signals. As described above, the luminance image generating unit 341 performs processing at high speed by using information of a luminance signal which is smaller than usual. That is, a luminance image is generated by a simple process, placing importance on the speed of the signal processing over the resolution.

  The luminance image generated by the luminance image generation unit 341 is supplied to the static filter unit 342 and the luminance information memory 343.

  The static filter unit 342 will be described. The static filter unit 342 performs a filtering process for extracting a feature from a one-frame luminance image without motion. The filtering process performed here is mainly a convolution process and a pooling process. The convolution processing is processing for smoothing or sharpening an image. The pooling process is a process of resampling the result of such convolution.

  Such processing is equivalent to deleting information while leaving only the features to be extracted. Since the processing for removing information is included as described above, even if the amount of information included in the luminance image is small to some extent, the feature extraction by the static filter unit 342 is not significantly affected. Therefore, the luminance image generation unit 341 can use a simple method that involves degradation of resolution in generating a luminance image.

  The static filter unit 342 has different filters for each feature to be extracted, and operates them in parallel. These different filters are pre-trained, for example, by deep learning of a neural network, and the parameters are tuned.

  The result extracted by the static filter unit 342 is supplied to the luminance information memory 343 and the luminance image recognition unit 346, respectively.

  Note that the imaging device 300 does not perform re-tuning (re-learning) of the filter parameters. This is because, if retuning is performed, when there are a plurality of imaging devices 300, the characteristics of each imaging device 300 change with time and deviate from a steady state. Filter parameters change only when provided externally in the form of a version upgrade.

The static filter unit 342 extracts, for example, the following features.
Shape extraction: Extract a specific shape. The shape includes a vertical line, a horizontal line, an oblique line, an angle, a peak with an acute angle, a square, a circle, a triangle, and the like.
・ Important object extraction: Extract important objects such as human body, car, animal, and sign. According to the present embodiment, it is possible to extract a feature only when a part is visible even when the whole is not visible.
Character / symbol extraction: Extracts whether there is a character or a symbol.
-Static contour extraction: Extract a place where luminance changes rapidly in a static image.
-Vanishing point extraction: Used to compare perspective.

  The latest luminance image from the luminance image generation unit 341 and the extraction result of the static filter unit 342 are written in the luminance information memory 343. The written latest luminance image and the extraction result are moved to the luminance information memory 344 before the luminance image and the extraction result of the next frame are written to the luminance information memory 343. Thus, the luminance information memory 344 stores information of at least one frame before.

  Note that the feature extraction from the luminance image by the static filter unit 342 need not be performed after the supply of the luminance image from the luminance image generation unit 341 is completed. This can be performed on a part of the luminance image.

The dynamic filter unit 345 extracts a feature related to motion by comparing information from the luminance information memory 343 with information from at least one frame before from the luminance information memory 344. The feature extraction related to such movement is, for example, as follows.
-Lump extraction: A group of pixels that move in the same way is extracted.
Active contour extraction: When one lump is extracted by lump extraction, a lump boundary is extracted as a contour. The dynamic contour extraction is effective when the background and the block have the same luminance, or when it is difficult to extract the contour from the static image.
-Movement direction extraction: The movement direction of the lump is extracted.
Movement feature extraction: Extract a particular movement. For example, the dynamic filter unit 345 extracts whether or not there is a specific characteristic motion such as a rotational motion, a walking motion of a person, a bouncing motion of a ball, and a flying motion of a bird.

  The dynamic filter unit 345 uses, for example, a filter whose parameters have been tuned and learned in advance by deep learning of a neural network. Further, the dynamic filter unit 345 has different filters for each feature to be extracted, and operates them in parallel.

  The features extracted by the static filter unit 342 or the dynamic filter unit 345 are supplied to the luminance image recognition unit 346. The luminance image recognition unit 346 collects the extracted features and recognizes the content of the luminance image. For example, whether the object extracted as a person by the static filter unit 342 is moving as a lump of objects in the result extracted by the dynamic filter unit 345, whether the feature of the movement is a person-specific movement, and the like. After confirming whether the plurality of features match, the luminance image recognition unit 346 recognizes the object as a person. As described above, by confirming from a plurality of features, the luminance image recognizing unit 346 increases the certainty of the recognition and lowers the probability that the doll is erroneously recognized as a human, for example.

The luminance image recognition unit 346 performs various estimation operations. The luminance image recognition unit 346 performs, for example, the following estimation.
-Estimation of context: Estimate the context of objects in a certain lump.
-Estimation of own movement: Estimate the relative relationship with the surroundings, such as whether or not you are moving.
-Estimation of the movement of the mass in the front-back direction (the depth direction).
Estimation of a hidden part: For example, when a linear object is hidden, it is estimated that the visible part is extended. Even if the recognized target is temporarily hidden and cannot be seen, it is assumed that the target is there.
Estimation (prediction) of future state: Estimate what kind of positional relationship will be after several frames from the estimation result of motion. Further, the luminance image recognition unit 346 checks whether the positional relationship after several frames is as estimated.

  By performing the above estimation, the luminance image recognition unit 346 can perform, for example, the following recognition. When the object A moves and disappears behind the object B, it is recognized that the object A is not visible but is behind the object B. Further, it is estimated that the object A comes out from the opposite side of the object B after a predetermined time, and it is confirmed whether or not it actually comes out.

  Next, recognition of color information will be described. Recognition of color information is, for example, recognition of a color emitted by a traffic light. Even if the static filter unit 342 extracts the traffic light, it is not known from the luminance image how many signals are illuminated, so that color information may need to be recognized. However, since the necessity of processing color information at high speed as in the case of processing a normal video signal is lower than that of luminance information, the present embodiment is configured to wait for the result of the color space processing circuit 335 and process the color information. .

  Of course, if there is a problem if there is a frame delay in the color information, the video signal processing and the parallel processing may be performed as in the case of the luminance signal. In that case, it is preferable that the feature is directly extracted from the RAW data of each color (normally, three colors of R, G, and B) without performing demosaicing.

  The recognition results of the luminance image recognizing unit 346 and the color image recognizing unit 347 are sent to the recognition result transmitting unit 348, and are transmitted from the terminal 351 as a recognition result signal according to a predetermined protocol. Since a huge amount of recognition results are sent from the luminance image recognition unit 346 and the color image recognition unit 347, the recognition result transmission unit 348 selects the recognition results and sets the transmission order.

  The method or reference for selecting the recognition result in the recognition result transmitting unit 348 may be adjusted to the characteristics of the device to be controlled.

  Note that the terminals 350 and 351 may be serial bus terminals or parallel bus terminals.

  Next, an example of how the luminance image recognition unit 346 recognizes an image will be described. FIG. 9 is an image 60 of one frame including a video signal output from the terminal 350 by the imaging device 300. The recognition result of this image 60 by the luminance image recognition unit 346 is, for example, as follows.

  The static filter unit 342 extracts a human feature from the object 62 and the object 66 and reports it to the luminance image recognition unit 346. On the other hand, the dynamic filter unit 345 reports to the luminance image recognizing unit 346 that the objects 62 and 66 are moving as a lump, and reports that the manner of movement is human walking. Since all reports are consistent, the luminance image recognition unit 346 recognizes that the objects 62 and 66 are human.

  The static filter unit 342 extracts the vanishing point 65, and from this vanishing point 65, the luminance image recognition unit 346 recognizes the positional relationship between the objects 62 and 66 in terms of distance. Also, the luminance image recognition unit 346 estimates the approximate current and future positional relationship of the objects 62 and 66 from the moving speeds and moving directions of the objects 62 and 66 reported by the dynamic filter unit 345.

  Furthermore, the luminance image recognition unit 346 recognizes that the object 66 is behind the object 68 from the estimation of the context of the object 66 and the object 68. Further, the luminance image recognition unit 346 recognizes that the object 66 has come out of the object 68 from the recognition of the moving direction. Further, since the luminance image recognition unit 346 recognizes that the object 66 is a human, it also recognizes that there is a part 67 of the human body that is not visible behind the object 68, and after a certain time, the human body 1 Predict that part 67 will be visible.

  Although the line 64 indicating the end of the road is partially hidden behind the object 62, the luminance image recognition unit 346 assumes that the line continues even where it cannot be seen. It is estimated that the broken line 63 actually appears as the object 62 moves.

  Further, since the static filter unit 342 extracts features of the shape of the object 61 from the object 61 and also extracts features as markers, the luminance image recognition unit 346 transmits such a recognition result to the recognition result transmitting unit 348. Send to In the case of a sign, since color information is often important, the recognition result transmission unit 348 transmits the color of each part of the object 61 recognized by the color image recognition unit 347 together with the result of the luminance image recognition unit 346 to the bus. The information is reported to the peripheral information acquisition circuit 23 via 230.

  As described above, the imaging apparatus 300 performs image recognition processing in parallel with video signal processing, and outputs a video signal and a recognition result signal substantially simultaneously.

  The peripheral information acquisition circuit 23 generates a peripheral information signal including the video signal and the recognition result signal, and supplies the generated peripheral information signal to the processing circuit 22. Therefore, the prediction circuit 223 receives the recognition result signal at the same time as the video signal from the imaging device 300 supplied via the processing circuit 221 and uses the signal for prediction. Since the prediction circuit 223 does not need to perform the video signal recognition process, the time until the prediction is completed can be shorter than when only the video signal is received. Alternatively, more accurate prediction can be made even with the same processing time. Therefore, the operability of remote operation can be further improved by using the imaging device 300.

  Note that the processing device 22 of the present invention may be configured by a hardware circuit, or may be executed by a computer using a CPU or the like by software. Similar processing may be performed by appropriately combining hardware and software.

REFERENCE SIGNS LIST 1 operation device 2 operated device 3 communication network 11 user interface 12 operation signal generation circuit 13, 21 communication circuit 14 display device 22 processing device 221 processing circuit 222 control signal generation circuit 223 prediction circuit 23 peripheral information acquisition circuit 24, 26 memory 25 Control circuit 251, 252, 253 Drive unit 300 Imaging device

Claims (6)

A processing device for the operated device that generates a control signal according to an operation signal indicating an instruction of a remote operator and outputs the control signal,
A prediction circuit that predicts an instruction issued next by the operator based on the peripheral information of the operated device, and generates a predicted operation signal based on the prediction;
A control signal generation circuit that generates the control signal based on the operation signal or the prediction operation signal,
When an error of the predicted operation signal with respect to the operation signal is within a first range, the control signal generation circuit outputs a first control signal generated based on the predicted operation signal before receiving the operation signal. A processing device characterized by the above-mentioned.   When an error of the predicted operation signal with respect to the operation signal is larger than a first range and is within a second range, a second control signal is generated by correcting the first control signal according to the error and output. The processing device according to claim 1, which performs the processing.   The processing apparatus according to claim 1, wherein the prediction circuit outputs a plurality of prediction results, and the control signal generation circuit generates a plurality of control signals corresponding to each of the plurality of prediction results.   The processing apparatus according to claim 1, further comprising an imaging unit configured to acquire video information included in the peripheral information.   The processing apparatus according to claim 4, wherein the imaging unit outputs a recognition result from the video information as a part of the peripheral information. The operated device has a drive unit that performs control based on the control signal,
The processing apparatus according to claim 1, wherein the processing unit instructs the driving unit specified by the first control signal to prepare for driving in advance.