WO2022123654A1 - Dispositif de traitement d'informations et procédé de traitement d'informations - Google Patents

Dispositif de traitement d'informations et procédé de traitement d'informations Download PDF

Info

Publication number
WO2022123654A1
WO2022123654A1 PCT/JP2020/045684 JP2020045684W WO2022123654A1 WO 2022123654 A1 WO2022123654 A1 WO 2022123654A1 JP 2020045684 W JP2020045684 W JP 2020045684W WO 2022123654 A1 WO2022123654 A1 WO 2022123654A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
warning
condition
unit
information processing
Prior art date
Application number
PCT/JP2020/045684
Other languages
English (en)
Japanese (ja)
Inventor
政人 土屋
良枝 今井
Original Assignee
三菱電機株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 三菱電機株式会社 filed Critical 三菱電機株式会社
Priority to PCT/JP2020/045684 priority Critical patent/WO2022123654A1/fr
Priority to JP2021524283A priority patent/JPWO2022123654A1/ja
Publication of WO2022123654A1 publication Critical patent/WO2022123654A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis

Definitions

  • This disclosure relates to an information processing device and an information processing method.
  • Patent Document 1 discloses a technique for visualizing a judgment basis in a case where a disease in the body is judged from an endoscopic image.
  • the warning system using the object detection technology it is necessary to select the information when notifying the user of the recognition result. For example, if the judgment is unreliable at the time of detection, or if the detection is not a threat at that time, telling the user may rather hinder the user's operation or concentration.
  • visualization techniques so far have been applied to single tasks such as image classification
  • conventional visualization techniques are used as they are for multitasking tasks such as object detection that combines image classification and position detection. Cannot be applied.
  • one or more aspects of the present disclosure are intended to enable visualization of the result of the judgment basis for object detection.
  • the information processing apparatus detects the object according to the feature map in the object detection unit that detects the object from the target image using the feature map and the arrangement of the same pixels as the target image. It is characterized by including a judgment basis visualization unit that generates a judgment basis visualization image, which is an image that enables identification of the importance of an image area that is a determination basis.
  • an object is detected from a target image using a feature map, and a judgment basis is used when the object is detected according to the feature map in the same pixel arrangement as the target image. It is characterized in that a judgment basis visualization image, which is an image capable of identifying the importance of the image area that has become, is generated.
  • FIG. 1 is a block diagram schematically showing a configuration of a warning system 100 as an information processing system (for example, a visualization system) according to an embodiment.
  • the warning system 100 includes a camera 101, a distance measuring sensor 102, a warning device 110 as an information processing device, a monitor 103, and a speaker 104.
  • the warning system 100 uses a warning device 110 mounted as an in-vehicle device to give a warning notification to a user such as a driver or other occupants together with visualization of the judgment basis.
  • this warning system 100 is not limited to being mounted on an automobile, and can be widely applied to mobile vehicles such as railroad vehicles and two-wheeled vehicles. Therefore, the description of the first embodiment does not narrow the scope of the right.
  • the warning device 110 is provided with a camera 101 as an image pickup device that acquires an image in the traveling direction, a distance measuring sensor 102 as a distance measuring device that measures the distance between an object in the traveling direction, and a warning content.
  • a monitor 103 as a display device that visually conveys the user to the user, and a speaker 104 as an acoustic device that conveys the warning content to the user by sound are connected.
  • the monitor 103 and the speaker 104 function as a warning output device that outputs a warning.
  • the camera 101 is installed facing the front of the vehicle and converts visible light into a digital signal, and can acquire an image of the front of the vehicle as an RGB value.
  • the camera 101 outputs an image output signal indicating the captured image to the warning device 110.
  • the camera 101 does not have to be a visible light camera, and may be an infrared camera, a depth camera, or the like.
  • the distance measuring sensor 102 measures the distance to the object in front of the camera 101 and transmits it to the warning device 110.
  • the distance measuring sensor 102 outputs a depth image output signal indicating a depth image representing the measured distance for each pixel to the warning device 110.
  • a conventional radar or a sensor such as LiDAR is assumed as the distance measuring sensor 102.
  • the monitor 103 is a device used to visually convey a warning from the warning device 110 to the user by an image.
  • an image obtained by superimposing a warning display on the image acquired from the camera 101 is used.
  • the speaker 104 is a device used to convey a warning from the warning device 110 to the user by voice.
  • FIG. 2 is a block diagram schematically showing the configuration of the warning device 110.
  • the warning device 110 includes an input unit 111, an image signal acquisition unit 112, a preprocessing unit 113, an object detection unit 114, a judgment basis visualization unit 115, a condition judgment unit 116, a warning execution unit 117, and an output unit. It is equipped with 118.
  • the input unit 111 receives the input of the image output signal from the camera 101 and the depth image output signal from the distance measuring sensor 102.
  • the input unit 111 gives the image output signal and the depth image output signal to the image signal acquisition unit 112.
  • the image signal acquisition unit 112 converts an image output signal, which is an analog signal from the camera 101, into a digitized image signal. Further, the image signal acquisition unit 112 converts the depth image output signal, which is an analog signal from the distance measuring sensor 102, into a digitized depth image signal. Then, the image signal acquisition unit 112 supplies the image signal and the depth image signal to the preprocessing unit 113.
  • the preprocessing unit 113 converts the image signal into a processed image signal in order to make the format convenient for the object detection unit 114 and the judgment basis visualization unit 115 for the convenience of calculation efficiency. Further, the preprocessing unit 113 also converts the depth image signal into the processed depth image signal.
  • the preprocessing includes various processes such as standardization and whitening, but here, the preprocessing is simply a resizing process for converting the resolution of a two-dimensional image signal into a square. Then, the preprocessing unit 113 gives the processed image signal and the processed depth image signal to the object detection unit 114, the judgment basis visualization unit 115, and the warning execution unit 117.
  • the image signal acquisition unit 112 and the preprocessing unit 113 configure an image acquisition unit that acquires an object image for detecting an object.
  • the target image is an image represented by the processed image signal.
  • the object detection unit 114 detects an object from the target image using the feature map. For example, the object detection unit 114 detects the object position based on the processed image and the processed depth image, which are the images indicated by the processed image signal and the processed depth image signal received from the preprocessing unit 113, and the object detection unit 114 thereof. Generates object detection result information, which is information indicating the object detection result that is the detection result. The object detection result information is given to the judgment basis visualization unit 115, the condition judgment unit 116, and the warning execution unit 117.
  • FIG. 3 is a schematic diagram showing a first example of object detection result information.
  • the object detection result information 120 shown in FIG. 3 is table-type information including a detection candidate ID row 120a, a detection position row 120b, an estimated distance row 120c, and a reliability row 120d.
  • the detection candidate ID column 120a stores the detection candidate ID which is the object candidate identification information as the identification information for identifying the detected object.
  • the detection position sequence 120b stores the detection position, which is the position where the object candidate is detected.
  • the detection position indicates the position of the detected object in the image space.
  • the position when the position information of the object detection model used by the object detection unit 114 is formulated by the Dirac Delta distribution is shown.
  • the estimated distance sequence 120c stores the distance of the detected object.
  • the reliability column 120d stores the reliability indicating the probability that the detected object is the target object.
  • FIG. 4 is a schematic diagram showing a second example of the object detection result information. Similar to the object detection result information 120 shown in FIG. 3, the object detection result information 121 shown in FIG. 4 has a detection candidate ID row 121a, a detection position row 121b, an estimated distance row 121c, and reliability. It is information in the form of a table including the sex column 121d. In the example shown in FIG. 4, the detection position row 121b shows the position when the position information of the object detection model used by the object detection unit 114 is formulated with a normal distribution.
  • Feature map is a general term that refers to a multidimensional tensor obtained as a result of performing a spatial convolution operation on an image signal.
  • the object detection unit 114 generates feature map information indicating this feature map, and gives the feature map information to the judgment basis visualization unit 115.
  • the judgment basis visualization unit 115 is an image in which the importance of the image region that is the judgment basis when detecting an object can be identified according to the feature map in the same pixel arrangement as the target image. Generate a certain judgment basis visualization image. For example, the judgment basis visualization unit 115 is based on the processed image signal and the processed depth image signal received from the preprocessing unit 113, the object detection result information and the feature map information received from the object detection unit 114, and the judgment basis visualization image. To generate.
  • the judgment basis visualization image is a shaded image having the same resolution as the processed image signal after resizing and the processed depth image signal, and is an image area that is an important judgment basis for the object detection unit 114 to perform object detection. It is an image in which a high value is set.
  • the judgment basis visualization image is converted from a low value to a color map to which colors such as blue, green, yellow, and red are assigned, and is often displayed superimposed on the original image by using alpha blending or the like.
  • the judgment basis visualization image is created based on the information of the detection position included in the object detection result, and the distribution of the detection position is used as it is for visualization.
  • the detection position of the object detection model used by the object detection unit 114 is formulated by the Dirac Delta distribution, as an example of simple visualization, there is a calculation method as shown in the following equation (1).
  • k is the detection candidate ID
  • x is the position of the object identified by k in the image space
  • conf is the reliability value of the object identified by k
  • Z is the standardized constant. It is a constant for setting the maximum value of E at each position x in all object classes at the time of object detection to 1.
  • c is a channel in the feature map.
  • the circle symbol in the equation (2) indicates the Hadamard product, and the outer double line indicates the L2 norm.
  • the judgment basis visualization unit 115 gives the judgment basis visualization signal indicating the judgment basis visualization image to the warning execution unit 117.
  • the condition determination unit 116 determines the warning setting level based on the object detection result information.
  • the reliability and estimated distance included in the object detection result are used to determine the warning setting level.
  • FIG. 5 is a table for determining the warning setting level L. As shown in FIG. 5, the warning setting level L is determined according to the combination with the reliability and the estimated distance. Since the warning setting level L is used to determine the operation of the warning execution unit 117, the warning setting level information indicating the warning setting level L is given to the warning execution unit 117.
  • the warning setting level 1 is a level when the first condition that no object is detected from the target image is satisfied.
  • the warning setting level 1 is a level when the reliability is less than 50% and the second condition that the distance is 10 m or more is satisfied.
  • the warning setting level 2 is a level when the third condition that the reliability is less than 50% and the distance is less than 10 m, or the reliability is 50% or more and the distance is 10 m or more is satisfied.
  • the warning setting level 3 is a level when the reliability is 50% or more and the fourth condition that the distance is less than 10 m is satisfied.
  • the second condition is a condition in which a warning for the detected object is more necessary than the first condition
  • the third condition is a warning for the detected object rather than the second condition.
  • the fourth condition is a condition in which the need for warning to the detected object is higher than the third condition.
  • the warning execution unit 117 is an output execution unit that determines the content of the warning to the user based on the warning setting level indicated by the warning setting level information received from the condition determination unit 116 and executes the output.
  • FIG. 6 is a table for determining the warning content from the warning setting level L. As shown in FIG. 6, the warning execution unit 117 determines the warning content according to the warning setting level L. Then, the warning execution unit 117 generates warning information according to the determined warning content, and gives a warning signal indicating the warning information to the output unit 118.
  • the warning execution unit 117 does not generate warning information. In this case, the warning execution unit 117 outputs a processed image signal indicating the target image to the output unit 118.
  • the warning execution unit 117 If the warning setting level L is 1, the warning execution unit 117 generates an output image using the judgment basis visualization image as a warning image, and generates warning information indicating the warning image. Specifically, the warning execution unit 117 generates an output image by alpha blending the judgment basis visualization image received from the judgment basis visualization unit 115 into the processed image indicated by the processed image signal.
  • the warning execution unit 117 If the warning setting level L is 2, the warning execution unit 117 generates a processed image in which the object detected in the target image is surrounded by a rectangle as a warning image, and generates warning information indicating the warning image. Specifically, the warning execution unit 117 sets the detection position included in the object detection result received from the object detection unit 114 as a rectangle and superimposes the warning image on the processed image indicated by the processed image signal. Generate and generate warning information indicating the warning image.
  • the warning execution unit 117 If the warning setting level L is 3, the warning execution unit 117 generates the above-mentioned processed image as a warning image, and also generates a warning sound which is a voice for warning, and the warning image and the warning sound indicating the warning sound are generated. Generate information. Specifically, the warning execution unit 117 superimposes the detection position included in the object detection result received from the object detection unit 114 on the processed image indicated by the processed image signal as a rectangle to display the warning image. At the same time as generating, a warning sound by voice is generated.
  • the output unit 118 outputs a warning information given by the warning execution unit 117 to convey a warning to the user through at least one of the monitor 103 and the speaker 104.
  • FIG. 7 is a hardware configuration diagram of the warning device 110. As shown in FIG. 7, the warning device 110 can be configured by a computer 130.
  • the memory 131 stores a program that functions as an image signal acquisition unit 112, a preprocessing unit 113, a judgment basis visualization unit 115, an object detection unit 114, a condition judgment unit 116, and a warning execution unit 117, and data used by these.
  • the memory 131 may be, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EPROM (Electrically Memory Memory, etc.)
  • An optical disk, a photomagnetic disk, or the like is used.
  • the processor 132 reads out a program that functions as an image signal acquisition unit 112, a preprocessing unit 113, a judgment basis visualization unit 115, an object detection unit 114, a condition judgment unit 116, and a warning execution unit 117, and executes processing.
  • the processor 132 uses, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a microcontroller, a DSP (Digital Signal Processor), or the like.
  • the acoustic interface 133 is used when the warning execution unit 117 outputs a warning by sound from the speaker 104 when instructing the user to give a warning.
  • the acoustic interface functions as an output unit 118.
  • the image interface 134 is for transmitting the analog signal obtained from the camera 101 and the distance measuring sensor 102 to the image signal acquisition unit 112. In this case, the image interface 134 functions as an input unit 111.
  • the image interface 134 is also used when the warning execution unit 117 outputs a final screen output signal to the monitor 103 when instructing the user to give a warning. In this case, the image interface 134 functions as an output unit 118.
  • the network interface 135 is used when the image signal acquisition unit 112 receives an image input signal from the external network environment.
  • the network interface 135 is not required unless it is configured to communicate with the external network environment. In this case, the network interface 135 functions as the input unit 111.
  • the memory 131 is arranged inside the warning device 110 in FIG. 2, it may be configured as an external memory such as a USB memory and may be configured to read a program and data. Further, as the memory 131, the memory in the device and the external memory may be used together.
  • FIG. 8 is a flowchart showing a process performed by the warning device 110.
  • the input unit 111 receives the input of the image output signal from the camera 101 and the depth image output signal from the distance measuring sensor 102 (S10).
  • the input image output signal and depth image output signal are given to the image signal acquisition unit 112, converted from an analog signal to a digital signal by the image signal acquisition unit 112, and used as an image signal and a depth image signal in the preprocessing unit.
  • the preprocessing unit 113 performs preprocessing on the image signal and the depth image signal to convert them into a processed image signal and a processed depth image signal, respectively (S11).
  • the processed image signal and the processed depth image signal are given to the object detection unit 114, the judgment basis visualization unit 115, and the warning execution unit 117.
  • the object detection unit 114 performs object detection from the processed image signal and the processed depth image signal, and determines the object detection result information indicating the object detection result which is the detection result. And the warning execution unit 117, and further, the feature map information indicating the feature map obtained in the process of object detection is given to the judgment basis visualization unit 115 (S12).
  • condition determination unit 116 determines the warning setting level according to the combination of the estimated distance and the reliability included in the object detection result information, and provides the warning setting level information indicating the determined warning setting level. It is given to the warning execution unit 117 (S13).
  • the warning execution unit 117 confirms the warning setting level indicated by the given warning setting level information (S14), ends the processing if the warning setting level is 0, and processes if the warning setting level is weak. To step S15, and if the warning setting level is medium to strong warning, the process proceeds to step S16.
  • step S15 the warning execution unit 117 generates a warning image by alpha blending the judgment basis visualization image received from the judgment basis visualization unit 115 into the processed image indicated by the processed image signal, and shows the warning image. Generate warning information. The generated warning information is given to the output unit 118. Then, the process proceeds to step S17.
  • step S16 if the warning setting level L is 2, the warning execution unit 117 processes the detected position included in the object detection result received from the object detection unit 114 as a rectangular shape and is indicated by the processed image signal. By superimposing on the finished image, a warning image is generated, and warning information indicating the warning image is generated.
  • the warning setting level L 3
  • a warning image is generated by superimposing the detection position included in the object detection result received from the object detection unit 114 on the processed image indicated by the processed image signal as a rectangle.
  • a warning sound by voice is generated, and a warning image and warning information indicating the warning sound are generated. Then, the process proceeds to step S17.
  • step S17 the output unit 118 issues a warning to the user using at least one of voice and image based on the warning information generated by the warning execution unit 117.
  • FIGS. 9 (A) and 9 (B) are schematic views showing an example of a screen image output to the monitor 103. Both the screen image 140 and the screen image 150 assume a situation in which vehicles 141 and 151 cross to the right from the corner of the road ahead.
  • the screen image 140 is a warning screen image generated by the warning execution unit 117 when it is determined that the warning setting level is medium to strong.
  • the detection position included in the object detection result is used, and a warning by the rectangle 142 is displayed at that position.
  • the screen image 150 is a warning screen image generated by the warning execution unit 117 when it is determined that the warning setting level is equivalent to a weak warning.
  • the attention area 152 in which the information indicated by the judgment basis visualization signal is superimposed on the original image is displayed as the warning content.
  • the warning execution unit 117 since the image is resized in the preprocessing unit 113, the warning execution unit 117 generates a warning image using the processed image processed by the preprocessing unit 113.
  • the present embodiment is not limited to such an example.
  • the warning execution unit 117 can generate a warning image using the image represented by the digitized image signal by the image signal acquisition unit 112. ..
  • warning system 101 camera, 102 distance measuring sensor, 103 monitor, 104 speaker, 110 warning device, 111 input unit, 112 image signal acquisition unit, 113 preprocessing unit, 114 object detection unit, 115 judgment basis visualization unit, 116 conditions Judgment unit, 117 warning execution unit, 118 output unit.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Closed-Circuit Television Systems (AREA)

Abstract

L'invention concerne un dispositif d'avertissement (110) équipé d'une unité de détection d'objet (114) pour détecter un objet en utilisant une carte de caractéristiques à partir d'une image cible, et d'une unité de visualisation de base de détermination (115) pour générer une image de visualisation de base de détermination, qui est une image qui permet d'identifier le niveau d'importance dans une région d'image qui est la base pour une détermination lors de la détection d'un objet selon une carte de caractéristiques dans un agencement de pixels identique à celui de l'image cible.
PCT/JP2020/045684 2020-12-08 2020-12-08 Dispositif de traitement d'informations et procédé de traitement d'informations WO2022123654A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2020/045684 WO2022123654A1 (fr) 2020-12-08 2020-12-08 Dispositif de traitement d'informations et procédé de traitement d'informations
JP2021524283A JPWO2022123654A1 (fr) 2020-12-08 2020-12-08

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/045684 WO2022123654A1 (fr) 2020-12-08 2020-12-08 Dispositif de traitement d'informations et procédé de traitement d'informations

Publications (1)

Publication Number Publication Date
WO2022123654A1 true WO2022123654A1 (fr) 2022-06-16

Family

ID=81973386

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/045684 WO2022123654A1 (fr) 2020-12-08 2020-12-08 Dispositif de traitement d'informations et procédé de traitement d'informations

Country Status (2)

Country Link
JP (1) JPWO2022123654A1 (fr)
WO (1) WO2022123654A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016114592A (ja) * 2014-12-12 2016-06-23 キヤノン株式会社 情報処理装置、情報処理方法、プログラム
JP2019156641A (ja) * 2018-03-08 2019-09-19 コニカミノルタ株式会社 フォークリフト用の画像処理装置、および制御プログラム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016114592A (ja) * 2014-12-12 2016-06-23 キヤノン株式会社 情報処理装置、情報処理方法、プログラム
JP2019156641A (ja) * 2018-03-08 2019-09-19 コニカミノルタ株式会社 フォークリフト用の画像処理装置、および制御プログラム

Also Published As

Publication number Publication date
JPWO2022123654A1 (fr) 2022-06-16

Similar Documents

Publication Publication Date Title
CN109017570B (zh) 车辆周围场景呈现方法和装置、车辆
JP4359710B2 (ja) 車両周辺監視装置、車両、車両周辺監視用プログラム、車両周辺監視方法
JP5121389B2 (ja) 対象体の大きさを測定するための超音波診断装置及び方法
JP4171501B2 (ja) 車両の周辺監視装置
EP2150053A1 (fr) Système de surveillance de périphérie de véhicule, programme de surveillance de périphérie de véhicule et procédé de surveillance de périphérie de véhicule
EP3499456B1 (fr) Dispositif de circuit, instrument électronique et procédé de détection d'erreur
JP5718920B2 (ja) 車両周辺監視装置
US9826166B2 (en) Vehicular surrounding-monitoring control apparatus
WO2012004938A1 (fr) Dispositif pour la surveillance de la proximité d'un véhicule
JP2009037622A (ja) 画像を評価するための方法および装置
EP3534331A1 (fr) Dispositif de circuit et appareil électronique
JP6068833B2 (ja) 車用色検出器
JP2008268190A (ja) 表面特性の決定
JP2003028635A (ja) 画像測距装置
JP4813304B2 (ja) 車両周辺監視装置
WO2022123654A1 (fr) Dispositif de traitement d'informations et procédé de traitement d'informations
JP5906696B2 (ja) 車両周辺撮影装置および車両周辺画像の処理方法
WO2010115020A2 (fr) Système de détection de motif et de couleur
JP5538969B2 (ja) 画像目標検出装置および画像目標検出方法
JP7505596B2 (ja) 画像処理装置、画像処理方法、及び画像処理プログラム
CN110556024B (zh) 一种防碰撞辅助驾驶方法、系统及计算机可读存储介质
JP2011081614A (ja) 認識装置、認識方法及びプログラム
JP4869835B2 (ja) 車両周辺監視システム
JP2018072884A (ja) 情報処理装置、情報処理方法およびプログラム
JP2966683B2 (ja) 車両用障害物検出装置

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2021524283

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20965038

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20965038

Country of ref document: EP

Kind code of ref document: A1