WO2023032338A1 - 色判別装置及び色判別方法 - Google Patents

色判別装置及び色判別方法 Download PDF

Info

Publication number
WO2023032338A1
WO2023032338A1 PCT/JP2022/016636 JP2022016636W WO2023032338A1 WO 2023032338 A1 WO2023032338 A1 WO 2023032338A1 JP 2022016636 W JP2022016636 W JP 2022016636W WO 2023032338 A1 WO2023032338 A1 WO 2023032338A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
color
unit
images
reflection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/016636
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
祐治 花田
正彦 南雲
君晴 佐藤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Semiconductor Solutions Corp
Sony Group Corp
Original Assignee
Sony Semiconductor Solutions Corp
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Semiconductor Solutions Corp, Sony Group Corp filed Critical Sony Semiconductor Solutions Corp
Priority to US18/293,748 priority Critical patent/US20240338852A1/en
Priority to JP2023545074A priority patent/JPWO2023032338A1/ja
Publication of WO2023032338A1 publication Critical patent/WO2023032338A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/46—Measurement of colour; Colour measuring devices, e.g. colorimeters
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/21—Polarisation-affecting properties
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/55—Specular reflectivity
    • G01N21/57—Measuring gloss
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the image acquisition unit 34 in FIG. 6 acquires two or more types of images from among the visible image color-converted by the color conversion unit 33, the reflection suppression image, and the reflection component image.
  • the reason why the image acquisition unit 34 acquires two or more types of images is that, for example, the colors of a plurality of target members 5 having different surface treatment specifications may not be accurately determined using only visible images. Since the reflection-suppressed image is an image that does not include a polarized light component, by combining the visible image and the reflection-suppressed image, it is possible to identify the difference in the surface treatment specification of the target member 5 .
  • the color discrimination device 30 discriminates the color of the target member 5 using machine learning.
  • the signal processing unit 74 performs the processing of the RAW image processing unit 31, the image extraction unit 32, the color conversion unit 33, and the image acquisition unit 34 shown in FIG. At least part of the processing of the RAW image processing unit 31 whose details are shown in FIG. 8 may be performed inside the polarization sensor 73 .
  • the color conversion unit 33 may be omitted in the present embodiment. That is, the visible image, the reflection suppression image, and the reflection component image extracted by the image extraction unit 32 may be input to the image acquisition unit 34 without undergoing color conversion.
  • the image acquisition unit 34 acquires two or more types of images from among a visible image without color conversion, a reflection suppression image, and a reflection component image. The acquired two or more types of images are input to the color discrimination section 35 .
  • the application processor 75 also has a function of sending and receiving data to and from the cloud server 72 via the network 77 .
  • the cloud server 72 manages models generated by machine learning. That is, the cloud server 72 has a function of a model building section that builds a model for discriminating the color of the target member 5 based on two or more types of input images.
  • the application processor 75 receives the trained model from the cloud server 72 via the network 77 and transfers it to the color discrimination section 35 .
  • FIG. 13 is a functional block diagram showing the internal configuration of the color discrimination section 35. As shown in FIG. As shown in FIG. 13 , the color discrimination section 35 has a learning section 81 and an inference section 82 .
  • the intermediate layer 92 is composed of only one layer, but the intermediate layer 92 may be composed of multiple layers. As the number of hierarchies of the intermediate layer 92 is increased, the learning effect can be enhanced, although the learning takes longer.
  • the first to fifth reference images are generated for one target member 5 whose color and surface treatment specifications are known, and using these first to fifth reference images, the weight of the neural network 90 You may make it update a coefficient.
  • 15A and 15B are diagrams showing examples in which the posture of the same target member 5 is changed variously. As shown in the figure, even if the color of the target member 5 is the same, the brightness of the polarized image captured by the camera system 71 may partially change depending on the orientation of the target member 5, and may be recognized as a different color. be. Especially when the surface of the target member 5 includes a curved surface, the color tends to change depending on the posture. Therefore, the learning unit 81 may perform the learning process of the neural network 90 using multi-orientation reference images obtained by variously changing the orientation of the target member 5 whose color and surface treatment specifications are known.
  • the learning unit 81 may perform learning processing for the neural network 90 using a plurality of reference images captured under a plurality of environments with different exposure conditions and white balances.
  • the learning unit 81 may acquire the image of the target member 5 with the new color and perform the learning process again.
  • the inference unit 82 preferably performs inference processing based on the re-learned result of the learning unit 81 .
  • the second embodiment in consideration of changes in environmental conditions and photographing conditions, presence/absence of reflection on the target member 5, changes in posture of the target member 5, target members 5 with similar colors, and the like, a plurality of reference By performing learning processing of the neural network 90 based on the image, it is possible to perform color discrimination processing with excellent robustness and expandability.
  • the color discrimination device 30 according to the third embodiment has the same block configuration as that of FIG. 12, but the processing operation of the color discrimination section 35 is different from that of the second embodiment.
  • a signal processing unit 74 according to the third embodiment performs processing of the RAW image processing unit 31, the image extraction unit 32, the color conversion unit 33, and the image acquisition unit 34 shown in FIG. Unlike the second embodiment, the color conversion unit 33 is essential in the third embodiment.
  • the camera system 71 photographs the target member 5 whose color is to be determined to obtain a visible image, a reflection suppression image, and a reflection component image (step S12). Each acquired image is converted into numerical data by the color conversion section 33 .
  • step S13 the difference ⁇ E between the digitized data of each image acquired in step S12 and the digitized data of the reference image is calculated.
  • the color conversion unit 33 generates digitized data by digitizing the RGB pixel data for each pixel.
  • step S13 a difference ⁇ E between values obtained by averaging numerical data of a plurality of pixels in the partial image extracted by the image extracting unit 32 is calculated.
  • the average value of the numerical data of each pixel in each partial image of the visible image, the reflection suppressed image, and the reflection component image of the target member 5 whose color is to be discriminated the visible image, the reflection suppressed image, which constitute the reference image, and the difference ⁇ E from the average value of the numerical data of each pixel in each partial image of the reflection component image.
  • step S14 the sum of the three types of ⁇ E calculated in step S13 is calculated (step S14).
  • a reference image with a smaller total value is closer to the color of the target member 5 .
  • the color discrimination device 30 according to the first to third embodiments can be applied to various use cases. For example, in a manufacturing factory that manufactures some parts, it is possible to identify the material of the part (for example, glass material, metal material, resin material, etc.) by distinguishing the color of the surface of the part.
  • the material of the part for example, glass material, metal material, resin material, etc.
  • the drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs.
  • the driving system control unit 12010 includes a driving force generator for generating driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism to adjust and a brake device to generate braking force of the vehicle.
  • the vehicle exterior information detection unit 12030 detects information outside the vehicle in which the vehicle control system 12000 is installed.
  • the vehicle exterior information detection unit 12030 is connected with an imaging section 12031 .
  • the vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image of the exterior of the vehicle, and receives the captured image.
  • the vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as people, vehicles, obstacles, signs, or characters on the road surface based on the received image.
  • the microcomputer 12051 calculates control target values for the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and controls the drive system control unit.
  • a control command can be output to 12010 .
  • the microcomputer 12051 realizes the functions of ADAS (Advanced Driver Assistance System) including collision avoidance or shock mitigation of vehicles, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, etc. Cooperative control can be performed for the purpose of ADAS (Advanced Driver Assistance System) including collision avoidance or shock mitigation of vehicles, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, etc. Cooperative control can be performed for the purpose of ADAS (Advanced Driver Assistance System) including collision avoidance or shock mitigation of vehicles, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving
  • At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information.
  • at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.
  • the microcomputer 12051 determines the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and changes in this distance over time (relative velocity with respect to the vehicle 12100). , it is possible to extract, as the preceding vehicle, the closest three-dimensional object on the traveling path of the vehicle 12100, which runs at a predetermined speed (for example, 0 km/h or more) in substantially the same direction as the vehicle 12100. can. Furthermore, the microcomputer 12051 can set the inter-vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including following stop control) and automatic acceleration control (including following start control). In this way, cooperative control can be performed for the purpose of automatic driving in which the vehicle runs autonomously without relying on the operation of the driver.
  • automatic brake control including following stop control
  • automatic acceleration control including following start control
  • the microcomputer 12051 converts three-dimensional object data related to three-dimensional objects to other three-dimensional objects such as motorcycles, ordinary vehicles, large vehicles, pedestrians, and utility poles. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into those that are visible to the driver of the vehicle 12100 and those that are difficult to see. Then, the microcomputer 12051 judges the collision risk indicating the degree of danger of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, an audio speaker 12061 and a display unit 12062 are displayed. By outputting an alarm to the driver via the drive system control unit 12010 and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be performed.
  • At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays.
  • the microcomputer 12051 can recognize a pedestrian by determining whether or not the pedestrian exists in the captured images of the imaging units 12101 to 12104 .
  • recognition of a pedestrian is performed by, for example, a procedure for extracting feature points in images captured by the imaging units 12101 to 12104 as infrared cameras, and performing pattern matching processing on a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian.
  • the technology according to the present disclosure can be applied to the imaging unit 12031 and the like among the configurations described above.
  • the color discrimination device 30 of the present disclosure can be applied to the imaging unit 12031 .
  • an image extraction unit that extracts a partial image of a specific region of the subject from each of the two or more types of images, and extracts a partial reference image of the specific region from each of the plurality of reference images; prepared, (2) or (3), wherein the color discrimination unit discriminates the color of the subject based on a result of comparing each of the partial images of two or more types and each of the plurality of partial reference images. color discriminator.
  • each of the plurality of reference images has a reference visible image including a visible light component, a reference reflection suppressed image in which the reflected light component is suppressed, and a reference reflected component image in which the reflected light component is extracted;
  • the difference calculation unit digitizes each of the visible image, the reflection suppression image, and the reflection component image corresponding to the two or more types of images by the digitization unit, and each of the plurality of reference images.
  • the color according to (12), wherein the difference is calculated based on values obtained by digitizing each of the reference visible image, the reference reflection suppression image, and the reference reflection component image corresponding to Discriminator.
  • the difference calculator calculates a first difference between a value obtained by digitizing the visible image by the digitizing unit and a value obtained by digitizing the reference visible image by the digitizing unit, and the reflection suppressed image. a second difference between the value digitized by the digitization unit and the value digitized from the reference reflection suppression image by the digitization unit; the value digitized for the reflection component image by the digitization unit; calculating a value obtained by summing the third difference between the reference reflection component image and the value digitized by the digitization unit;
  • the color discrimination device determines the color of the reference image that minimizes the total value calculated by the difference calculation section as the color of the subject.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Spectrometry And Color Measurement (AREA)
PCT/JP2022/016636 2021-08-30 2022-03-31 色判別装置及び色判別方法 Ceased WO2023032338A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US18/293,748 US20240338852A1 (en) 2021-08-30 2022-03-31 Color discrimination device and color discrimination method
JP2023545074A JPWO2023032338A1 (https=) 2021-08-30 2022-03-31

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021140498 2021-08-30
JP2021-140498 2021-08-30

Publications (1)

Publication Number Publication Date
WO2023032338A1 true WO2023032338A1 (ja) 2023-03-09

Family

ID=85412046

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/016636 Ceased WO2023032338A1 (ja) 2021-08-30 2022-03-31 色判別装置及び色判別方法

Country Status (3)

Country Link
US (1) US20240338852A1 (https=)
JP (1) JPWO2023032338A1 (https=)
WO (1) WO2023032338A1 (https=)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007047045A (ja) * 2005-08-10 2007-02-22 Olympus Corp 画像処理装置及び方法並びにプログラム
JP2009055624A (ja) * 2007-05-31 2009-03-12 Panasonic Corp カラー偏光撮像装置および画像処理装置
US20140293091A1 (en) * 2012-05-21 2014-10-02 Digimarc Corporation Sensor-synchronized spectrally-structured-light imaging
JP2018124814A (ja) * 2017-02-01 2018-08-09 キヤノン株式会社 画像処理装置、撮像装置、画像処理方法、画像処理プログラム、および、記憶媒体
JP2019080223A (ja) * 2017-10-26 2019-05-23 株式会社ソニー・インタラクティブエンタテインメント カメラシステム
WO2020262615A1 (ja) * 2019-06-28 2020-12-30 関西ペイント株式会社 光輝性顔料判定方法、光輝性顔料判定装置および光輝性顔料判定プログラム

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2089332A1 (en) * 1992-03-12 1993-09-13 Robert Bishop Method of and apparatus for object or surface inspection employing multicolor reflection discrimination
JPH09163382A (ja) * 1995-12-07 1997-06-20 Toyo Ink Mfg Co Ltd 色ずれ修正方法及び装置
JP4274632B2 (ja) * 1999-05-25 2009-06-10 オリンパス株式会社 色再現システム
US7158672B2 (en) * 2003-06-11 2007-01-02 E. I. Du Pont De Nemours And Company Recipe calculation method for matt color shades
US9208394B2 (en) * 2005-09-05 2015-12-08 Alpvision S.A. Authentication of an article of manufacture using an image of the microstructure of it surface
US7602493B2 (en) * 2006-02-14 2009-10-13 John Ramirez Electronic color matching apparatus and method of display
US20120321759A1 (en) * 2007-01-05 2012-12-20 Myskin, Inc. Characterization of food materials by optomagnetic fingerprinting
FI124452B (fi) * 2010-07-09 2014-09-15 Teknologian Tutkimuskeskus Vtt Menetelmä ja laite pinnan värin ja muiden ominaisuuksien mittaamiseksi
WO2013081833A1 (en) * 2011-11-28 2013-06-06 U.S. Coatings Ip Co. Llc Colour recipe calculating method for matt colour standards
WO2013081834A1 (en) * 2011-11-28 2013-06-06 U.S. Coatings Ip Co. Llc Method for determining the surface gloss of a colour standard
GB2532075A (en) * 2014-11-10 2016-05-11 Lego As System and method for toy recognition and detection based on convolutional neural networks
JP6729428B2 (ja) * 2017-02-01 2020-07-22 オムロン株式会社 画像処理システム、光学センサ、及び学習装置
JP7272349B2 (ja) * 2018-03-22 2023-05-12 凸版印刷株式会社 色対応情報生成システム、プログラム及び色対応情報生成方法
US11922662B2 (en) * 2020-11-02 2024-03-05 Datacolor Inc. Matching two color measurement devices using artificial neural network
EP4046594A1 (de) * 2021-02-18 2022-08-24 Ivoclar Vivadent AG Verfahren zur festlegung einer zahnfarbe
EP4309125A4 (en) * 2021-03-15 2025-01-22 Ortelligence, Inc. SYSTEMS AND METHODS FOR DYNAMIC IDENTIFICATION OF A SURGICAL TRAY AND ITEMS CONTAINED THEREIN

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007047045A (ja) * 2005-08-10 2007-02-22 Olympus Corp 画像処理装置及び方法並びにプログラム
JP2009055624A (ja) * 2007-05-31 2009-03-12 Panasonic Corp カラー偏光撮像装置および画像処理装置
US20140293091A1 (en) * 2012-05-21 2014-10-02 Digimarc Corporation Sensor-synchronized spectrally-structured-light imaging
JP2018124814A (ja) * 2017-02-01 2018-08-09 キヤノン株式会社 画像処理装置、撮像装置、画像処理方法、画像処理プログラム、および、記憶媒体
JP2019080223A (ja) * 2017-10-26 2019-05-23 株式会社ソニー・インタラクティブエンタテインメント カメラシステム
WO2020262615A1 (ja) * 2019-06-28 2020-12-30 関西ペイント株式会社 光輝性顔料判定方法、光輝性顔料判定装置および光輝性顔料判定プログラム

Also Published As

Publication number Publication date
JPWO2023032338A1 (https=) 2023-03-09
US20240338852A1 (en) 2024-10-10

Similar Documents

Publication Publication Date Title
US11910099B2 (en) Real-time HDR video for vehicle control
US12302004B2 (en) Image recognition device and image recognition method
JP7372322B2 (ja) 色覚困難の知覚を改善する多重スペクトル多重偏光（ｍｓｍｐ）フィルタ処理
US20150042808A1 (en) Vehicle vision system with image classification
JP2022513428A (ja) 知覚を改善するための多重チャネル多重偏光イメージング
TWI775808B (zh) 攝像裝置、攝像模組、攝像系統及攝像裝置之控制方法
WO2018179671A1 (ja) 画像処理装置と画像処理方法および撮像装置
US10372139B2 (en) Color filter array for machine vision system
CN114270798B (zh) 摄像装置
CN116320794A (zh) 从配备摄像头的车辆上的窄光谱带数据生成人工彩色图像
US12604105B2 (en) Signal processing device and method, and program
KR20230142640A (ko) 깊이 맵에 신뢰도 추정치를 제공하는 방법 및 시스템
US20200014899A1 (en) Imaging device, imaging system, and method of controlling imaging device
US10630952B2 (en) Image sensor
EP3182453A1 (en) Image sensor for a vision device and vision method for a motor vehicle
WO2023032338A1 (ja) 色判別装置及び色判別方法
US11347974B2 (en) Automated system for determining performance of vehicular vision systems
Kidono et al. Visibility estimation under night-time conditions using a multiband camera
WO2021166601A1 (ja) 撮像装置、および撮像方法
WO2025062831A1 (ja) 画像処理装置、画像処理方法、および、プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22863928

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023545074

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22863928

Country of ref document: EP

Kind code of ref document: A1