WO2024203225A1 - 画像処理装置、画像処理方法及びプログラム、撮影システム - Google Patents

画像処理装置、画像処理方法及びプログラム、撮影システム Download PDF

Info

Publication number
WO2024203225A1
WO2024203225A1 PCT/JP2024/009326 JP2024009326W WO2024203225A1 WO 2024203225 A1 WO2024203225 A1 WO 2024203225A1 JP 2024009326 W JP2024009326 W JP 2024009326W WO 2024203225 A1 WO2024203225 A1 WO 2024203225A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
visible light
processor
infrared
specific
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2024/009326
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
冨田　晃
卓 山下
篤志 松嶋
鈴木　比奈子
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Priority to JP2025510229A priority Critical patent/JPWO2024203225A1/ja
Publication of WO2024203225A1 publication Critical patent/WO2024203225A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects

Definitions

  • the present invention relates to an image processing device, an image processing method and program, and an imaging system, and in particular to a technology for detecting areas of a specific temperature, such as an animal, using two types of cameras.
  • Patent Document 1 describes an image processing device that includes an area extraction unit that extracts an area of interest in a visible light image captured by a visible light camera, a modal conversion unit that converts an image of the area of interest in a far-infrared image captured by a far-infrared camera observing the same object as the visible light camera into a modal image that reproduces the surface texture or contour/posture of an object detected in the area of interest based on the far-infrared image, and a superimposition unit that generates a presentation image by superimposing the modal image on the area of interest in the visible light image.
  • Patent Document 2 describes a surveillance system having a lidar that outputs a distance image in which the distribution of distance values obtained by scanning a laser beam toward a first region is shown in a three-dimensional coordinate system, an infrared camera that captures a second region that at least partially overlaps with the first region and outputs an infrared image shown in a two-dimensional coordinate system, a control unit that acquires a distance image from the lidar, detects an object from the acquired distance image, acquires an infrared image from the infrared camera, associates a temperature corresponding to the position of the detected object from the temperature distribution obtained from the infrared image, and outputs an image that displays information on the position of the object in the three-dimensional coordinate system and temperature information together with the object, and a display that displays the image from the control unit.
  • Patent document 3 describes a night vision device that includes a first imaging unit that images a first area based on visible light, a second imaging unit that images a second area based on infrared light, an extraction unit that extracts an object from the image of the second area, and an output unit that outputs information indicating the object along with the image of the first area.
  • Patent No. 7188397 International Publication No. 2019/163212 International Publication No. 2016/171190
  • One embodiment of the technology disclosed herein provides an image processing device, an image processing method and program, and an imaging system that use an infrared image to detect an area of a specific temperature in a visible light image.
  • the image processing device is an image processing device that includes at least one processor and at least one memory that stores instructions to be executed by the processor, and the processor acquires a visible light image obtained by imaging in a wavelength range including the visible light range and an infrared image obtained by imaging in a wavelength range including the infrared range, acquires the position and size of a specific temperature area in the visible light image based on temperature information indicated by the infrared image, and generates an image for display in which a specific image that identifies the specific temperature area according to the temperature of the specific temperature area is superimposed on the visible light image.
  • the processor in the image processing device according to the first aspect, it is preferable that the processor generates a display image in which a specific image is superimposed on the visible light image when the amount of movement within the angle of view of the visible light image of the specific temperature region is equal to or greater than a threshold value.
  • the processor obtains the specific temperature region by excluding noise regions of the infrared image.
  • the specific image includes a frame surrounding the specific temperature region.
  • the processor determines the color of the frame depending on the temperature of the specific temperature region.
  • the processor determines the thickness of the frame according to the temperature of the specific temperature region.
  • the processor obtains a specific temperature region outside the angle of view of the visible light image based on temperature information indicated by the infrared image, the temperature information having a certain range of temperatures, and generates a display image in which an arrow indicating the direction in which the specific temperature region exists is superimposed on the visible light image.
  • the processor determines the color of the arrow according to the temperature of the specific temperature region.
  • the processor determines the thickness of the arrow according to the temperature of the specific temperature region.
  • the processor preferably receives a user's designation of the type of animal, detects a plurality of animals in the visible light image based on temperature information indicated by the infrared image, obtains information on at least one of the shape, temperature, and color of the plurality of animals detected based on at least one of the visible light image and the infrared image, identifies the types of the plurality of animals detected based on the information, and generates a display image in which a first specific image that identifies an area of a first animal that is relatively closest in type to an animal of the designated type among the plurality of animals detected, and a second specific image different from the first specific image that identifies an area of the second animal that is different from the first animal among the plurality of animals detected, are superimposed on the visible light image.
  • the image processing device may be an image processing device that includes at least one processor and at least one memory that stores instructions to be executed by the processor, and the processor may receive a user's designation of the type of animal, acquire a visible light image obtained by imaging in a wavelength range including the visible light range and an infrared image obtained by imaging in a wavelength range including the infrared range, detect multiple animals in the visible light image based on temperature information indicated by the infrared image, acquire information on at least one of the shape, temperature, and color of the multiple animals detected based on at least one of the visible light image and the infrared image, identify the types of the multiple animals detected based on the information, and generate a display image in which a first specific image that identifies an area of a first animal that is relatively closest in type to an animal of the designated type among the multiple animals detected, and a second specific image that is different from the first specific image and identifies an area of a second animal that is different from the first animal among the multiple animals detected
  • the processor preferably receives a user's designation of the type of animal, detects a plurality of animals outside the angle of view of the visible light image based on temperature information indicated by the infrared image, obtains at least one of shape and temperature information of the plurality of animals detected based on the infrared image, identifies the types of the plurality of animals detected based on the information, and generates a display image in which a first arrow indicating the direction of a first animal of the plurality of detected animals that is relatively closest in type to the specified type of animal, and a second arrow different from the first arrow indicating the direction of a second animal of the plurality of detected animals that is different from the first animal, are superimposed on the visible light image.
  • the processor in the image processing device according to the twelfth aspect of the present disclosure, in the image processing device according to any one of the first to eleventh aspects, it is preferable that the processor generates a display image in which a first specific image that identifies a current specific temperature area and a second specific image that is different from the first specific image and identifies a past specific temperature area are superimposed on the visible light image.
  • the processor in the image processing device according to the thirteenth aspect of the present disclosure, in the image processing device according to any one of the first to twelfth aspects, it is preferable that the processor generates a display image in which a specific image that identifies the current specific temperature area and a line that indicates the trajectory of the movement of the specific temperature area are superimposed.
  • the processor preferably obtains a specific temperature region outside the angle of view of the visible light image based on temperature information indicated by the infrared image, predicts the future position of the specific temperature region when the amount of movement of the specific temperature region within the angle of view of the visible light image is equal to or greater than a threshold, and generates a display image in which an arrow indicating the direction of the future position of the specific temperature region is superimposed on the visible light image.
  • the imaging system is an imaging system including a visible light camera that captures visible light images, an infrared camera that captures infrared images, a monitor that displays an image for display, and an image processing device according to any one of the first to fourteenth aspects.
  • the image processing method is an image processing method in which at least one processor acquires a visible light image obtained by imaging in a wavelength range including the visible light range and an infrared image obtained by imaging in a wavelength range including the infrared range, acquires the position and size of a specific temperature area in the visible light image based on temperature information indicated by the infrared image, and generates an image for display in which a specific image that identifies the specific temperature area according to the temperature of the specific temperature area is superimposed on the visible light image.
  • the program according to the seventeenth aspect of the present disclosure is a program that causes a computer to execute the image processing method according to the sixteenth aspect.
  • This disclosure also includes non-transitory computer-readable recording media, such as a CD-ROM (Compact Disk-Read Only Memory) that stores a program relating to the seventeenth aspect.
  • CD-ROM Compact Disk-Read Only Memory
  • FIG. 1 is a perspective view of an imaging system including an image processing device according to the present disclosure, as viewed obliquely from the front.
  • FIG. 2 is a rear view of the imaging system shown in FIG.
  • FIG. 3 is a diagram showing an example of the relationship between the angle of view of a visible light camera and the angle of view of an infrared camera.
  • FIG. 4 is a block diagram showing an example of the internal configuration of the imaging system.
  • FIG. 5 is a flowchart showing steps of a display control method for live view images using the imaging system according to the first embodiment.
  • FIG. 6 is a diagram showing an example of a live view image displayed on the EVF or LCD.
  • FIG. 7 is a flow chart showing steps of a method for detecting warm-blooded animals.
  • FIG. 1 is a perspective view of an imaging system including an image processing device according to the present disclosure, as viewed obliquely from the front.
  • FIG. 2 is a rear view of the imaging system shown in FIG.
  • FIG. 8 is a table showing an example of the relationship between the temperature of the high-temperature portion and the color and thickness of the frame.
  • FIG. 9 is a diagram for explaining a process to be performed when a high-temperature portion exists outside the angle of view of a visible light image.
  • FIG. 10 is a table showing an example of the relationship between the temperature of the high-temperature portion and the color and thickness of the arrow.
  • FIG. 11 is a flowchart showing steps of a live view image display control method according to the fifth embodiment.
  • FIG. 12 is a diagram showing an example of a live view image displayed on the EVF or LCD.
  • FIG. 13 is a diagram showing an example of a live view image displayed on the EVF or LCD.
  • FIG. 14 is a flowchart showing steps of a live view image display control method according to the seventh embodiment.
  • FIG. 15 is a diagram showing an example of a live view image.
  • FIG. 16 is a diagram for explaining a live view image displayed on the EVF or
  • ⁇ Shooting system> When photographing wild animals and the like with a camera for consumer use, it is difficult for the camera or the photographer to find the animal. For example, when a telephoto lens is used, the angle of view is narrow and the animal may not be included in the angle of view of the camera. Also, the animal may not be found because the animal and the background are the same color.
  • the photography system according to the present disclosure makes it easy to find the animal by informing the user, who is the photographer, of the animal's location.
  • FIG. 1 is a perspective view of an imaging system including an image processing device according to the present disclosure, seen diagonally from the front, and FIG. 2 is a rear view of the imaging system shown in FIG. 1.
  • the imaging system 10 includes a visible light camera 100 and an infrared camera 200.
  • the visible light camera 100 captures a visible light image by capturing an image in a wavelength range that includes the visible light range.
  • the visible light range refers to the wavelength range of electromagnetic waves from 360 nm to 830 nm.
  • a visible light image may be captured in a wavelength range that includes at least a portion of the visible light range.
  • the visible light camera 100 is a mirrorless digital single-lens camera that is composed of an interchangeable lens 102 and a camera body 104 to which the interchangeable lens 102 can be attached or detached.
  • a body mount (not shown) on which the interchangeable lens 102 is attached is provided on the front of the camera body 104.
  • On the top surface of the camera body 104 mainly the shutter release button 22, the shutter speed dial 23, the exposure compensation dial 24, the power lever 25, and the hot shoe 30 are provided.
  • the rear of the camera body 104 is provided with an EVF (Electronic View Finder) 26, a MENU/OK key 27, a cross key 28, a playback button 29, and an LCD (Liquid Crystal Display) 40.
  • EVF Electronic View Finder
  • MENU/OK key a MENU/OK key
  • MENU/OK key a MENU/OK key
  • MENU/OK key a MENU/OK key
  • LCD Liquid Crystal Display
  • the LCD 40 functions as a display that displays a live view image in shooting mode, plays back and displays captured images in playback mode, and also displays various menu screens. Note that when you bring your eye close to the EVF 26 in shooting mode, an eye sensor (not shown) automatically switches the display to the EVF 26, and when you move your eye away, the display switches to the LCD 40.
  • the EVF 26 and LCD 40 correspond to the "monitor" in this disclosure.
  • the MENU/OK key 27 is an operation key that functions both as a menu button for issuing a command to display a menu on the screen of the LCD 40, and as an OK button for issuing a command to confirm and execute a selection.
  • the cross key 28 is an operation unit for inputting instructions in four directions, up, down, left, and right, and functions as a button for selecting an item from a menu screen and for instructing the selection of various setting items from each menu.
  • the up and down keys of the cross key 28 function as a zoom switch when shooting or a playback zoom switch when in playback mode
  • the left and right keys function as frame advance (forward and reverse advance) buttons when in playback mode.
  • the playback button 29 is a button for switching to a playback mode in which a captured still image or video is displayed on the LCD 40.
  • the hot shoe 30 is an attachment part for attaching an external accessory to the visible light camera 100.
  • the external accessory and the visible light camera 100 can perform bidirectional communication control via electrical contacts provided on the hot shoe 30.
  • an infrared camera 200 is attached to a hot shoe 30.
  • the visible light camera 100 and the infrared camera 200 are connected via a USB (Universal Serial Bus) standard communication cable 12 so that data can be sent and received.
  • USB Universal Serial Bus
  • the infrared camera 200 captures an infrared image that corresponds to the visible light image captured by the visible light camera 100 by capturing an image in a wavelength range that includes at least a portion of the infrared range.
  • the infrared range refers to a wavelength range of electromagnetic waves between about 0.8 ⁇ m and 1000 ⁇ m.
  • the infrared image may be captured in a wavelength range that includes at least a portion of the infrared range.
  • Corresponding to a visible light image means that the infrared image is captured at approximately the same time and with approximately the same angle of view as the visible light image.
  • the visible light image and the infrared image are captured at approximately the same time, but they do not have to be captured exactly at the same time. They may be captured with a time difference within a range that allows for an acceptable shift in the position of a common subject between the images due to the time difference in the capture timing of the two images.
  • the visible light image and the infrared image may be captured within a time period that can be considered to be approximately simultaneous, including a time difference within a range that is perceived as substantially simultaneous by human perception.
  • the infrared camera 200 includes the concept of a thermography device and a thermal camera that obtains a heat map image that visualizes the temperature distribution of a subject.
  • a heat map image may also be called a thermography image (thermal image).
  • the infrared image may be an image before it is converted into a heat map, or it may be a heat map image.
  • the term "infrared image” includes the concept of a heat map image, unless there is a contradiction in the context.
  • the infrared camera 200 is an infrared sensor composed of an infrared lens 202 and a camera body 204 to which the infrared lens 202 is attached.
  • the infrared camera 200 may be equipped with a light source such as an infrared LED (Light Emitting Diode) that projects infrared rays toward a subject.
  • a light source such as an infrared LED (Light Emitting Diode) that projects infrared rays toward a subject.
  • the infrared lens 202 is a lens that selectively transmits and focuses light in the infrared range.
  • the optical axis of the infrared lens 202 may be parallel to the optical axis of the interchangeable lens 102 of the visible light camera 100.
  • FIG. 3 is a diagram showing an example of the relationship between the angle of view AV of the visible light camera 100 and the angle of view AI of the infrared camera 200.
  • F3A in FIG. 3 shows a case where the optical axis of the interchangeable lens 102 of the visible light camera 100 and the optical axis of the infrared lens 202 of the infrared camera 200 are parallel, the focal length of the interchangeable lens 102 and the focal length of the infrared lens 202 are the same, and the size of the image sensor 110 (see FIG. 4) of the visible light camera 100 and the size of the image sensor 210 (see FIG. 4) of the infrared camera 200 are the same.
  • the angles of view AV and AI have the same width, and the positions of the angles of view AV and AI are shifted up and down depending on the amount of vertical shift between the optical axis of the interchangeable lens 102 and the optical axis of the infrared lens 202.
  • the angle of view AV of the visible light camera 100 and the angle of view AI of the infrared camera 200 overlap at least partially.
  • F3B in FIG. 3 shows a case where the angle of view AI of the infrared camera 200 is wider than the angle of view AV of the visible light camera 100, and the respective optical axes are shifted up and down and left and right.
  • F3C in FIG. 3 shows a case where the angle of view AI of the infrared camera 200 is narrower than the angle of view AV of the visible light camera 100, and the respective optical axes are shifted up and down and left and right. Even in this relationship, the visible light image and the infrared image partially overlap, and have approximately the same angle of view.
  • the imaging system 10 acquires in advance the correspondence between the pixels of the visible light image and the infrared image according to the difference in the angle of view between the two images.
  • the correspondence between the positions of the pixels of the visible light image and the infrared image can be determined based on conditions such as the configuration of the imaging optical systems of the visible light camera 100 and the infrared camera 200 and the spatial arrangement of the two cameras.
  • the correspondence between the pixel positions of the two images can be determined based on information on corresponding points of the subject that are common to both images.
  • the imaging system 10 is not limited to a system configuration that combines the visible light camera 100 and the infrared camera 200, but may also be configured to capture visible light images and infrared images of the same imaging range at the same angle of view by splitting the optical path of the imaging optical system, such as a single-lens camera equipped with both a visible light sensor and an infrared sensor.
  • Fig. 4 is a block diagram showing an example of the internal configuration of the imaging system 10.
  • the visible light camera 100 includes an image sensor 110, a processor 120, a memory 130, a display driver 140, an operation unit 150, an input/output interface 160, a sensor driver 170, and an AFE (Analog Front End) 180.
  • the image sensor 110 is composed of a CMOS (Complementary Metal-Oxide Semiconductor) type color image sensor. Note that the image sensor 110 is not limited to a CMOS type, and may also be a CCD (Charge Coupled Device) type image sensor.
  • CMOS Complementary Metal-Oxide Semiconductor
  • CCD Charge Coupled Device
  • the image sensor 110 is made up of multiple pixels composed of photoelectric conversion elements (photodiodes) arranged two-dimensionally in the x direction (horizontal direction) and y direction (vertical direction), on which red (R), green (G), and blue (B) color filters are arranged in a periodic color array (e.g., Bayer array, X-Trans (registered trademark), etc.), and a microlens is arranged on each photodiode.
  • photoelectric conversion elements photodiodes
  • x direction horizontal direction
  • y direction vertical direction
  • red (R), green (G), and blue (B) color filters are arranged in a periodic color array (e.g., Bayer array, X-Trans (registered trademark), etc.)
  • a microlens is arranged on each photodiode.
  • the optical image of the subject formed on the light receiving surface of the image sensor 110 by the imaging optical system of the interchangeable lens 102 is converted into an electrical signal by the image sensor 110.
  • An electric charge corresponding to the amount of incident light is accumulated in each pixel of the image sensor 110, and an electrical signal corresponding to the amount of electric charge (signal charge) accumulated in each pixel is read out from the image sensor 110 as an image signal.
  • the processor 120 executes instructions stored in the memory 130.
  • the hardware structure of the processor 120 is various processors as shown below.
  • the various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and acts as various functional units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically to execute specific processing.
  • a CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • PLD Programmable Logic Device
  • ASIC Application Specific Integrated Circuit
  • a single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU).
  • Multiple functional units may also be configured with one processor.
  • multiple functional units As an example of configuring multiple functional units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as represented by a computer such as a client or server, and this processor acts as multiple functional units. Second, there is a form in which a processor is used that realizes the functions of the entire system including multiple functional units with a single IC (Integrated Circuit) chip, as represented by a SoC (System On Chip). In this way, the various functional units are configured using one or more of the above-mentioned various processors as a hardware structure.
  • IC Integrated Circuit
  • the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.
  • the processor 120 performs overall control of each part of the visible light camera 100 and various processes in accordance with user operations using the operation unit 150.
  • the various processes performed by the processor 120 include a process that uses an infrared image to assist in capturing a visible light image, which will be described later.
  • Memory 130 stores instructions for processor 120 to execute.
  • Memory 130 includes a flash memory (not shown), a RAM (Random Access Memory) (not shown), and a ROM (Read Only Memory) (not shown).
  • Memory 130 includes a memory card that is removable from camera body 104.
  • the flash memory and ROM are non-volatile memories that store firmware, various programs including the display control program related to this disclosure, and captured images (still images, videos), etc.
  • the RAM functions as a working area for processing by the processor 120, and also temporarily stores firmware and display control programs stored in the non-volatile memory. Note that the processor 120 may also incorporate a portion of the memory 130 (RAM).
  • the display driver 140 converts the input digital image signal into a signal format for display and outputs it sequentially to the EVF 26 or LCD 40.
  • the operation unit 150 includes the shutter release button 22, the shutter speed dial 23, the exposure compensation dial 24, the power lever 25, the MENU/OK key 27, the cross key 28, the playback button 29, etc., shown in Figures 1 and 2.
  • the LCD 40 may be configured as a touch panel display and used as the operation unit 150.
  • the input/output interface 160 includes the hot shoe 30 shown in FIG. 1 and FIG. 2, and a connector portion (not shown) to which the communication cable 12 is connected.
  • the input/output interface 160 transmits and receives data and signals to and from the infrared camera 200.
  • the sensor driver 170 controls the reading of image signals from the image sensor 110 according to instructions from the processor 120.
  • the sensor driver 170 also has an electronic shutter function that, in response to an electronic shutter control signal from the processor 120, discharges (resets) the electric charge accumulated in each pixel of the image sensor 110 and starts exposure.
  • AFE 180 performs various analog signal processing on the analog image signal obtained by capturing an image of a subject with image sensor 110, and converts the processed image signal into a digital image signal.
  • Analog processing in AFE 180 includes, for example, color separation processing and AGC (Automatic Gain Control).
  • AGC functions as a sensitivity adjustment section that adjusts the sensitivity (ISO sensitivity (ISO: International Organization for Standardization)) during shooting, and adjusts the gain of the amplifier that amplifies the input image signal so that the signal level of the image signal falls within an appropriate range.
  • the RGB pixel-by-pixel image data (mosaic image data) output via the image sensor 110 and the AFE 180 is input to the memory 130 and temporarily stored therein.
  • the image sensor 110 is a CMOS image sensor
  • the AFE 180 is often built into the image sensor 110.
  • the processor 120 also functions as a digital signal processing unit that performs various types of digital signal processing on image data temporarily stored in the memory 130. That is, the processor 120 performs digital signal processing such as offset processing, gain control processing including sensitivity correction, gamma correction processing, demosaicing processing (also called demosaicing processing or synchronization processing), and RGB/YCrCb conversion processing on the image data input via the AFE 180, and stores the image data after digital signal processing back in the memory 130.
  • digital signal processing such as offset processing, gain control processing including sensitivity correction, gamma correction processing, demosaicing processing (also called demosaicing processing or synchronization processing), and RGB/YCrCb conversion processing on the image data input via the AFE 180, and stores the image data after digital signal processing back in the memory 130.
  • demosaicing is a process that calculates all RGB color information for each pixel from a mosaic image made of RGB, for example, in the case of the image sensor 110 made of RGB three-color filters, and generates synchronized RGB three-plane image data from the mosaic data (dot-sequential RGB data).
  • the RGB/YCrCb conversion process converts the synchronized RGB data into luminance data (Y) and color difference data (Cr, Cb).
  • the processor 120 when recording a still image or a moving image, the processor 120 performs a compression process on the uncompressed luminance data Y and color difference data Cb, Cr temporarily stored in the RAM of the memory 130.
  • the data In the case of a still image, the data is compressed in, for example, JPEG (Joint Photographic coding Experts Group) format, and in the case of a moving image, the data is compressed in, for example, H.264 format.
  • the compressed image data is recorded in the flash memory of the memory 130.
  • the processor 120 reads out the compressed image data from the flash memory of the memory 130 in the playback mode, performs an expansion process on the read image data, generates uncompressed image data, and displays it on the LCD 40 or the like via the display driver 140.
  • the processor 120 When displaying a live view image on the EVF 26 or LCD 40, the processor 120 captures visible light images at a predetermined frame rate (e.g., 30 fps (frames per second), 60 fps). That is, the sensor driver 170 controls the reading of image signals from the image sensor 110 at a predetermined frame rate. The processor 120 also outputs to the display driver 140 the digital image signals that have been read from the image sensor 110 and digitally processed by the AFE 180. The display driver 140 converts the input time-series digital image signals into a signal format for display and outputs them sequentially to the EVF 26 or LCD 40. This allows the live view image to be displayed in real time on the EVF 26 or LCD 40.
  • a predetermined frame rate e.g., 30 fps (frames per second), 60 fps.
  • the shutter release button 22 is a shooting instruction section for inputting shooting instructions for still images and videos, and is configured as a two-stroke switch with a so-called “half press” (S1 press) and “full press” (S2 press).
  • an S1_ON signal is output, and when it is pressed further from the "half-press” to the "full press,” an S2_ON signal is output.
  • the processor 120 executes shooting preparation processes such as AF control (Auto Focus control: automatic focus adjustment) and AE control (Auto Exposure control: automatic exposure control), and when the S2_ON signal is output, it executes still image shooting and recording processes.
  • the processor 120 calculates the values required for AF control based on the digital image signal.
  • contrast AF for example, the processor 120 calculates the integrated value (focus evaluation value) of the high frequency components of the G signal within a specified AF area.
  • the processor 120 moves the focus lens included in the lens group of the interchangeable lens 102 to the position where the focus evaluation value is maximized during AF control (i.e., the position where the contrast is maximized).
  • AF is not limited to contrast AF, and may be, for example, phase difference AF that detects the amount of defocus based on pixel data from phase difference detection pixels provided in the image sensor and moves the focus lens so that this defocus amount becomes zero.
  • the processor 120 When performing AE control, the processor 120 detects the brightness of the subject (subject luminance) and calculates a numerical value (exposure value (EV value)) required for AE control that corresponds to the subject luminance.
  • the processor 120 can determine the F-number, shutter speed, and ISO sensitivity from a specified program diagram based on the calculated EV value, and perform AE control.
  • AF control and AE control are performed automatically when the auto mode is set by the operation unit 150, and AF control and AE control are not performed when the manual mode is set.
  • the camera body 104 in the case of video recording mode, when the shutter release button 22 is fully pressed and an S2_ON signal is output, the camera body 104 enters a video recording mode in which video recording begins, and performs image processing and recording processing of the video. After that, when the shutter release button 22 is fully pressed again and an S2_ON signal is output, the camera body 104 enters a standby state and temporarily suspends the video recording processing.
  • the infrared camera 200 also includes an image sensor 210, an AFE 220, and an input/output interface 230.
  • the image sensor 210 receives light in a wavelength range including the infrared range and outputs an electrical signal.
  • the image sensor 210 may be a sensor having high sensitivity to any of the wavelength ranges of the near infrared range (0.8 ⁇ m to 2.5 ⁇ m), mid infrared range (2.5 ⁇ m to 4.0 ⁇ m), and far infrared range (4.0 ⁇ m to 1000 ⁇ m). Note that as an infrared sensor corresponding to the far infrared range, for example, a sensor having sensitivity in the range of 4.0 ⁇ m to 14 ⁇ m can be used.
  • the image sensor 210 can be a thermal type infrared sensor such as a microbolometer or SOI (Silicon on Insulator) diode type.
  • the resolution of the image sensor 210 and the resolution of the image sensor 110 may be the same or different.
  • the optical image of the subject formed on the light receiving surface of the image sensor 210 by the imaging optical system of the infrared lens 202 is converted into an electrical signal by the image sensor 210.
  • An electric charge corresponding to the amount of incident light is accumulated in each pixel of the image sensor 210, and an electrical signal corresponding to the amount of electric charge (signal charge) accumulated in each pixel is read out from the image sensor 210 as an image signal.
  • the AFE 220 performs various analog signal processing on the analog image signal obtained by capturing an image of a subject using the image sensor 210, and converts the image signal after the analog processing into a digital image signal.
  • each pixel in the infrared image captured by the infrared camera 200 indicates the intensity of the infrared light, which is proportional to the temperature of the subject.
  • the input/output interface 230 includes electrical contacts that are connected to the hot shoe 30 shown in Figures 1 and 2, and a connector portion (not shown) to which the communication cable 12 is connected.
  • the input/output interface 230 transmits and receives data and signals to and from the visible light camera 100.
  • the infrared image captured by the infrared camera 200 is output to the visible light camera 100 via the input/output interface 230.
  • the visible light camera 100 and the infrared camera 200 enter a shooting standby state.
  • the visible light camera 100 and the infrared camera 200 start shooting video.
  • the frame rate of the infrared camera 200 may be the same as the frame rate of the visible light camera 100.
  • the frame rate of the infrared camera 200 may be lower than the frame rate of the visible light camera 100.
  • the frame rate of the visible light camera 100 and the frame rate of the infrared camera 200 do not have to be synchronized. Even in this relationship, the visible light image and the infrared image are approximately synchronized.
  • the video captured by the infrared camera 200 is input to the visible light camera 100 via the communication cable 12.
  • the video captured by the visible light camera 100 is superimposed with a specific image (described later) and displayed as a live view image on the EVF 26 or LCD 40.
  • the user can visually check the live view image displayed on the EVF 26 or LCD 40 to determine the composition, confirm the subject they want to shoot, and set shooting conditions.
  • the following describes a method for controlling the display of live view images using the shooting system 10.
  • First Embodiment 5 is a flowchart showing steps of a display control method for a live view image according to the first embodiment.
  • the display control method is realized by the processor 120 reading and executing a display control program from the memory 130.
  • the display control program may be provided via the input/output interface 160.
  • step ST1 the imaging system 10 starts capturing video for live view images. That is, the visible light camera 100 captures visible light images, and the infrared camera 200 captures infrared images.
  • the captured infrared images are input to the visible light camera 100 via the input/output interface 160.
  • the following processing is performed for each frame.
  • step ST2 the processor 120 acquires temperature information indicated by the infrared image captured in step ST1.
  • step ST3 the processor 120 obtains the position and size of a high temperature area (an example of a "specific temperature area"), which is a relatively high temperature area in the infrared image, from the temperature information obtained in step ST2. Also, in step ST3, the processor 120 obtains the position and size of the high temperature area in the visible light image that corresponds to the high temperature area in the infrared image, taking into account the correspondence between the pixels of both images according to the difference in the angle of view between the visible light image and the infrared image.
  • a high temperature area an example of a "specific temperature area”
  • step ST4 the processor 120 generates a specific image that identifies the position and size of the high temperature area in the visible light image according to the temperature of the high temperature area in the infrared image.
  • the specific image may include a frame that surrounds the high-temperature portion of the visible light image.
  • the shape of the frame is not limited, and may be any shape such as a rectangle, polygon, circle, or ellipse, or may be a shape that follows the outline of the high-temperature portion.
  • the frame is not limited to a shape that surrounds the entire high-temperature portion, and may be a shape that surrounds more than half of the high-temperature portion.
  • step ST5 the processor 120 generates an image for display by superimposing the specific image generated in step ST4 on the visible light image captured in step ST1.
  • Superimposing the specific image on the visible light image means changing the pixels of the visible light image that correspond to the positions of the specific image using the pixels of the specific image.
  • the specific image may be transparent.
  • step ST6 the processor 120 causes the display image generated in step ST5 to be displayed as a live view image on the EVF 26 or the LCD 40. Then, the process returns to step ST1, and the imaging system 10 continues capturing and displaying the live view image.
  • FIG. 6 shows the live view image LV1 displayed on the EVF 26 or LCD 40 in step ST6.
  • the live view image LV1 is an image in which a frame FR1, which is a specific image, is superimposed on a visible light image (not shown).
  • the frame FR1 is a square with each side parallel to the angle of view of the visible light image, and surrounds the entire animal AN1, which corresponds to the high temperature area in the visible light image.
  • the user can find a subject in a high temperature area, such as an animal, from the position of the frame in the live view image, even if the visibility of the subject in the visible light image is poor. Furthermore, the user can determine the composition taking into account the position of the frame in the live view image, and take a still image of the visible light image of the animal by fully pressing the shutter release button 22. Note that the frame is not included in the still image of the visible light image taken by fully pressing the button.
  • steps ST1 to ST5 correspond to the image processing method according to the present disclosure.
  • the processor 120 acquires high temperature areas from an infrared image, but in the second embodiment, in order to extract only the animal area, the following areas are acquired:
  • the processor 120 detects high temperature areas within a certain temperature range as animals.
  • the temperature range is, for example, 30 degrees or higher and 45 degrees or lower.
  • the lower limit of the temperature range may be 10 degrees higher than the ambient temperature of the imaging system 10.
  • the temperature range may be changeable depending on the body temperature of the animal being photographed. If the subject is a human, dog, cat, etc., the temperature range may be 35 degrees or higher and 38 degrees or lower. Also, if the subject is a bird, the temperature range may be 39 degrees or higher and 45 degrees or lower.
  • Processor 120 removes noise areas from the infrared image to obtain high temperature areas.
  • a noise area is one where there is something over 100 degrees and there is radiation in the surrounding area.
  • Processor 120 divides the infrared image into multiple areas and removes noise within the areas, rather than for each pixel of the infrared image.
  • Processor 120 acquires the amount of movement within the angle of view of the visible light image of the high temperature area, and detects the high temperature area as an animal if the acquired amount of movement is equal to or greater than a threshold. That is, processor 120 generates an image for display in which a specific image is superimposed on the visible light image if the acquired amount of movement is equal to or greater than a threshold. Processor 120 acquires the amount of movement of the high temperature area by comparing the current visible light image with a past visible light image.
  • the threshold is, for example, an amount of movement in 0.1 seconds that is equal to or greater than 1% of the angle of view of the visible light image, and preferably an amount of movement in 0.1 seconds that is equal to or greater than 3% of the angle of view of the visible light image.
  • Figure 7 is a flowchart showing the steps of a method for detecting warm-blooded animals with surface temperatures between 30 and 45 degrees.
  • step ST11 the processor 120 detects high temperature areas in the visible light image captured by the visible light camera 100 from the infrared image captured by the infrared camera 200.
  • step ST12 processor 120 determines whether the high-temperature area detected in step ST11 is moving. If processor 120 determines that the high-temperature area is moving, it proceeds to processing in step ST13. If processor 120 determines that the high-temperature area is not moving, it proceeds to processing in step ST14.
  • processor 120 obtains the amount of movement within the angle of view of the visible light image of the high-temperature area, and determines that the high-temperature area is moving if the amount of movement is equal to or greater than a threshold value.
  • step ST13 the processor 120 determines that the high temperature area detected in step ST11 is an animal.
  • the processor 120 also generates a display image in which a specific image that specifies the position and size of the high temperature area in the visible light image and corresponds to the temperature of the high temperature area is superimposed on the visible light image, and the processing of this flowchart ends.
  • processor 120 determines whether or not a relatively large noise area is present in the visible light image. Processor 120 divides the visible light image into multiple regions, and if an area of 100 degrees or more exists, it determines that the area is a noise area. If processor 120 determines that a noise area exists, it proceeds to processing of step ST15. If processor 120 determines that a noise area does not exist, it proceeds to processing of step ST16.
  • step ST15 the processor 120 determines that the high temperature areas of the visible light image, excluding noise areas, that are between 30 and 45 degrees are homeothermic animals.
  • the processor 120 also generates a display image in which a specific image that specifies the position and size of the high temperature area and corresponds to the temperature of the high temperature area is superimposed on the visible light image, and the processing of this flowchart ends.
  • step ST16 the processor 120 determines that the high temperature areas of the visible light image between 30° and 45° are warm-blooded animals.
  • the processor 120 also generates a display image in which a specific image that specifies the position and size of the high temperature area and corresponds to the temperature of the high temperature area is superimposed on the visible light image, and the processing of this flowchart ends.
  • high temperature areas where the amount of movement within the angle of view of the visible light image is equal to or exceeds a threshold, or high temperature areas of 30 to 45 degrees after removing noise areas, are determined to be warm-blooded animals, and a display image is generated in which a frame identifying the position of the warm-blooded animal is superimposed on the visible light image, thereby assisting in the photographing of warm-blooded animals.
  • the visibility of the frame display is improved.
  • the processor 120 changes the color of the frame superimposed on the visible light image according to the temperature of the high-temperature area obtained from the infrared image.
  • the processor 120 also changes the thickness of the frame superimposed on the visible light image according to the temperature of the high-temperature area obtained from the infrared image.
  • Figure 8 is a table showing an example of the relationship between the temperature of the high-temperature area and the color and thickness of the frame.
  • the color and thickness of the frame change every three degrees.
  • the temperature of the high-temperature area is 45 degrees
  • the color of the frame is red and the thickness of the frame is the thickest.
  • the temperature of the high-temperature area is 42 degrees
  • the color of the frame is orange and the thickness of the frame is the second thickest.
  • the temperature of the high-temperature area is 39 degrees
  • the color of the frame is orange + white (light orange) and the thickness of the frame is the third thickest.
  • the color of the frame is light blue + white (light light blue) and the thickness of the frame is the fourth thickest.
  • the temperature of the high-temperature area is 33 degrees, the color of the frame is light blue and the thickness of the frame is the fifth thickest.
  • the temperature of the high-temperature area is 30 degrees, the color of the frame is blue and the thickness of the frame is the thinnest.
  • the processor 120 selects the temperature closest to the temperature of the high temperature area from these six temperature levels, and determines the color and thickness of the frame that corresponds to the selected temperature. In this way, by making the frame a warmer color for higher temperatures and a cooler color for lower temperatures, the user can properly identify the temperature of the high temperature area. Also, by making the frame thicker for higher temperatures and thinner for lower temperatures, the user can properly identify the temperature of the high temperature area.
  • the processor 120 may change only one of the frame color and frame thickness and keep the other constant.
  • step ST3 the processor 120 obtains the position and size of high temperature areas, which are relatively high temperature areas in the infrared image, from the temperature information obtained in step ST2.
  • the processor 120 If a high-temperature area is present in the infrared image but not in the visible light image, the processor 120 generates a specific image in step ST4 that identifies the direction in which the high-temperature area is present outside the angle of view of the visible light image.
  • step ST5 the processor 120 generates a display image by superimposing the specific image on the visible light image, and in step ST6, displays the generated display image on the EVF 26 or the LCD 40.
  • the specific image includes an arrow pointing in the direction where the high temperature area is located.
  • the shape of the arrow is not limited, and includes any symbol that indicates a direction, such as an indication mark in the shape of a human hand.
  • the processor 120 may change the length of the arrow depending on the distance between the center of the angle of view of the visible light image and the high temperature area, in order to indicate the degree of deviation between the optical axis of the visible light camera 100 and the high temperature area.
  • FIG. 9 is a diagram for explaining processing when a high-temperature part exists outside the angle of view of a visible light image.
  • F9A in FIG. 9 is a diagram showing an example of the relationship between the angles of view of a captured visible light image PV and an infrared image PI.
  • an animal AN2 corresponding to the high-temperature part in the infrared image PI is outside the angle of view of the visible light image PV and does not exist in the visible light image PV.
  • F9B in Fig. 9 shows a live view image LV2 displayed on the EVF 26 or LCD 40 when the visible light image PV and infrared image PI shown in F9A are captured.
  • the live view image LV2 is an image in which an arrow AR1 of a specific image is superimposed on the visible light image PV.
  • the arrow AR1 is positioned on a straight line connecting the center of the angle of view of the visible light image PV and the animal AN2 corresponding to the high temperature area of the infrared image PI, and points in the direction of the animal AN2.
  • the visibility of the arrow display is improved.
  • the processor 120 changes the color of the arrow superimposed on the visible light image according to the temperature of the high-temperature area obtained from the infrared image.
  • the processor 120 also changes the thickness of the arrow superimposed on the visible light image according to the temperature of the high-temperature area obtained from the infrared image.
  • FIG. 10 is a table showing an example of the relationship between the temperature of the high-temperature part and the color and thickness of the arrow.
  • the color and thickness of the arrow are changed every three degrees.
  • the temperature of the high-temperature part is 45 degrees
  • the color of the arrow is red and the thickness of the arrow is the thickest.
  • the temperature of the high-temperature part is 42 degrees
  • the color of the arrow is orange and the thickness of the arrow is the second thickest.
  • the temperature of the high-temperature part is 39 degrees
  • the color of the arrow is orange + white (light orange) and the thickness of the arrow is the third thickest.
  • the color of the arrow is light blue + white (light light blue) and the thickness of the arrow is the fourth thickest.
  • the temperature of the high-temperature part is 33 degrees, the color of the arrow is light blue and the thickness of the arrow is the fifth thickest.
  • the temperature of the high-temperature part is 30 degrees, the color of the arrow is blue and the thickness of the arrow is the thinnest.
  • the processor 120 selects the temperature closest to the temperature of the high temperature area from these six temperature levels, and determines the color and thickness of the arrow that corresponds to the selected temperature. In this way, by making the arrow a warmer color for higher temperatures and a cooler color for lower temperatures, the user can properly identify the temperature of the high temperature area. Also, by making the arrow thicker for higher temperatures and thinner for lower temperatures, the user can properly identify the temperature of the high temperature area.
  • the processor 120 may change only one of the color and thickness of the arrow and keep the other constant. Furthermore, when the processor 120 detects a high temperature area within the angle of view of the visible light image and a high temperature area outside the angle of view of the visible light image, the processor 120 may display both a specific image of the frame and a specific image of the arrow in the live view image.
  • FIG. 11 is a flowchart showing the steps of a live view image display control method according to the fifth embodiment.
  • step ST21 the user specifies the type of animal they wish to photograph using the operation unit 150.
  • the processor 120 may display a number of animal types on the LCD 40 so that the user can select from them. For example, animal types such as “dog,” “cat,” “rabbit,” and “bird” may be displayed.
  • the processor 120 accepts the type of animal specified by the user.
  • step ST22 the imaging system 10 starts capturing live view images. That is, the visible light camera 100 captures visible light images, and the infrared camera 200 captures infrared images.
  • the captured infrared images are input to the visible light camera 100 via the input/output interface 160.
  • step ST23 the processor 120 acquires temperature information indicated by the infrared image captured in step ST22.
  • processor 120 detects animals present in the infrared image from the temperature information acquired in step ST23. Also in step ST24, processor 120 detects animals in the visible light image that correspond to animals present in the infrared image, taking into account the correspondence between the pixels of the visible light image and the infrared image according to the difference in the angle of view between the two images.
  • processor 120 detects animals present in the infrared image from the temperature information acquired in step ST23. Also in step ST24, processor 120 detects animals in the visible light image that correspond to animals present in the infrared image, taking into account the correspondence between the pixels of the visible light image and the infrared image according to the difference in the angle of view between the two images.
  • processor 120 detects animals present in the infrared image from the temperature information acquired in step ST23. Also in step ST24, processor 120 detects animals in the visible light image that correspond to animals present in the infrared image, taking into account the correspondence between the pixels of the visible light image and the infrared image according to the difference in the
  • step ST25 the processor 120 acquires at least one of information on shape, temperature, and color for each of the multiple animals detected from the visible light image based on at least one of the visible light image and the infrared image.
  • step ST26 the processor 120 identifies each of the types of animals detected from the visible light image based on the information acquired in step ST25.
  • processor 120 In step ST27, processor 120 generates a specific image that identifies the position and size of the animal in the visible light image, the specific image corresponding to the type of animal.
  • processor 120 generates a first specific image that identifies the area of a first animal that is relatively closest in type to the animal type specified in step ST21 among the multiple detected animals, and a second specific image that is different from the first specific image and identifies the area of the second animal that is different from the first animal among the multiple detected animals.
  • the first specific image and the second specific image each include a frame that surrounds the animal in the visible light image.
  • step ST28 the processor 120 generates a display image in which the first specific image and the second specific image are superimposed on the visible light image.
  • step ST29 the processor 120 displays the display image generated in step ST28 on the EVF 26 or LCD 40.
  • FIG. 12 is a diagram showing an example of a live view image LV3 displayed on the EVF 26 or LCD 40 in step ST29.
  • the live view image LV3 is an image in which frames FR2 and FR3, which are specific images, are superimposed on a visible light image (not shown).
  • Frame FR2 surrounds animal AN3 in the visible light image.
  • Frame FR3 surrounds animal AN4 in the visible light image.
  • animal AN3 is the animal that is relatively closest in type to the animal type accepted in step ST11.
  • frame FR2 surrounding animal AN3, which is the first specific image is a solid-line frame that corresponds to the animal type.
  • animal AN4 is a different type of animal from animal AN3, and is relatively distant in type from the animal type accepted in step ST11.
  • frame FR3 surrounding animal AN4, which is the second specific image is a dashed-line frame that corresponds to the animal type.
  • the processor 120 distinguishes whether the animal is the closest in type to the accepted animal type by drawing frame FR2 with a solid line and frame FR3 with a dashed line, but the distinction may also be made by changing the color, thickness, shape, etc. Also, the processor 120 may determine the color and thickness of frames FR2 and FR3 according to the temperature, as in the third embodiment.
  • step ST24 the processor 120 detects animals present in the infrared image from the temperature information acquired in step ST23.
  • the processor 120 detects animals present in the infrared image from the temperature information acquired in step ST23.
  • the processor 120 acquires at least one of shape and temperature information for each of the multiple detected animals based on the infrared image in step ST25. In addition, the processor 120 identifies the type of each of the multiple detected animals based on the information acquired in step ST25 in step ST26.
  • step ST27 the processor 120 generates a specific image that identifies the direction in which an animal is present, and corresponds to the type of animal.
  • a first specific image is generated that identifies the direction in which a first animal that is relatively closest in type to the type of animal specified in step ST21 among the multiple detected animals is present, and a second specific image that is different from the first specific image and identifies the direction in which a second animal that is different from the first animal among the multiple detected animals is present.
  • the first specific image and the second specific image each include an arrow that indicates the direction in which the animal is present.
  • step ST28 the processor 120 generates a display image by superimposing the first specific image and the second specific image on the visible light image, and in step ST29, displays the display image generated in step ST28 on the EVF 26 or the LCD 40.
  • FIG. 13 is a diagram showing an example of a live view image LV4 displayed on the EVF 26 or LCD 40 in step ST29 of the sixth embodiment.
  • the live view image LV4 is an image in which specific images, arrows AR2 and AR3, are superimposed on a visible light image (not shown).
  • the arrows AR2 and AR3 indicate the directions in which animals AN5 and AN6 are located, respectively, detected in the infrared image.
  • animal AN5 is the animal that is relatively closest in type to the animal type accepted in step ST21.
  • arrow AR2 an example of a "first arrow” pointing in the direction of animal AN5 is a solid arrow that corresponds to the type of animal.
  • animal AN6 is a different type of animal from animal AN5, and is relatively distant in type from the animal type accepted in step ST21.
  • arrow AR3 an example of a "second arrow" pointing in the direction of animal AN6 is a dashed arrow that corresponds to the type of animal.
  • the processor 120 distinguishes whether the animal is the closest in type to the accepted animal type by drawing the arrow AR2 as a solid line and the arrow AR3 as a dashed line, but the distinction may be made by changing the color, thickness, shape, etc. Also, the processor 120 may determine the color and thickness of the arrows AR2 and AR3 according to the temperature, as in the fourth embodiment.
  • the trend of a moving hot spot is displayed.
  • FIG. 14 is a flowchart showing the steps of a live view image display control method according to the seventh embodiment.
  • steps ST31 to ST34 is similar to the processing in steps ST1 to ST4 in the first embodiment.
  • step ST35 it is determined whether the high-temperature area acquired in step ST33 is moving. Here, if the amount of movement within the angle of view of the visible light image of the high-temperature area is equal to or greater than a threshold, it is determined that the high-temperature area is moving. If the processor 120 determines that the high-temperature area is not moving, it proceeds to processing in step ST36, and if it determines that the high-temperature area is moving, it proceeds to processing in step ST37.
  • step ST36 the processor 120 generates a display image by superimposing the specific image generated in step ST34 on the visible light image.
  • step ST37 the processor 120 generates a display image by superimposing the specific image generated in step ST34 and the specific image stored in the memory 130 on the visible light image.
  • the specific image stored in the memory 130 will be described later.
  • step ST38 the processor 120 causes the display image generated in step ST36 or step ST37 to be displayed on the EVF 26 or LCD 40.
  • step ST39 the processor 120 stores the specific image generated in step ST34 in the memory 130.
  • the specific image stored here is a past specific image showing a past high temperature area, and is stored as a specific image that can be distinguished from the current specific image.
  • the straight line frame generated in step ST34 is changed to a dashed line frame and stored.
  • step ST39 does not have to be performed for all frames of the live view image, but may be performed for every certain number of frames, for example.
  • step ST40 the processor 120 determines whether or not to end the display of the live view image.
  • the display of the live view image is ended, for example, when the shutter release button 22 is pressed all the way down to execute the shooting and recording process of a still image, or when the power to the visible light camera 100 is cut off.
  • step ST40 If it is determined in step ST40 that the display of the live view image is to be ended, the processor 120 ends the processing of this flowchart. On the other hand, if it is determined that the display of the live view image is not to be ended, the processor 120 proceeds to the processing of step ST31.
  • step ST35 After processing steps ST31 to ST34, if the processor 120 determines in step ST35 that the high temperature area is moving, it generates a display image in step ST37 by superimposing the specific image generated in step ST34 and the specific image stored in memory 130 in step ST39 on the visible light image.
  • FIG. 15 is a diagram showing an example of a live view image according to the seventh embodiment.
  • F15A in FIG. 15 is a diagram showing a live view image LV5 displayed on the EVF 26 or LCD 40 in step ST36.
  • the live view image LV5 is an image in which a frame FR4, which is a first specific image, and frames FR5-1, FR5-2, and FR5-3, which are second specific images, are superimposed on a visible light image.
  • the frame FR4 is a solid-line frame that surrounds an animal AN7 that corresponds to the high-temperature part of the current visible light image.
  • the frame FR5-1 is a dashed-line frame that is arranged at the position of the animal AN7 that corresponds to the high-temperature part of the first visible light image several frames before the current frame.
  • the frame FR5-2 is a dashed-line frame that is arranged at the position of the animal AN7 that corresponds to the high-temperature part of the second visible light image several frames before.
  • the frame FR5-3 is a dashed-line frame that is arranged at the position of the animal AN7 that corresponds to the high-temperature part of the third visible light image several frames before.
  • the processor 120 may generate a display image as a second specific image by superimposing a line showing the trajectory of the movement of the high temperature area on the visible light image.
  • F15B in FIG. 15 is a diagram showing a live view image LV6 displayed on the EVF 26 or LCD 40 in step ST28.
  • the live view image LV6 is an image in which a frame FR4, which is a first specific image, and arrows AR4-1, AR4-2, and AR4-3, which are second specific images, are superimposed on a visible light image (not shown).
  • the frame FR4 is a solid-line frame that surrounds the animal AN7 that corresponds to the high-temperature part in the current visible light image.
  • the arrow AR4-1 is a solid-line arrow that is arranged on a straight line connecting the center of the frame FR4 and the center of the position of the animal AN7 that corresponds to the high-temperature part in the first visible light image several frames earlier.
  • the arrow AR4-2 is a straight-line arrow that is arranged on a straight line that connects the center of the position of the animal AN7 in the first visible light image and the center of the animal AN7 in the second visible light image several frames earlier.
  • the arrow AR4-3 is a straight arrow that is located on a straight line that connects the center of the position of the animal AN7 in the second visible light image and the center of the animal AN7 in the third visible light image several frames earlier. Note that the dashed line shown in F15B is illustrated for explanatory purposes and is not displayed in the live view image LV6.
  • the processor 120 may predict the trend of the moving high temperature area and display a specific image showing the future high temperature area. For example, when the high temperature area is outside the angle of view of the visible light image, the processor 120 may predict and display the direction of the future position of the high temperature area.
  • FIG. 16 is a diagram for explaining a live view image LV7 displayed on the EVF 26 or LCD 40.
  • the live view image LV7 is an image in which an arrow AR5, which is a specific image, is superimposed on a visible light image (not shown).
  • Animal AN8 shown in Figure 16 is a high-temperature area that exists outside the angle of view of the visible light image, and is a high-temperature area detected from an infrared image (not shown).
  • Position PS1 shown in Figure 16 indicates the position of animal AN8 detected from a first infrared image several frames before the present (not shown), and position PS2 indicates the position of animal AN8 detected from a second infrared image several frames before that (not shown).
  • Position PS0 shown in Figure 16 is the future position to which animal AN8 is predicted to move several frames after the present. Note that the frame, dashed line, and arrow outside the range of live view image LV7 are shown for explanatory purposes, and in reality only animal AN8 exists.
  • Arrow AR5 is positioned on a line connecting the center of the angle of view of the visible light image (not shown) and the animal AN8 in the infrared image, and points in the direction of the animal AN8.
  • the user can know in which direction to point the optical axis of the visible light camera 100 in order to capture the high-temperature area.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
PCT/JP2024/009326 2023-03-29 2024-03-11 画像処理装置、画像処理方法及びプログラム、撮影システム Ceased WO2024203225A1 (ja)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2025510229A JPWO2024203225A1 (https=) 2023-03-29 2024-03-11

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023053925 2023-03-29
JP2023-053925 2023-03-29

Publications (1)

Publication Number Publication Date
WO2024203225A1 true WO2024203225A1 (ja) 2024-10-03

Family

ID=92904443

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/009326 Ceased WO2024203225A1 (ja) 2023-03-29 2024-03-11 画像処理装置、画像処理方法及びプログラム、撮影システム

Country Status (2)

Country Link
JP (1) JPWO2024203225A1 (https=)
WO (1) WO2024203225A1 (https=)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013013050A (ja) * 2011-05-27 2013-01-17 Ricoh Co Ltd 撮像装置及びこの撮像装置を用いた表示方法
JP2018036228A (ja) * 2016-09-02 2018-03-08 リコーインダストリアルソリューションズ株式会社 ガス画像センサ装置およびガス画像撮像測定装置およびガス画像撮像測定システム
WO2019111464A1 (ja) * 2017-12-04 2019-06-13 ソニー株式会社 画像処理装置及び画像処理方法
JP2020119096A (ja) * 2019-01-21 2020-08-06 キヤノン株式会社 画像処理装置、画像処理方法、および、プログラム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013013050A (ja) * 2011-05-27 2013-01-17 Ricoh Co Ltd 撮像装置及びこの撮像装置を用いた表示方法
JP2018036228A (ja) * 2016-09-02 2018-03-08 リコーインダストリアルソリューションズ株式会社 ガス画像センサ装置およびガス画像撮像測定装置およびガス画像撮像測定システム
WO2019111464A1 (ja) * 2017-12-04 2019-06-13 ソニー株式会社 画像処理装置及び画像処理方法
JP2020119096A (ja) * 2019-01-21 2020-08-06 キヤノン株式会社 画像処理装置、画像処理方法、および、プログラム

Also Published As

Publication number Publication date
JPWO2024203225A1 (https=) 2024-10-03

Similar Documents

Publication Publication Date Title
US7747158B2 (en) Photographing apparatus and focusing control method
US8885026B2 (en) Imaging device and imaging method
US9258545B2 (en) Stereoscopic imaging apparatus
JP2013013050A (ja) 撮像装置及びこの撮像装置を用いた表示方法
JP2012227839A (ja) 撮像装置
JP2008182485A (ja) 撮影装置及び撮影方法
US9838667B2 (en) Image pickup apparatus, image pickup method, and non-transitory computer-readable medium
CN102739958B (zh) 图像处理装置、图像处理方法
KR101455071B1 (ko) 디지털 합성을 이용한 야간 촬영 성능 개선방법
JP2007281873A (ja) 撮像装置
JP2010130314A (ja) 撮像装置
JP4842919B2 (ja) 表示装置、撮影装置、および表示方法
CN103227899B (zh) 摄像设备及其控制方法
WO2024203225A1 (ja) 画像処理装置、画像処理方法及びプログラム、撮影システム
JP4661269B2 (ja) 撮影装置及びプログラム
KR20160129168A (ko) 디지털 합성을 이용한 야간 촬영 성능 개선방법
WO2024203226A1 (ja) 表示制御装置、表示制御方法及びプログラム、撮影システム
JP6900577B2 (ja) 画像処理装置及びプログラム
JP2004128588A (ja) 撮像装置
JP2004350192A (ja) ディジタル・カメラおよびその制御方法
WO2024203227A1 (ja) 表示制御装置、表示制御方法、撮影システム及びプログラム
US12301991B2 (en) Outputting information in a case in which a focus evaluation value of the latest frame exceeds an allowable value set based on the maximum evaluation value during a manual focus period
JP5054209B2 (ja) 撮影装置及び合焦制御方法
JP2019101106A (ja) 撮像装置及びその制御方法
JP6293334B2 (ja) 撮像装置、撮像方法およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24779355

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025510229

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2025510229

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 24779355

Country of ref document: EP

Kind code of ref document: A1