JP2023065313A - Imaging apparatus, image processing device, and method - Google Patents

Abstract

To provide an imaging apparatus and a method for generating image data for displaying to assist a user to quickly gaze at an intended location or subject.SOLUTION: An imaging apparatus is capable of detecting a user's gazing location in an image being displayed. The imaging apparatus applies a processing process that visually emphasizes a feature area over other areas when generating image data to be displayed when the detection of the gazing location is valid. The feature area is an area of a subject of a type determined on the basis of the setting of the imaging apparatus.SELECTED DRAWING: Figure 6

Description

The present invention relates to an imaging device, an image processing device, and a method.

Japanese Unexamined Patent Application Publication No. 2002-101001 discloses an imaging device that detects a gaze position of a user in a display image and enlarges and displays an area including the gaze position.

JP 2004-215062 A

According to the technology described in Patent Document 1, it becomes easier for the user to confirm whether or not the user is gazing at the intended position in the display image. However, it is not possible to reduce the time required for the user's line of sight to match the intended position (subject) in the displayed image.

The present invention has been made in view of such problems of the prior art. In one aspect, the present invention provides an imaging apparatus and method for generating image data for display to assist a user in quickly gazing at an intended position or object.

The above-mentioned object is an imaging device, which has detection means capable of detecting a gaze position of a user in an image displayed by the imaging device, and generation means for generating image data for display, and generating The means applies processing to the image data generated when the detection means is active to visually emphasize the characteristic region from other regions, and the characteristic region is determined based on the settings of the imaging device. is a subject area of .

According to one aspect of the present invention, it is possible to provide an imaging apparatus and method for generating image data for display that assists a user in quickly gazing at an intended position or subject.

1 is a block diagram showing the configuration of an imaging device according to an embodiment of the present invention; FIG. FIG. 2 is a diagram showing a correspondence relationship between a pupil plane of pixels and a photoelectric conversion unit of an imaging device according to an embodiment of the present invention; The figure which shows the structure of the line-of-sight input part concerning embodiment of this invention. Flowchart of the first embodiment of the present invention Shooting mode setting method according to the embodiment of the present invention Image processing example 1 according to the first embodiment of the present invention Image processing example 2 of the first embodiment of the present invention Image pickup device configuration of the second embodiment of the present invention Flowchart of the second embodiment of the present invention Image processing example 3 of the second embodiment of the present invention Image processing example 4 of the second embodiment of the present invention FIG. 11 is a diagram showing an example of a calibration screen presented by an imaging device according to the third embodiment; FIG. A diagram showing an example of a scene where it is appropriate to perform processing according to visual characteristics and an example of processing. A diagram showing an example of a scene where it is appropriate to perform processing according to visual characteristics and an example of processing. A diagram showing an example of a scene where it is appropriate to perform processing according to visual characteristics and an example of processing. A diagram showing an example of a scene where it is appropriate to perform processing according to visual characteristics and an example of processing. Flowchart relating to display image data generation operation in the third embodiment A diagram showing an example of a virtual space presented in the fourth embodiment. A diagram showing an example of highlighting in the fourth embodiment A diagram showing an example of the relationship between the type of virtual space and the type of subject that can be highlighted in the fourth embodiment. A diagram showing an example of a main subject type selection GUI in the fourth embodiment. A diagram showing an example in which image data is used as metadata in the fourth embodiment. A diagram showing an example of indices displayed together with a virtual space image in the fourth embodiment. FIG. 11 is a diagram showing an example of a display system according to a fifth embodiment and its operation; FIG. 12 is a block diagram showing a functional configuration example of a computer device that can be used as a server in the fifth embodiment; FIG. A diagram showing an example of a display area in the fifth embodiment FIG. 12 is a diagram showing another example of the display system according to the fifth embodiment and its operation; FIG. 11 is a diagram showing a configuration example of a camera used in the fifth embodiment;

The invention will now be described in detail on the basis of its exemplary embodiments with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. In addition, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and the plurality of features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

In the following, the present invention will be described with respect to implementation in an imaging device such as a digital camera. However, the present invention can be implemented with any electronic device capable of detecting the gaze position of the display screen. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smart phones, game machines, robots, vehicle-mounted devices, and the like, in addition to imaging devices. These are examples, and the present invention can also be implemented in other electronic devices.

● (first embodiment)
[Description of configuration of imaging device]
FIG. 1 is a block diagram showing a functional configuration example of an imaging device 1 as an example of an image processing device according to an embodiment. The imaging device 1 has a main body 100 and a lens unit 150 . Although the lens unit 150 is an interchangeable lens unit detachable from the main body 100 here, it may be a lens unit integrated with the main body 100 .

Lens unit 150 and body 100 are mechanically and electrically connected via a lens mount. Communication terminals 6 and 10 provided on the lens mount are contacts for electrically connecting the lens unit 150 and the main body 100 . Communication between the lens unit control circuit 4 and the system control circuit 50 is possible through the communication terminals 6 and 10 . Electric power required for the operation of the lens unit 150 is also supplied from the body 100 to the lens unit 150 through the communication terminals 6 and 10 .

The lens unit 150 constitutes an imaging optical system that forms an optical image of a subject on the imaging surface of the imaging section 22 . A lens unit 150 has an aperture 102 and a plurality of lenses 103 including a focus lens. The diaphragm 102 is driven by the diaphragm driving circuit 2, and the focus lens is driven by the AF driving circuit 3, respectively. The operations of the aperture drive circuit 2 and the AF drive circuit 3 are controlled by the lens system control circuit 4 according to instructions from the system control circuit 50 .

A focal plane shutter 101 (hereinafter simply referred to as shutter 101 ) is driven under the control of system control circuit 50 . The system control circuit 50 controls the operation of the shutter 101 so as to expose the imaging section 22 according to the photographing conditions when photographing a still image.

The imaging unit 22 is an imaging device having a plurality of pixels arranged two-dimensionally. The imaging unit 22 converts an optical image formed on the imaging surface into a pixel signal group (analog image signal) by a photoelectric conversion unit of each pixel. The imaging unit 22 may be, for example, a CCD image sensor or a CMOS image sensor.

The imaging unit 22 of the present embodiment can generate a pair of image signals used for phase-difference automatic focus detection (hereinafter referred to as phase-difference AF). FIG. 2 shows the correspondence relationship between the pupil plane of the lens unit 150 and the photoelectric conversion units of the pixels of the imaging unit 22 . FIG. 2A shows an example of a configuration in which each pixel has a plurality of (here, two) photoelectric conversion units 201a and 201b, and FIG. there is

Each pixel is provided with one microlens 251 and one color filter 252 . The color of the color filter 252 differs for each pixel, and the colors are arranged in a predetermined pattern. Here, as an example, it is assumed that the color filters 252 are arranged in a primary color Bayer pattern. In this case, the color of the color filter 252 that each pixel has is red (R), green (G), or blue (B).

In the configuration of FIG. 2A, the light input from the region 253a of the pupil plane 253 to the pixel enters the photoelectric conversion unit 201a, and the light input from the region 253b enters the photoelectric conversion unit 201b. For a plurality of pixels, phase difference AF can be performed by using a signal group obtained from the photoelectric conversion unit 201a and a signal group obtained from the photoelectric conversion unit 201b as a pair of image signals.

When the signals obtained by the photoelectric conversion units 201a and 201b are individually handled, the individual signals function as focus detection signals. On the other hand, when the signals obtained by the photoelectric conversion units 201a and 201b of the same pixel are collectively (added), the countable signal functions as a pixel signal. Therefore, the pixel having the configuration of FIG. 2A functions both as a pixel for focus detection and as a pixel for photographing. It is assumed that all pixels of the imaging unit 22 have the configuration shown in FIG. 2(a).

On the other hand, FIG. 2B shows a configuration example of dedicated focus detection pixels. The pixel shown in FIG. 2B is provided with a light-shielding mask 254 between the color filter 252 and the photoelectric conversion section 201 to restrict light incident on the photoelectric conversion section 201 . Here, the light shielding mask 254 has openings such that only the light from the region 253 b of the pupil plane 253 is incident on the photoelectric conversion section 201 . As a result, the pixel becomes substantially the same as having only the photoelectric conversion unit 201b in FIG. 2(a). Similarly, by configuring the aperture of the light shielding mask 254 so that only the light from the region 253a of the pupil plane 253 is incident on the photoelectric conversion section 201, the pixel is converted to the photoelectric conversion section 201a of FIG. It can be substantially the same as the state having. Even if a plurality of pairs of these two types of pixels are arranged in the imaging unit 22, signal pairs for phase difference AF can be generated.

Instead of phase-difference AF, or in combination with phase-difference AF, contrast detection-type automatic focus detection (hereinafter referred to as contrast AF) may be performed. When only contrast AF is performed, the pixels can have a configuration in which the light shielding mask 254 is omitted from FIG. 2(b).

The A/D converter 23 converts the analog image signal output from the imaging section 22 into a digital image signal. If the imaging unit 22 can output a digital image signal, the A/D converter 23 can be omitted.

The image processing unit 24 applies predetermined image processing to the digital image signal from the A/D converter 23 or the memory control unit 15 to generate a signal or image data according to the application, or perform various processing. Obtain and/or generate information. The image processing unit 24 may be, for example, a dedicated hardware circuit such as an ASIC designed to implement a specific function, or a programmable processor such as a DSP executing software to perform the specific function. It may be a configuration to realize.

Here, the image processing applied by the image processing unit 24 includes preprocessing, color interpolation processing, correction processing, detection processing, data processing processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing includes signal amplification, reference level adjustment, defective pixel correction, and the like. Color interpolation processing is processing that interpolates values of color components that cannot be obtained at the time of shooting, and is also called demosaicing processing or synchronization processing. The correction processing includes white balance adjustment, gradation correction (gamma processing), processing for correcting the effects of optical aberration and vignetting of the lens 103, processing for color correction, and the like. The detection processing includes detection of feature regions (for example, face regions and human body regions) and their movements, recognition of persons, and the like. The data processing includes synthesis processing, scaling processing, encoding and decoding processing, header information generation processing, and the like. The evaluation value calculation processing includes generation of signals and evaluation values used for automatic focus detection (AF), calculation processing of evaluation values used for automatic exposure control (AE), and the like. The special effect processing includes addition of blur, change of color tone, relighting processing, and processing applied when gaze position detection, which will be described later, is enabled. Note that these are examples of image processing that can be applied by the image processing unit 24, and the image processing that is applied by the image processing unit 24 is not limited.

A specific example of the feature region detection processing will be described. The image processing unit 24 applies horizontal and vertical band-pass filters to image data to be detected (for example, live view image data) to extract edge components. After that, the image processing unit 24 applies matching processing using a template prepared in advance according to the type of characteristic region to be detected to the edge component, and detects an image region similar to the template. For example, when detecting a human face region as a feature region, the image processing unit 24 applies matching processing using templates of facial parts (eg, eyes, nose, mouth, and ears).

A group of candidate regions for the eyes, nose, mouth, and ears is detected by matching processing. The image processing unit 24 narrows down the group of eye candidates to those that satisfy other eye candidates and preset conditions (for example, the distance and inclination of two eyes). Then, the image processing unit 24 associates other parts (nose, mouth, ears) that satisfy the positional relationship with the group of narrowed-down eye candidates. Furthermore, the image processing unit 24 detects a face area by applying a preset non-face condition filter and excluding a combination of parts that do not correspond to a face. The image processing unit 24 outputs the total number of detected face areas and information on each face area (position, size, detection reliability, etc.) to the system control circuit 50 . The system control circuit 50 stores the information of the feature area obtained from the image processing section 24 in the system memory 52 .

Note that the method of detecting the human face region described here is an example, and any other known method such as a method using machine learning can be used. In addition, detection is not limited to human faces, and other types of feature regions such as human bodies, limbs, animal faces, landmarks, characters, automobiles, airplanes, and railroad vehicles may be detected.

The detected characteristic area can be used, for example, for setting the focus detection area. For example, a main face area can be determined from the detected face areas, and a focus detection area can be set in the main face area. As a result, AF can be performed so as to focus on the face area existing within the shooting range. Note that the main face area may be selected by the user.

Output data from the A/D converter 23 is stored in the memory 32 via the image processing section 24 and the memory control section 15, or via the memory control section 15 alone. The memory 32 is used as a buffer memory for still image data and moving image data, a working memory for the image processing section 24, a video memory for the display section 28, and the like.

The D/A converter 19 converts image data for display stored in the video memory area of the memory 32 into analog signals and supplies the analog signals to the display unit 28 . The display unit 28 displays on a display device such as a liquid crystal display according to the analog signal from the D/A converter 19 .

The display unit 28 can function as an electronic viewfinder (EVF) by continuously generating and displaying display image data while shooting a moving image. An image displayed so that the display unit 28 functions as an EVF is called a through image or a live view image. The display unit 28 may be arranged inside the main body 100 so as to be observed through the eyepiece, or may be arranged on the housing surface (for example, the rear surface) of the main body 100, or may be provided on both sides. good too.

In this embodiment, the display unit 28 is arranged at least inside the main body 100 in order to detect the gaze position of the user.

The nonvolatile memory 56 is an electrically rewritable EEPROM, for example. The nonvolatile memory 56 stores programs executable by the system control circuit 50, various setting values, GUI data, and the like.

The system control circuit 50 has one or more processors (also called CPU, MPU, etc.) capable of executing programs. The system control circuit 50 realizes the functions of the imaging apparatus 1 by loading the program recorded in the nonvolatile memory 56 into the system memory 52 and executing it by the processor.

The system memory 52 is used to store programs executed by the system control circuit 50 and constants and variables used during execution of the programs.
The system timer 53 measures the time used for various controls and the time of the built-in clock.

A power switch 72 is an operation member for switching ON/OFF of the power of the imaging apparatus 1 .
A mode changeover switch 60 , a first shutter switch 62 , a second shutter switch 64 , and an operation section 70 are operation members for inputting instructions to the system control circuit 50 .

A mode switch 60 switches the operation mode of the system control circuit 50 between a still image recording mode, a moving image shooting mode, a reproduction mode, and the like. Modes included in the still image recording mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), and a shutter speed priority mode (Tv mode). In addition, there are various scene modes, program AE modes, custom modes, etc., which are shooting settings for each shooting scene. A mode selector switch 60 allows direct switching to any of these modes contained in the menu button. Alternatively, after switching to the menu button once with the mode switching switch 60, any of these modes included in the menu button may be switched using another operation member. Similarly, the movie shooting mode may also include multiple modes.

The first shutter switch 62 is turned on by half-pressing the shutter button 61 and generates a first shutter switch signal SW1. The system control circuit 50 recognizes the first shutter switch signal SW1 as a still image shooting preparation instruction, and starts shooting preparation operations. The shooting preparation operation includes, for example, AF processing, automatic exposure control (AE) processing, auto white balance (AWB) processing, EF (flash pre-emission) processing, etc., but these are not essential, and other processing may be included. may be included.

The second shutter switch 64 is turned on when the shutter button 61 is fully pressed, and generates a second shutter switch signal SW2. The system control circuit 50 recognizes the second shutter switch signal SW2 as an instruction to shoot a still image, and executes shooting processing and recording processing.

The operation unit 70 is a general term for operation members other than the shutter button 61 , the mode switching switch 60 and the power switch 72 . The operation unit 70 includes, for example, direction keys, a set (execution) button, a menu button, a video shooting button, and the like. If the display unit 28 is a touch display, the operation unit 70 also includes software keys realized by display and touch operation. When the menu button is operated, system control circuit 50 causes display unit 28 to display a menu screen that can be operated using the direction keys and the set button. The user can change the settings of the imaging device 1 by operating software keys and menu screens.

FIG. 3A is a side view schematically showing a configuration example of the line-of-sight input unit 701. FIG. The line-of-sight input unit 701 acquires an image (image for line-of-sight detection) for detecting the rotation angle of the optical axis of the eyeball 501a of the user looking into the display unit 28 provided inside the main body 100 through the eyepiece. It is a unit that

The image for sight line detection is processed by the image processing unit 24 to detect the rotation angle of the optical axis of the eyeball 501a. Since the rotation angle represents the direction of the line of sight, the gaze position on the display unit 28 can be estimated based on the rotation angle and the preset distance from the eyeball 501a to the display unit 28 . In estimating the gaze position, the unique information of the user acquired by the calibration operation performed in advance may be taken into consideration. The gaze position may be estimated by the image processing unit 24 or by the system control circuit 50 . The line-of-sight input unit 701 and the image processing unit 24 (or the system control circuit 50) constitute detection means capable of detecting the gaze position of the user in the image displayed on the display unit 28 by the imaging device 1. FIG.

The image displayed on the display unit 28 is visually recognized by the user through the eyepiece lens 701d and the dichroic mirror 701c. The illumination light source 701e emits infrared light to the outside of the housing through the eyepiece. The infrared light reflected by the eyeball 501a enters the dichroic mirror 701c. The dichroic mirror 701c reflects the incident infrared light upward. A light receiving lens 701b and an imaging device 701a are arranged above the dichroic mirror 701c. The imaging element 701a captures an image of infrared light formed by the light receiving lens 701b. The imaging device 701a may be a monochrome imaging device.

The imaging element 701 a outputs an analog image signal obtained by photographing to the A/D converter 23 . The A/D converter 23 outputs the obtained digital image signal to the image processing section 24 . The image processing unit 24 detects an eyeball image from the image data, and further detects a pupil region within the eyeball image. The image processing unit 24 calculates the eyeball rotation angle (line-of-sight direction) from the position of the pupil region in the eyeball image. A known method can be used to detect the line-of-sight direction from an image containing an eyeball image.

FIG. 3B is a side view schematically showing a configuration example of the line-of-sight input section 701 when the display section 28 is provided on the back surface of the imaging device 1. As shown in FIG. Also in this case, the infrared light is emitted in the direction in which the face 500 of the user observing the display unit 28 is likely to exist. Then, an infrared image of the user's face 500 is captured by a camera 701f provided on the back surface of the imaging device 1, and a pupil region is detected from the image of the eyeballs 501a and/or 501b. to detect

As long as the gaze position on the display unit 28 can be finally detected, the configuration of the line-of-sight input unit 701 and the processing of the image processing unit 24 (or the system control circuit 50) are not particularly limited, and any other configuration can be used. and processing can be employed.

Returning to FIG. 1, the power supply control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching blocks to be energized, and the like. If the power supply unit 30 is a battery, the power supply control unit 80 detects the presence/absence of attachment, the type, and the remaining amount. Also, the power supply control unit 80 controls the DC-DC converter based on these detection results and instructions from the system control circuit 50, and supplies necessary voltage to each unit including the recording medium 200 for a necessary period.

The power supply unit 30 may use one or more of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries and Li batteries, and/or AC adapters.

A recording medium I/F 18 is an interface with a recording medium 200 such as a memory card or hard disk. The recording medium 200 may or may not be detachable. A recording medium 200 is a recording destination of image data obtained by photographing.

The communication unit 54 transmits and receives image signals and audio signals to and from an external device connected wirelessly or by wire. The communication unit 54 supports one or more communication standards such as wireless LAN (Local Area Network) and USB (Universal Serial Bus). The system control circuit 50 can transmit image data (including through images) captured by the imaging unit 22 and image data recorded on the recording medium 200 to an external device through the communication unit 54 . The system control circuit 50 can also receive image data and other various information from an external device through the communication unit 54 .

The orientation detection unit 55 detects the orientation of the imaging device 1 with respect to the direction of gravity. Based on the posture detected by the posture detection unit 55, it is possible to determine whether the imaging device 1 was oriented horizontally or vertically at the time of photographing. The system control circuit 50 can add the attitude of the imaging device 1 at the time of photographing to the image data file, or can record the image after aligning the orientation of the image. An acceleration sensor, a gyro sensor, or the like can be used as the posture detection unit 55 .

[Gaze position detection operation]
FIG. 4 is a flowchart relating to gaze position detection operation of the imaging device 1 . The gaze position detection operation is executed when the line-of-sight detection function is enabled. Further, the gaze position detection operation can be performed in parallel with the live view display operation.

In S2, the system control circuit 50 acquires the currently set shooting mode. The shooting mode can be set with the mode changeover switch 60 . When the scene selection mode is set by the mode changeover switch 60, the type of scene set in the scene selection mode is also treated as the shooting mode.

FIG. 5 is a diagram showing an example of the appearance of the imaging device 1. As shown in FIG. FIG. 5A shows an arrangement example of the mode changeover switch 60. FIG. FIG. 5B is a top view of the mode changeover switch 60 and shows examples of selectable shooting modes. For example, Tv indicates shutter speed priority mode, Av indicates F value priority mode, M indicates manual setting mode, P indicates program mode, and SCN indicates scene selection mode. A desired photographing mode can be set by rotating the mode switch 60 so that the character indicating the desired photographing mode is positioned at the position of the mark 63 . FIG. 5(b) shows a state in which the scene selection mode is set.

Scene selection mode is a shooting mode for shooting a specific scene or a specific subject. Therefore, in the scene selection mode, it is necessary to set the type of scene and subject. The system control circuit 50 sets shooting conditions (shutter speed, aperture value, sensitivity, etc.) and AF mode suitable for the set scene and subject type.

In this embodiment, the type of scene and subject in the scene selection mode can be set by operating the menu screen displayed on the display unit 28, as shown in FIG. 5(c). Here, one of portrait, landscape, kids, and sports can be set as an example, but more options may exist. As described above, in the scene selection mode, the set scene and subject type are treated as the shooting mode.

In S3, the system control circuit 50 acquires image data for display. The system control circuit 50 reads the image data for live view display to be displayed from now stored in the video memory area of the memory 32 and supplies the data to the image processing section 24 .

In S4, the image processing unit 24 as generating means applies processing to the display image data supplied from the system control circuit 50 to generate display image data for gaze position detection. Then, the image processing unit 24 stores the generated display image data in the video memory area of the memory 32 again. In this case, the image data for display acquired from the video memory area in S3 is processed. You may generate the image data for a display for.

Here, some examples of processing applied by the image processing unit 24 for gaze position detection will be described. The processing applied for gaze position detection is processing that visually emphasizes a characteristic region determined based on setting information (here, as an example, a shooting mode) of the imaging device 1 over other regions. This processing makes it easier for the user to quickly match the line of sight with the desired subject. What kind of feature information should be detected in correspondence with the setting information, parameters necessary for detecting the feature information, etc. can be stored in advance in, for example, the non-volatile memory 56 for each setting information. For example, in association with a shooting mode for shooting a specific scene, pre-storing the type of the main subject according to the specific scene and the template and parameters for detecting the characteristic region of the main subject. can be done.

(Example 1)
FIG. 6 schematically shows an example of processing that can be applied when the scene is set to "sports" in the scene selection mode. FIG. 6(a) shows an image represented by the display image data before processing, and FIGS. 6(b) to 6(d) respectively show images represented by the display image data after processing.

If "sports" is set in the scene selection mode, it can be inferred that the user intends to shoot a sports scene. In this case, the image processing unit 24 determines that the area of the moving human subject is a characteristic area to be emphasized, and applies processing to emphasize the characteristic area.

Specifically, the image processing unit 24 detects the human body region as the characteristic region, and compares the detection result with the previous detection result (for example, the detection result in the live view image of the previous frame) to determine whether the moving person is moving. Identify areas. Then, the image processing unit 24 applies processing for emphasizing the moving person region to the live view image of the current frame.

Here, it is assumed that moving person areas P1, P2, and P3 are detected in the image of the current frame shown in FIG. 6(a). FIG. 6B shows an example in which processing for superimposing frames A1 to A3 surrounding the person regions is applied as the processing for emphasizing the person regions P1 to P3.

FIG. 6(c) shows an example in which, as processing for emphasizing the person areas P1 to P3, the display of the area surrounding the person areas P1 to P3 is not changed, and the process of lowering the brightness of the other area A4 is applied. showing. Also, in FIG. 6D, as the processing for emphasizing the person areas P1 to P3, the display of the rectangular area A5 surrounding all of the person areas P1 to P3 is not changed, and the process of lowering the luminance of other areas is applied. example.

In this way, by detecting a characteristic region according to the set scene and the type of the main subject, and applying processing that emphasizes the detected characteristic region, the user can easily find the intended main subject. can be expected to By making it easier for the user to find the main subject intended by the user, an effect of shortening the time required for the user's line of sight to gaze at the main subject can be expected.

Note that the processing for emphasizing the characteristic region is not limited to the example described above. For example, processing may be performed to emphasize the edges of the human body regions P1, P2, and P3 detected as characteristic regions. Also, the frames A1 to A3 can be blinked or displayed in a specific color. Further, instead of lowering the brightness in FIGS. 6(c) and 6(d), monochrome display may be used. Moreover, when the feature region is a human or animal region, the human or animal region may be emphasized by converting the entire image into a thermography-like pseudo-color image.

(Example 2)
FIGS. 7A and 7B schematically show examples of processing that can be applied when the main subject is set to "kids" in the scene selection mode. FIG. 7A shows an image represented by the display image data before processing, and FIG. 7B shows an image represented by the display image data after processing.

If "Kids" is set in the scene selection mode, it can be inferred that the user intends to shoot a child as the main subject. In this case, the image processing unit 24 determines that the area of the human subject presumed to be a child is a characteristic area to be emphasized, and applies processing to emphasize the characteristic area.

Whether a human region detected as a feature region is an adult or a child can be determined as a child if, for example, the ratio of head length to body length or height is less than a threshold, or machine learning can be used. However, it is not limited to these. Only persons registered as children in advance may be detected by face authentication.

Here, it is assumed that person areas P1, K1, and K2 are detected in the image of the current frame shown in FIG. 7A, and areas K1 and K2 are determined to be children. In FIG. 7B, as processing for emphasizing the child areas K1 and K2, the edges of the child areas K1 and K2 are emphasized, and the gradation of areas other than the child areas K1 and K2 is reduced. An example of applying the treatment is shown. Reduction of gradation may be reduction of maximum luminance (compression of luminance), reduction of number of luminance gradations (from 256 gradations to 16 gradations), etc., but is not limited to these. Reduction in luminance or monochrome display as shown in Example 1 may be applied.

(Example 3)
FIGS. 7A and 7C schematically show examples of processing that can be applied when the main subject is set to "text" in the scene selection mode. FIG. 7A shows an image represented by the display image data before processing, and FIG. 7C shows an image represented by the display image data after processing.

When "character" is set in the scene selection mode, it can be inferred that the user intends to take a picture while paying attention to the characters existing in the scene. In this case, the image processing unit 24 determines that an area estimated to be a character as a characteristic area is a characteristic area to be emphasized, and applies processing to emphasize the characteristic area.

Here, it is assumed that the character area MO is detected in the image of the current frame shown in FIG. 7(a). FIG. 7(c) shows an example in which processing for emphasizing the edges of the character area MO and reducing the gradation of areas other than the character area MO is applied as processing for emphasizing the character area MO. . The gradation reduction method may be the same as in Example 2.

As described in Examples 2 and 3, different processing can be applied to the same original image (FIG. 7A) depending on the set shooting mode. In addition, in Examples 2 and 3, processing similar to that in Example 1 may be applied. Further, only one of the edge enhancement for the region to be enhanced and the luminance or gradation reduction for the other region may be applied.

Processing that can be applied to image data for gaze position detection in the present embodiment is to visually emphasize a region to be emphasized (characteristic region) determined based on setting information of an imaging device from other regions. It is a processing process that Processing may be, for example, any one of the following four types.
(1) A process that does not process the area that should be emphasized and processes other areas (by reducing the brightness or gradation) so that it does not stand out (2) Emphasizes the area that should be emphasized (emphasis of edges, etc.) ) and do not process other areas. (3) Emphasize the area to be emphasized (edge enhancement, etc.), and process other areas to make them less noticeable (such as by reducing brightness and gradation). (4) processing the entire image to emphasize the area to be emphasized (conversion to a pseudo-color image, etc.)
It should be noted that these are examples, and arbitrary processing can be applied such that the area to be emphasized is visually emphasized (conspicuous) with respect to other areas.

Returning to FIG. 4, in S5, the system control circuit 50 causes the display unit 28 to display the display image data generated by the image processing unit 24 in S4. The system control circuit 50 also acquires from the image processing unit 24 the rotation angle of the optical axis of the eyeball detected by the image processing unit 24 based on the sight line detection image from the sight line input unit 701 . Based on the acquired rotation angle, the system control circuit 50 obtains the coordinates (gaze position) within the image displayed on the display unit 28 where the user is gazing. Note that the system control circuit 50 may notify or feed back the gaze position to the user by superimposing a mark indicating the obtained gaze position on the live view image.

Thus, the gaze position detection operation ends. The gaze position obtained by the gaze position detection operation can be used for setting the focus detection area, selecting the main subject, etc., but is not limited to these. When recording image data obtained by photographing, information on the gaze position detected at the time of photographing may be recorded in association with the image data. For example, it is possible to record the gaze position information at the time of photographing as accompanying information recorded in the header of a data file that stores image data. The gaze position information recorded in association with the image data can be used to identify the main subject in an application program or the like that handles the image data.

Note that when the line-of-sight input function is not set to be valid, the image processing unit 24 does not apply processing for supporting line-of-sight input to image data for display, but may apply processing for different purposes. .

As described above, in the present embodiment, in the image displayed when line-of-sight input is valid, processing is performed in which the characteristic region determined based on the setting information of the imaging device is visually emphasized over the other regions. applied. As a result, it becomes easier for the user to visually recognize the area that is likely to be intended as the main subject, and the effect of shortening the time required to gaze at the main subject can be expected.

Note that in the present embodiment, the region to be emphasized is determined based on the shooting mode settings. may

● (Second embodiment)
Next, a second embodiment will be described. The second embodiment is an embodiment in which XR goggles (head-mounted display device or HMD) are used as the display unit 28 in the first embodiment. Note that XR is a general term for VR (virtual reality), AR (augmented reality), and MR (mixed reality).

The left view of FIG. 8A is a perspective view showing an example of the appearance of the XR goggles 800. FIG. The XR goggles 800 are generally worn on the face area SO shown in the right diagram of FIG. 8(a). FIG. 8(b) is a diagram schematically showing the mounting surface of the XR goggles 800 (the surface in contact with the face). FIG. 8C is a top view schematically showing the positional relationship between the eyepiece 701d of the XR goggles 800, the display units 28A and 28B, and the user's right eye 501a and left eye 501b when the XR goggles 800 are worn. .

The XR goggles 800 have a display unit 28A for the right eye 501a and a display unit 28B for the left eye 501b, and display the right eye image and the left eye image on the display unit 28B. This enables stereoscopic viewing. Therefore, the eyepiece 701d described in the first embodiment is provided for each of the display sections 28A and 28B.

In this embodiment, it is assumed that the imaging unit 22 has pixels configured as shown in FIG. 2(a). In this case, a right-eye image can be generated from the pixel signal group obtained from the photoelectric conversion unit 201a, and a left-eye image can be generated from the pixel signal group obtained from the photoelectric conversion unit 201b. The right-eye image and the left-eye image may be generated using another configuration, such as using a lens capable of capturing stereo images as the lens unit 150 . Also, the line-of-sight input unit 701 is provided in the eyepiece of the XR goggles, and generates a line-of-sight detection image for either the right eye or the left eye.

Since the rest of the configuration can be implemented with the same configuration as that of the imaging device 1 shown in FIG. Note that in the present embodiment, the display image is generated using the right-eye image and the left-eye image recorded in advance on the recording medium 200 instead of the live view image.

FIG. 9 is a flowchart relating to gaze position detection operation in this embodiment, and redundant description is omitted by assigning the same reference numerals as in FIG. 4 to steps that perform the same processing as in the first embodiment.

In S91, the system control circuit 50 acquires the currently set experience mode. Since no shooting is performed in this embodiment, an experience mode related to XR is acquired. The experience mode is, for example, the type of virtual environment in which the XR experience is performed, and options such as "art museum", "museum", "zoo", and "diving" are prepared. Note that the experience mode can be set by using a remote controller, using an input device provided in the XR goggles, or by displaying a menu screen and making a selection with the line of sight. It is assumed that the recording medium 200 stores display image data corresponding to each virtual environment that can be selected as the experience mode.

In S3, the system control circuit 50 acquires the display image data corresponding to the experience mode selected in S91 by reading it from the recording medium 200, and supplies it to the image processing section .

In S92, the image processing unit 24 applies processing to the display image data supplied from the system control circuit 50 to generate display image data for gaze position detection. In the present embodiment, the display image data is stereo image data including a right-eye image and a left-eye image, so the image processing unit 24 applies processing to both the right-eye image and the left-eye image.

In the processing applied by the image processing unit 24 in S92, the characteristic region determined based on the setting information (here, the experience mode as an example) of the device that provides the XR experience (here, the imaging device 1) is different from other regions. It is a processing that is visually emphasized. This processing can be expected to increase the immersive feeling of the XR experience.

An example of processing applied by the image processing unit 24 in S92 will be described.
(Example 4)
FIG. 10 schematically shows an example of processing that can be applied when the experience mode is set to "diving". FIG. 10A shows an image represented by display image data before processing.

When "diving" is set in the experience mode, it can be inferred that the user is interested in underwater creatures. In this case, the image processing unit 24 determines that the area of moving underwater creatures is a characteristic area to be emphasized, and applies processing to emphasize the characteristic area.

Specifically, the image processing unit 24 detects areas of fish, sea animals, and the like as characteristic areas, and identifies moving characteristic areas by comparing them with past detection results. Then, the image processing unit 24 applies a processing process for emphasizing a moving characteristic region to the frame image to be processed.

Here, it is assumed that characteristic regions f1 to f4, which are regions of moving fish and humans, are detected in the frame image to be processed shown in FIG. 10(a). In this case, the image processing unit 24 maintains the display of the characteristic regions f1 to f4 and performs processing to reduce the number of colors in other regions (for example, to monochrome) as the processing to emphasize the characteristic regions f1 to f4. Apply. The processing for emphasizing the characteristic region may be other processing including the one described in the first embodiment.

Returning to FIG. 9, in S5, the system control circuit 50 performs image processing on the rotation angle of the optical axis of the eyeball detected by the image processing unit 24 based on the sight line detection image from the sight line input unit 701. Acquired from the unit 24 . Based on the acquired rotation angle, the system control circuit 50 obtains the coordinates (gaze position) within the image displayed on the display unit 28A or 28B that the user is gazing at. Then, the system control circuit 50 superimposes marks indicating gaze positions on the right-eye and left-eye image data generated by the image processing unit 24 in S92, and displays them on the display units 28A and 28B.

In S93, the system control circuit 50 determines whether or not to apply further processing to the display image using the gaze position information detected in S5. This determination can be performed based on arbitrary determination conditions, for example, based on user settings regarding the use of gaze position information.

If it is determined that the gaze position information is not used, the system control circuit 50 ends the gaze position detection operation. On the other hand, the system control circuit 50 executes S94 if it is determined that the line-of-sight position information is to be used.

In S<b>94 , the system control circuit 50 reads the display image data stored in the video memory area of the memory 32 and supplies it to the image processing section 24 . The image processing unit 24 further applies processing to the display image data using the gaze position detected in S5.

FIGS. 10(b) and 10(c) show an example of processing using gaze position information performed in S94. A marker P1 indicating the gaze position detected in S5 is superimposed on the display image data. Here, since the detected gaze position p1 is within the characteristic region f1, there is a high possibility that the user is interested in the characteristic region f1. Therefore, among the characteristic regions f1 to f4 emphasized by applying the processing processing in S92, the processing is applied such that the characteristic region f1 is visually emphasized more than the other characteristic regions f2 to f4.

For example, in S92, it is assumed that the characteristic regions f1 to f4 are emphasized by maintaining the color display of the characteristic regions f1 to f4 and displaying the other regions in monochrome. In this case, in S94, the image processing unit 24 also displays the characteristic areas f2 to f4 in monochrome, and displays the characteristic area f1 or the area including the gaze position and the characteristic area closest to the gaze position (here, the characteristic area f1) in color. to maintain FIG. 10(b) schematically shows a state in which color display is maintained in an area C1 including the gaze position p1 and the characteristic area f1, and processing is performed to display the other areas including the characteristic areas f2 to f4 in monochrome. clearly shown. Here, the characteristic areas f2 to f4 are changed to the same display form as the areas other than the characteristic amount area, but the characteristic areas f2 to f4 are less conspicuous than the characteristic area f1 and more conspicuous than the areas other than the characteristic amount area. It is good also as a display form.

In this way, by narrowing down the feature areas to be emphasized using the gaze position, the feature area that the user is interested in can be grasped more accurately than when the gaze position is not used, and processing for enhancing can be applied. be able to. Therefore, the effect of increasing the immersive feeling during the XR experience can be expected. In addition, it is possible to realize an effect that the user can easily confirm that the user is gazing at the intended subject in an application that uses the gaze position.

It is also possible to use the temporal change in the detected gaze position. In FIG. 10C, it is assumed that the gaze position detected at time T=0 is P1, and the gaze position detected at T=1 (arbitrary unit) is P2. In this case, since the gaze position moves from P1 to P2 between times T=0 and 1, it can be seen that the user moves the line of sight to the left.

In this case, by emphasizing the characteristic region f4 existing in the movement direction of the gaze position, an effect that the user can easily gaze at a new characteristic region can be expected. Here, an example is shown in which C2 that maintains color display is expanded to include the characteristic region f4 that has the shortest distance in the movement direction of the gaze position. Although an example in which the region to be emphasized is expanded in the moving direction of the gaze position is shown here, the region may be moved according to the movement of the gaze position without expanding the region.

In this way, by determining the area to be emphasized in consideration of the temporal change of the line-of-sight position, it is possible to emphasize the subject that the user is likely to gaze at in the future, and to facilitate the user to gaze at the desired subject. You can expect the effect that can be supported.

Another example of processing applied by the image processing unit 24 in S92 will be described.
(Example 5)
FIG. 11 schematically shows an example of processing that can be applied when the experience mode is set to "museum". FIG. 11A shows an image represented by display image data before processing.

If "museum" is set in the experience mode, it can be inferred that the user is interested in works of art such as paintings and sculptures. In this case, the image processing unit 24 determines that the area of the artwork is a characteristic area to be emphasized, and applies processing to emphasize the characteristic area.

Here, it is assumed that feature areas B1 to B5 are detected as areas of works of art in the frame image to be processed shown in FIG. 11(a). In this case, in S92, the image processing unit 24 maintains the display of the characteristic regions B1 to B5 and maintains the display of the characteristic regions B1 to B5, and increases the brightness of the other regions as processing processing for emphasizing the characteristic regions B1 to B5, as shown in FIG. 11B, for example. apply processing that reduces The processing for emphasizing the characteristic region may be other processing including the one described in the first embodiment.

When applying further processing to the display image using the gaze position information, in S94 the image processing unit 24, for example, as shown in FIG. Accompanying information CM1 stored in advance can be superimposed on the area B2. The accompanying information CM1 is not particularly limited, and in the case of a painting, it may be information corresponding to the type of characteristic region, such as bibliographic information such as the name of the painting, the author, and the year of production. Note that, in this embodiment, since the display image data is prepared in advance, information regarding the position of the art object in the image and accompanying information regarding the art object can also be prepared in advance. Therefore, the image processing unit 24 can identify the artwork present at the gaze position and acquire its associated information.

In this example, the accompanying information of the artwork present at the gaze position is additionally displayed for further emphasis. good too.

As described above, in the present embodiment, in addition to the processing described in the first embodiment, by applying the processing processing that takes into account the gaze position, the features that the user is likely to be interested in are Areas can be emphasized more effectively. Therefore, it is possible to assist the user in quickly gazing at a desired subject and provide a more immersive XR experience.

● (Third Embodiment)
Next, a third embodiment will be described. The line-of-sight input function is a function that uses the user's visual sense, and there are individual differences in the visual characteristics of the user. Therefore, in the present embodiment, the user-friendliness of the line-of-sight input function is improved by applying a processing process to the display image data in consideration of the user's visual characteristics.

Examples of individual differences in visual characteristics include:
(1) Individual differences in luminance range (dynamic range) where differences in brightness can be discerned (2) Individual differences in central vision (1 to 2 degrees around the point of gaze) and effective visual field (4 to 20 degrees around central vision) (3) There are individual differences in the ability to recognize hue differences. These individual differences can be congenital or acquired (typically due to aging).

Therefore, in the present embodiment, by registering visual information reflecting the individual differences (1) to (3) for each user and applying processing reflecting the visual information to the image data for display, individual user's To provide an easy-to-use line-of-sight input function for

A specific example of a calibration function for acquiring visual information will be described below. The calibration function can be executed by the system control circuit 50, for example, when the execution is instructed by the user through a menu screen or when the user's visual characteristics are not registered.

The luminance dynamic range of (1) can be the range between the maximum luminance and the minimum luminance that does not make the user feel uncomfortable. For example, the system control circuit 50 causes the display unit 28 to display an achromatic gradation chart that expresses the maximum luminance to the minimum luminance with a predetermined number of gradations, as shown in FIG. 12(a). Then, the user is allowed to select a brightness range in which the user does not feel uncomfortable, for example, by operating the operation unit 70 . The user can adjust the positions of the upper and lower ends of the bar 1201 using, for example, the up and down keys of the 4-way key, and can identify the difference between the maximum brightness that does not feel dazzling and the adjacent gradation (or does not feel too dark). A minimum brightness can be set.

The system control circuit 50 registers luminance ranges KH and KL that are preferably not used for the user, based on the positions of the upper and lower ends of the bar 1201 when the set (determine) button is pressed, for example. It should be noted that luminance corresponding to the positions of the upper end and the lower end of the bar 1201 may be registered.

Alternatively, the system control circuit 50 increases the brightness of the entire screen when the up key of the 4-way key is pressed, and decreases the brightness of the entire screen when the down key is pressed. and minimum luminance. Then, the system control circuit 50 prompts the user to press the set button in a state in which the display is made with the maximum brightness that does not cause glare. Then, the system control circuit 50 registers the display brightness at the time of detecting pressing of the set button as the maximum brightness. In addition, the system control circuit 50 prompts the user to press the set button in a state where the display is at the minimum luminance at which the difference from the adjacent gradation can be discerned (or at which the user does not feel that it is too dark). Then, the system control circuit 50 registers the display brightness at the time of detecting pressing of the set button as the minimum brightness. Also in this case, instead of the maximum luminance and the minimum luminance, a luminance range KH on the high luminance side and a luminance range KL on the low luminance side that are preferably not used by the user may be registered.

The user's visual characteristics regarding luminance dynamic range can be used to determine whether or not luminance adjustment is necessary, and to determine parameters for luminance adjustment.

(2) The effective visual field is a range in which information can be identified, including central vision. A useful field of view may be, for example, a field of view called a useful field of view (UFOV). The system control circuit 50 causes the display unit 28 to display an image in which a circle 1202 whose size is variable is displayed against a background of a relatively fine pattern as shown in FIG. 12(b), for example. Then, the user is urged to adjust the size of the circle 1202 to a range in which the background pattern can be clearly discerned while gazing at the center of the circle 1202 . The user adjusts the size of the circle 1202 using, for example, the up and down keys of the four direction keys so as to correspond to the maximum range in which the background pattern can be clearly recognized, and presses the set key to change the size of the effective field of view. can be set. The system control circuit 50 changes the size of the circle 1202 upon detection of pressing of the up/down key, and registers the range of effective visual field according to the size of the circle 1202 at that time upon detection of pressing of the set key.

The user's visual characteristics regarding the effective field of view can be used to extract the gaze range.

(3) is the magnitude of the hue difference at which a difference can be recognized between similar colors. The system control circuit 50 causes the display unit 28 to display an image in which a plurality of color samples of similar colors with gradually changing hues are arranged in a selectable manner, for example, as shown in FIG. 12(c). The color samples displayed here can be colors such as green, yellow, and blue, which may occupy a large area in the background of the subject. Also, information may be obtained for a plurality of color systems such as green, yellow, and blue.

In FIG. 12C, the color samples are arranged using the image of colored pencils, but strip-shaped color samples may be used. Colored pencils on the left end are the reference colors, and color samples with different hues are arranged in the right direction. The system control circuit 50 prompts the user to select the leftmost colored pencil among the colored pencils that can be recognized as having a different color from the leftmost colored pencil. The user selects the corresponding colored pencil using, for example, the left and right keys of the 4-way key, and presses the set key. When the system control circuit 50 detects pressing of the left/right key, it moves the colored pencil in the selected state, and when it detects pressing of the set key, the difference between the hue corresponding to the colored pencil currently selected and the hue of the reference color is detected. is registered as the minimum hue difference recognizable by the user. When registering information for a plurality of color systems, the same operation is repeatedly executed for each color system.

The user's visual characteristics related to hue difference recognition ability can be used to determine whether hue adjustment is necessary or not, and to determine parameters for hue adjustment.

The above-described visual characteristics (1) to (3) with individual differences and the method of acquiring the unique information of the user regarding the visual characteristics (1) to (3) are merely examples. The user's information can be registered with respect to other visual characteristics and/or the information with respect to visual aptitudes (1)-(3) can be registered in other ways.

Next, a specific example of processing using the registered user's visual characteristics (1) to (3) will be described. Note that when visual characteristics can be registered for a plurality of users, the visual characteristics related to the user selected through the setting screen, for example, are used.

FIG. 13A shows a scene in which a plurality of airplanes E1 are present against a background of a backlit sky with high brightness. If the background has such high luminance, the background may be dazzling depending on the user's visual characteristics, making it difficult to gaze at the airplane E1.

In order to deal with such a situation, the image processing unit 24, when generating image data for display when the line-of-sight input function is enabled, sets the luminance value of the background (for example, the average luminance value) to the user's visual characteristic (luminance dynamic range) can be determined. The image processing unit 24 determines that the luminance is not appropriate for the user's visual characteristics when the luminance value of the background is out of the luminance dynamic range of the user (when it is included in the luminance range KH in FIG. 12). . Then, the image processing unit 24 performs processing to lower the brightness so that the brightness value of the background area of the image is within the user's brightness dynamic range (the brightness range represented by the bar 1201 in FIG. 12). apply to

FIG. 13(b) schematically shows a state in which the processing for reducing the brightness of the background area is applied. M1 is the main subject area. A region of the image excluding the main subject region M1 is assumed to be a background region. Here, the image processing unit 24 separates the main subject area M1 from the background area by defining an area having a size that includes a characteristic area (here, an airplane) existing within a certain range from the gaze position of the user. Note that the size of the main subject area may be the size of the user's effective field of view. Also, determination of the main subject area based on the gaze position of the user may be based on other methods.

It should be noted that in the case of applying the processing for adjusting the luminance to suit the luminance dynamic range of the user, the target luminance value can be appropriately determined within the luminance dynamic range. For example, it may be the median value of the luminance dynamic range. Although only the processing for adjusting (correcting) the luminance value of the background area has been described here, the luminance value of the main subject area can be similarly adjusted. When adjusting the brightness of both the background area and the main subject area, the visibility of the main subject area can be improved by setting the target brightness of the main subject area higher than that of the background area.

FIG. 14(a) shows a scene in which a large number of similar subjects move in various directions, such as a group sport or a game, as an example of a scene in which the main subject is easily lost. In FIG. 14A, it is assumed that the main subject intended by the user is E2.
If the user loses sight of the main subject E2 and the main subject moves out of the user's effective field of view, the main subject will be recognized as blurred like the other subjects, making it even more difficult to distinguish between them.

In order to cope with such a situation, the image processing unit 24 applies a process of lowering (blurring) the resolution of the area (background area) other than the main subject area M2, as shown in FIG. 14(b). As a result, since the sharpness of the main subject area M2 is relatively increased, even if the user loses sight of the main subject E2, he/she can easily find it. The main subject area M2 can be determined in the same manner as described for brightness adjustment.

If the size of the main subject area is larger than the range of the central field of view, processing may be applied to the area outside the range of the central field of view in the main subject area as a background area. By relatively increasing the sharpness of the main subject in this way, the user's attention is naturally directed to the main subject, and as a result, it is possible to achieve the effect of supporting subject tracking based on the gaze position.

FIG. 15A shows a scene in which the main subject is an animal moving in a dark place, as an example of a scene in which it is difficult for the user to recognize the main subject due to the low brightness of the main subject. In FIG. 15A, it is assumed that the main subject intended by the user is E3.

In order to deal with such a situation, when generating image data for display when the line-of-sight input function is enabled, the image processing unit 24 sets the luminance value (for example, average luminance value) of the peripheral region of the gaze position to the user's visual perception. It can be determined whether or not the characteristics (luminance dynamic range) are appropriate. When the luminance value of the peripheral region of the gaze position is out of the user's luminance dynamic range (when it is included in the luminance range KL in FIG. 12), the image processing unit 24 determines that the luminance is appropriate for the user's visual characteristics. determine that it is not. Then, the image processing unit 24 performs processing to increase the luminance so that the luminance value of the peripheral region of the gaze position is within the luminance dynamic range of the user (the luminance range represented by the bar 1201 in FIG. 12). Apply to data. FIG. 15(b) schematically shows a state in which processing for increasing the brightness of the peripheral region M3 of the gaze position is applied. The area around the gaze position may be, for example, an area corresponding to the effective visual field, a characteristic area including the gaze position, or an area used as a template for tracking.

Here, the reason why the brightness is adjusted (increased) only for the area around the gaze position, not for the entire image, is that if the brightness of the image in a dark scene is increased, the visibility of the image is reduced due to noise components. When the brightness of the entire screen is increased, the accuracy of detecting a moving object between frames tends to decrease due to the influence of noise. In addition, when noise is visible on the entire screen, the user's eyes are likely to get tired due to the flickering of the noise.

It should be noted that when the scene is dark, it is quite conceivable that the main subject does not exist at the gaze position. Therefore, until the gaze position stabilizes, the brightness of the entire screen is increased, and once the gaze position stabilizes, the brightness of the area other than the peripheral area of the gaze position is restored (no processing is applied). good too. The system control circuit 50 can determine that the gaze position has stabilized, for example, if the amount of movement of the gaze position is equal to or less than a threshold for a certain period of time.

FIG. 16A shows a scene in which a similar-colored bird E4 is moving against a background of grass, as an example of a scene in which the main subject and the background are similar in color and the main subject is easy to lose sight of. In FIG. 16A, it is assumed that the main subject intended by the user is the bird E4. When the user loses sight of the bird E4, it is difficult to find the bird E4 because the color of the background and the bird E4 are similar.

Therefore, the image processing unit 24 determines that the difference between the hue of the main subject area (the area of the bird E4) and at least the hue of the background area around the main subject is determined in light of the user's ability to recognize the hue difference among the visual characteristics. It can be determined whether it is appropriate or not. If the difference in hue between the main subject area and the background area is less than or equal to the difference in hue recognizable by the user, the image processing unit 24 determines that the image is inappropriate. In this case, the image processing unit 24 displays processing for changing the hue of the main subject area so that the difference in hue between the main subject area and the surrounding background area is greater than the difference in hue that the user can perceive. Applies to image data. FIG. 16(b) schematically shows a state in which processing for changing the hue of the main subject area M4 is applied.

It should be noted that it is possible to apply not only the processing processing exemplified here, but also processing processing using the user's visual characteristics. Further, it is also possible to combine and apply a plurality of processing processes according to the luminance and hue of the main subject area and the background area.

FIG. 17 is a flow chart relating to the display image data generation operation according to the present embodiment. This operation can be executed in parallel with the gaze position detection when the line-of-sight input function is enabled.
In S<b>1701 , the system control circuit 50 captures an image of one frame with the imaging unit 22 and supplies a digital image signal to the image processing unit 24 through the A/D converter 23 .

In S1702, the image processing unit 24 detects a feature area to be the main subject area based on the most recently detected gaze position. Here, the image processing unit 24 detects the characteristic region of the type determined from the shooting mode as described in the first embodiment, and then detects the characteristic region including the gaze position or the feature closest to the gaze position. The area may be the main subject area.

In S1703, the image processing unit extracts the feature area (main subject area) detected in S1702. As a result, the main subject area and the other area (background area) are separated.

In S1704, the image processing unit 24 acquires information about the user's visual characteristics stored in the nonvolatile memory 56, for example.

In S1705, the image processing unit 24 calculates the difference in average luminance and hue between the main subject area and the background area. Then, the image processing unit 24 compares the calculated average luminance and hue difference with the user's visual characteristics to determine whether or not processing needs to be applied to the main subject area. As described above, the image processing unit 24 needs to apply processing to the main subject area when the luminance of the main subject and the difference in hue between the main subject area and the background area are not appropriate for the visual characteristics of the user. Determine that there is. The image processing unit 24 executes S1706 if it is determined that it is necessary to apply processing to the main subject area, and executes S1707 if it is not determined.

In S1706, the image processing unit 24 applies processing to the main subject area according to the content determined as inappropriate, and then executes S1707.

In S1707, the image processing unit 24 determines whether or not it is necessary to apply processing to another area (background area) in the same manner as in S1705. The image processing unit 24 executes S1708 if it is determined that it is necessary to apply processing to the background area. Also, if it is not determined that it is necessary to apply processing to the background area, the image processing device unit 24 executes S1701 and starts the operation for the next frame.

In S1708, the image processing unit 24 executes S1701 after applying processing to the background area according to the content determined to be inappropriate.

Depending on what is appropriate for the user's visual characteristics, it is possible to determine in advance what kind of processing is to be applied to each of the main subject area and the background area. Therefore, depending on the determination result in S1705, whether to apply the processing to only the main subject area, only to the background area, or to apply the processing to both the main subject area and the background area. The content of the processing to be applied is specified.

As described above, according to the present embodiment, when the line-of-sight input function is effective, the display image data is generated by applying the processing that takes into consideration the user's visual characteristics. Therefore, display image data suitable for the visual characteristics of individual users can be generated, and a user-friendly line-of-sight input function can be provided.

Note that the processing for making it easier to select the main subject based on the line of sight described in the first embodiment and the processing for making an image suitable for the user's visual characteristics described in the present embodiment are: They can also be applied in combination.

● (Fourth Embodiment)
Next, a fourth embodiment will be described. The present embodiment relates to improving the visibility of a virtual space experienced using XR goggles (head-mounted display device or HMD) incorporating the components of the imaging device 1 . An image of the virtual space viewed through the XR goggles is generated by drawing display image data prepared in advance for each virtual space according to the orientation and posture of the XR goggles. The display image data may be pre-stored in the recording medium 200, or may be obtained from an external device.

Here, as an example, it is assumed that the display data for providing the experience modes “diving” and “art museum” in the virtual space is stored in the recording medium 200 . However, there are no particular restrictions on the types and number of virtual spaces to be provided.

Examples of virtual space images for providing the experience modes "diving" and "museum" are schematically shown in FIGS. 18(a) and (b). Here, in order to facilitate explanation and understanding, it is assumed that the entire virtual space is represented by a CG image. Therefore, the main subject to be highlighted is part of the CG image. By highlighting the main subject included in the virtual space image, the visibility of the main subject can be improved. The main subject is set by the imaging apparatus 1 (system control circuit 50) at least in the initial state. The main subject set by the imaging device 1 may be changed by the user.

For example, when displaying a composite image in which CG is superimposed as a virtual space image on an image of the real space, such as a video see-through HMD, the main subject region (feature region) to be highlighted is included in the actual image portion. In some cases, it is included in the CG part.

FIGS. 19(a) and 19(b) are diagrams schematically showing an example of applying processing for emphasizing the main subject to the scenes shown in FIGS. 18(a) and 18(b), respectively. Here, the main subject is emphasized by reducing the chroma saturation of the subjects other than the main subject, and the visibility of the main subject is improved. It should be noted that the main subject may be emphasized by another method for processing.

The example shown in FIG. 19 is processing for not processing the area of the main subject and making the other areas inconspicuous. In addition, processing may be processing that emphasizes the main subject and does not process other areas, or processing that emphasizes the main subject and makes other areas inconspicuous. Alternatively, the processing may be processing that emphasizes the object by processing the entire image, or processing that emphasizes the area of the main subject by another method.

When experiencing diving in a virtual space, the main subject is considered to be a "creature." When experiencing an art museum in a virtual space, the main subjects are considered to be "exhibits" (paintings, sculptures, etc.) and objects with characteristic colors (here, rich colors). In other words, the main subject to be highlighted may differ depending on the type of virtual space and experience to be presented.

FIG. 20 is a diagram showing the relationship between the type of virtual space (or experience) to be provided and the type of subject (type of characteristic region) that can be highlighted. Here, the type of subject that can be highlighted is associated with the type of virtual space as metadata. In addition, the type of main subject highlighted by default is also associated with the type of virtual space. Here, the types of subjects listed as metadata correspond to the types of subjects detectable by the image processing unit 24 . For each type of virtual space, the type of subject that can be set as the main subject is indicated by ◯, and the type of subject that is selected as the main subject by default is indicated by ⊚. Therefore, the user can select a new main subject from the subjects indicated by ◯.

There are no particular restrictions on how the user can change the type of main subject. For example, the system control circuit 50 displays a GUI for changing the main subject on the display section 28 of the imaging device 1 or the display section of the XR goggles in response to the operation of the menu screen through the operation section 70 . The system control circuit 50 can change the setting of the main subject for the type of virtual space currently provided according to the operation of the GUI through the operation unit 70 .

FIG. 21 is a diagram showing an example of a GUI displayed for changing the main subject. FIG. 21A shows a GUI that imitates a mode dial. By operating the dial included in the operation unit 70, one of the options can be set as the main subject. FIG. 21A shows a state in which a landscape is set as the main subject. The options displayed on the GUI for changing the main subject correspond to the types of metadata circled in FIG. In the example of FIG. 21(a), options include "OFF" for setting not to perform highlighting, in addition to the types of metadata. FIG. 21(b) shows another example of the GUI for changing the type of main subject. The GUI is the same as the GUI shown in FIG. By selecting a desired option using the operation unit 70, the user can change the main subject to be highlighted (and turn off the highlighting). It should be noted that it may be possible to select an option using the line of sight.

FIG. 22 is an image showing an example of metadata for the virtual space types "diving", "museum", and "safari". For example, the subject area detected by the image processing unit 24 in the virtual space image to be presented can be extracted as metadata for each type of subject and stored in the memory 32 . This makes it possible to easily deal with a change in the main subject to be highlighted. Note that if the image of the virtual space to be displayed on the XR goggles can be generated in advance, metadata can also be recorded in advance. Also, the metadata may be numerical information representing the subject area (for example, center position and size, outer edge coordinate data, etc.).

Alternatively, the type of subject that the user is interested in may be specified using the gaze position information described in the second embodiment, and the specified type of subject area may be highlighted. In this case, since the type of the main subject to be highlighted changes according to the gaze position, the user can change the main subject without explicitly changing the settings.

In addition, when the main subject does not exist in the current field of view, or when the number or size of the main subject area is less than a threshold value, an index indicating the direction in which more of the main subject enters the field of view is superimposed on the virtual space image. good too.

FIG. 23(a) shows an example of a virtual space image currently being presented to the XR goggles in the experience mode "diving". The virtual space image being presented does not include a fish area, which is the main subject. In this case, the system control circuit 50 can superimpose the index P1 in the direction in which the main subject exists on the virtual space image. The system control circuit 50 can specify the direction in which the fish enters the field of view of the XR goggles, for example, based on the position information of the fish object in the virtual space data for generating display image data.

The user can visually recognize the fish as shown in FIG. 23(b) by shaking the head so as to look in the direction indicated by the index P1. Note that a plurality of indices indicating the direction in which the main subject exists may be superimposed. In this case, the system control circuit 50 selects the index indicating the direction in which the line of sight movement required to include the main subject in the field of view is the shortest, or the index indicating the direction in which the maximum number of main subjects can be included in the field of view. It can be displayed prominently (eg, large).

According to this embodiment, the type of subject area corresponding to the provided virtual space is highlighted. As a result, it becomes easier for the user to visually recognize a region of the virtual space image that is likely to be the main subject, and the effect of shortening the time required to gaze at the main subject can be expected.

● (Fifth embodiment)
Next, a fifth embodiment will be described. This embodiment relates to a display system that acquires a virtual space image to be displayed on the XR goggles in the fourth embodiment from an external device of the XR goggles, such as a server.

FIG. 24(a) is a schematic diagram of a display system in which the XR goggles DP1 and the server SV1 are communicably connected. A network such as a LAN or the Internet may exist between the XR goggles DP1 and the server SV1.

In general, generation of a virtual space image requires large-capacity virtual space data and computing power to generate (render) a virtual space image from the virtual space data. Therefore, the XR goggles output information necessary for generating a virtual space image, such as posture information detected by the posture detection unit 55, to the server. Then, the server generates a virtual space image to be displayed on the XR goggles and transmits it to the XR goggles.

By having the virtual space data (three-dimensional data) in the server SV1, it becomes possible for a plurality of XR goggles connected to the server to share the same virtual space.

FIG. 25 is a block diagram showing a configuration example of a computer device that can be used as the server SV1. In the figure, a display 2501 displays information on data being processed by an application program, various message menus, etc., and is composed of an LCD (Liquid Crystal Display) or the like. A CRTC 2502 as a video RAM (VRAM) display controller controls screen display on the display 2501 . A keyboard 2503 and a pointing device 2504 are used for inputting characters and operating icons and buttons in a GUI (Graphical User Interface). A CPU 2505 controls the entire computer apparatus.

A ROM (Read Only Memory) 2506 stores programs executed by the CPU 2505, parameters, and the like. A RAM (Random Access Memory) 2507 is used as a work area when the CPU 2505 executes various programs, a buffer for various data, and the like.

A hard disk drive (HDD) 2508 and removable media drive (RMD) 2509 function as external storage devices. A removable media drive is a device that reads, writes, or reads a removable recording medium, and may be an optical disk drive, a magneto-optical disk drive, a memory card reader, or the like.

Note that the programs that implement various functions of the server SV1, the OS, application programs such as browsers, data, libraries, etc., are stored in one or more of the ROM 2506, HDD 2508, and RMD 2509 (recording media thereof) according to their use. It is

The expansion slot 2510 is, for example, a slot for mounting an expansion card conforming to the PCI (Peripheral Component Interconnect) bus standard. Various expansion boards such as a video capture board and a sound board can be installed in the expansion slot 2510 .

A network interface 2511 is an interface for connecting the server SV1 to a local network or an external network. In addition to the network interface 2511, the server device SV1 has one or more communication interfaces with external devices conforming to the standard. Examples of standards include USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), wireless LAN, Bluetooth (registered trademark), and the like.

A bus 2512 consists of an address bus, a data bus and a control bus, and connects the blocks described above.

Next, the operation of the server SV1 and the XR goggles DP1 will be described using the flowchart shown in FIG. 24(b). The operation of the server SV1 is implemented by the CPU 2501 executing a predetermined application.

In S2402, the type of virtual space (FIG. 20) is specified from the XR goggles DP1 to the server SV1. The system control circuit 50 displays, for example, a GUI for designating the type of virtual space on the display section 28 of the XR goggles DP1. When the system control circuit 50 detects a selection operation through the operation unit 70, the system control circuit 50 transmits data indicating the selected type to the server SV1 through the communication unit 54. FIG.

Here, it is assumed that the range of the virtual space displayed on the XR goggles DP1 is fixed. Therefore, the server SV1 transmits image data (virtual space image data) of a specified type of specific scene in the virtual space to the XR goggles DP1 together with accompanying metadata.

In S2403, the system control circuit 50 receives the virtual space image data and accompanying metadata from the server SV1.

In S2404, the system control circuit 50 stores the virtual space image data and metadata received from the server SV1 in the memory 32. FIG.

In S2405, the system control circuit 50 uses the image processing unit 24 to apply the main subject region enhancement processing described with reference to FIG. 19 to the virtual space image data. Then, the virtual space image data subjected to the enhancement processing is displayed on the display unit 28 . Note that when the virtual space image is composed of an image for the right eye and an image for the left eye, enhancement processing is applied to each image.

FIG. 24(c) is a flowchart regarding the operation of the server SV1 when the server SV1 generates virtual space image data according to the posture (line-of-sight direction) of the XR goggles DP1 and applies enhancement processing to the virtual space data. . The operation of the server SV1 is implemented by the CPU 2501 executing a predetermined application.

In S2411, the server SV1 receives data designating the type of virtual space from the XR goggles DP1.

The operations after S2412 are executed for each frame of the moving image displayed on the XR goggles DP1.
In S2412, the server SV1 receives posture information from the XR goggles DP1.
In S2413, the server SV1 generates virtual space image data corresponding to the posture of the XR goggles DP1. The virtual space data can be generated by any known method such as rendering of three-dimensional data, clipping from an omnidirectional image, or the like. For example, as shown in FIG. 26, the server SV1 can determine the display area of the XR goggles DP1 from the virtual space image based on the orientation information of the XR goggles DP1, and cut out the range corresponding to the display area. Note that the XR goggles DP1 may transmit information specifying the display area (for example, center coordinates) instead of the posture information.

In S2415, the server SV1 receives the type of main subject from the XR goggles DP1. Note that reception of the main subject type in S2415 is executed when the main subject type is changed in the XR goggles DP1, and is skipped when there is no change.

In S2416, the server SV1 applies main subject area enhancement processing to the virtual space image data generated in S2413. If there is no change in the type of main subject, the server SV1 applies enhancement processing to the default main subject area corresponding to the type of virtual space.

In S2417, the server SV1 transmits the virtual space image data to which the enhancement processing is applied to the XR goggles DP1. The XR goggles DP1 causes the display unit 28 to display the received virtual space image data.

FIG. 27(a) is a schematic diagram of a display system in which a camera CA capable of generating a VR image is added to the configuration of FIG. 26(a). Here, as a type of virtual space, it is assumed that the case of experience sharing mentioned in the example of FIG. 20 is used. By displaying an image recorded with XR information added by the camera CA on the XR goggles, the wearer of the XR goggles DP1 can also simulate the experience of the user of the camera CA.

FIG. 28 is a block diagram showing a configuration example of the camera CA. The camera CA has a main body 100' and a lens unit 300 attached to the main body 100'. The lens unit 300 and the main body 100 ′ can be attached and detached by lens mounts 304 and 305 . Also, the lens system control circuit 303 of the lens unit 300 and the system control circuit 50 (not shown) of the main body 100' can communicate with each other through the communication terminals 6 and 10 provided on the lens mounts 304 and 305. can.

The lens unit 300 is a stereo fisheye lens, and the camera CA can capture a stereo circular fisheye image with a viewing angle of 180°. Specifically, each of the two optical systems 301L and 301R of the lens unit 300 has a horizontal angle (horizontal angle, azimuth angle, yaw angle) of 180 degrees and a vertical direction (vertical angle, elevation/depression angle, pitch angle) of 180 degrees. A circular fisheye image is generated by projecting the field of view onto a circular two-dimensional plane.

The main body 100' has a configuration similar to that of the main body 100 of the imaging apparatus 1 shown in FIG. 1, although only a part of the configuration is shown. An image (for example, a moving image conforming to the VR180 standard) captured by the camera CA having such a configuration is recorded in the recording medium 200 as an XR image.

The operation of the display system shown in FIG. 27(a) will be described using the flowchart shown in FIG. 27(b). It is assumed that the server SV1 is in a state of being able to communicate with the XR goggles DP1 and the camera CA.

In S2602, image data is transmitted from the camera CA to the server SV1. Image data is accompanied by additional information including Exif information such as shooting date and shooting conditions, photographer's line-of-sight information recorded at the time of shooting, main subject information detected at the time of shooting, and the like. Note that instead of communicating between the camera CA and the server SV1, the recording medium 200 of the camera CA may be attached to the server SV1 to read image data.

In S2603, the server SV1 generates image data and metadata to be displayed on the XR goggles DP1 from the image data received from the camera CA. In this embodiment, since the camera CA records a stereo circular fisheye image, image data for display is generated by cutting out the display range by a known method and converting it into a rectangular image. The server SV1 also detects a predetermined type of subject area from the display image data, and generates information about the detected subject area as metadata. The server SV1 transmits the generated display image data and metadata to the XR goggles DP1. The server SV1 also transmits additional information at the time of photographing, such as main subject information and line-of-sight information acquired from the camera CA, to the XR goggles DP1.

The operations performed by the system control circuit 50 of the XR goggles DP1 in S2604 and S2605 are the same as the operations in S2404 and S2405, so description thereof will be omitted. The system control circuit 50 can determine the type of main subject to which enhancement processing is applied in S2605 based on the main subject information received from the server SV1. Further, the system control circuit 50 may apply enhancement processing to the main subject area specified based on the line-of-sight information of the photographer. In this case, the subject that the photographer was gazing at at the time of shooting is highlighted, so that the experience of the photographer can be shared even more.

FIG. 27(c) is a flowchart relating to the operation of the server SV1 in the display system shown in FIG. 27(a) when the server SV1 executes the highlighting process in the same manner as in FIG. 24(c).

Since S2612 is the same as S2602, description thereof is omitted.
Also, S2613 to S2617 are the same as S2412, S2413, and S2415 to S2417, respectively, so description thereof will be omitted. Note that the type of the main subject to which the highlight display is applied is the designated type if specified by the XR goggles DP1, and is determined based on the main subject information at the time of shooting if not specified.

According to this embodiment, appropriate enhancement processing can be applied to virtual space images and VR images as well. In addition, by executing unmanageably large processing on an external device such as a server, the resources required for the XR goggles can be reduced, and multiple users can easily share the same virtual space.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Disclosure of this embodiment includes the following imaging device, method, image processing device, image processing method, and program.
(Item 1)
An imaging device,
a detection means capable of detecting a gaze position of a user in an image displayed by the imaging device;
and generating means for generating image data for the display,
The generation means applies processing to the image data generated when the detection means is active to visually emphasize a characteristic region over other regions,
wherein the characteristic region is a region of a subject of a type determined based on settings of the imaging device;
An imaging device characterized by:
(Item 2)
the settings are settings for shooting a specific scene or a specific subject;
The imaging apparatus according to item 1, wherein the characteristic area is an area of a subject of a type corresponding to the specific scene or an area of the specific subject.
(Item 3)
The processing treatment is
A process of processing other areas inconspicuously without processing the characteristic area,
processing that emphasizes the characteristic region and does not process other regions;
A process of emphasizing the characteristic region and processing other regions inconspicuously,
A process of processing the entire image including the characteristic region to emphasize the characteristic region;
3. The imaging device according to item 1 or 2, characterized by being any one of
(Item 4)
4. The imaging apparatus according to any one of items 1 to 3, wherein the generating means generates the image data as image data for live view display.
(Item 5)
5. The imaging apparatus according to any one of items 1 to 4, further comprising setting means for setting a focus detection area based on the gaze position detected by the detection means.
(Item 6)
A method executed by an imaging device having detection means capable of detecting a gaze position of a user in a displayed image,
a generating step of generating image data for the display;
In the generating step,
Applying processing to the image data generated when the detection means is active to visually emphasize the characteristic region from other regions,
The image data generated when the detection means is not effective is not processed to visually emphasize the characteristic region from other regions,
wherein the characteristic region is a region of a subject of a type determined based on settings of the imaging device;
A method characterized by:
(Item 7)
A program for causing a computer of an imaging device to function as each unit of the imaging device according to any one of items 1 to 5.
(Item 8)
having generating means for generating image data to be displayed on a head-mounted display device;
The generation means generates the image data by applying a processing process that visually emphasizes a characteristic region according to the type of virtual environment provided to the user through the display device from other regions.
An image processing apparatus characterized by:
(Item 9)
The processing treatment is
A process of processing other areas inconspicuously without processing the characteristic area,
processing that emphasizes the characteristic region and does not process other regions;
A process of emphasizing the characteristic region and processing other regions inconspicuously,
A process of processing the entire image including the characteristic region to emphasize the characteristic region;
9. The image processing apparatus according to item 8, characterized by being any one of
(Item 10)
Furthermore, it has a detection means capable of detecting a user's gaze position in the image displayed by the display device,
10. According to item 8 or 9, wherein the generating means generates the image data by applying further processing based on the gaze position detected by the detecting means after applying the processing. image processing device.
(Item 11)
11. The image processing apparatus according to item 10, wherein the further processing is processing for visually emphasizing a characteristic region including the gaze position among the characteristic regions more than other characteristic regions.
(Item 12)
11. The image processing apparatus according to item 10, wherein the further processing is a processing of superimposing accompanying information related to a characteristic region including the gaze position among the characteristic regions.
(Item 13)
11. The image processing apparatus according to item 10, wherein the further processing is processing for visually emphasizing a characteristic region existing in the moving direction of the gaze position.
(Item 14)
9. The method according to item 8, wherein for each type of the virtual environment, a type of characteristic region to which the processing is applicable and a type of the characteristic region to which the processing is applied by default are associated with each other. Image processing device.
(Item 15)
15. The image processing apparatus according to item 14, wherein the generating means applies the processing based on the type specified by the user from among the types of characteristic regions associated with the virtual environment being provided to the user. .
(Item 16)
The generating means, if not specified by the user, applies the processing based on the type of characteristic region to which the processing is applied by default, which is associated with the virtual environment being provided to the user. 16. The image processing device according to item 14 or 15.
(Item 17)
Furthermore, it has a detection means capable of detecting a user's gaze position in the image displayed by the display device,
17. The image processing apparatus according to any one of items 14 to 16, wherein the generation means applies the processing to the feature area based on the gaze position detected by the detection means.
(Item 18)
18. Any one of items 14 to 17, wherein when the generated image data does not include the characteristic region, the generating means includes an index indicating a direction in which the characteristic region exists in the image data. 10. The image processing device according to claim 1.
(Item 19)
19. The image processing device according to any one of items 14 to 18, wherein the head-mounted display device is an external device capable of communicating with the image processing device.
(Item 20)
19. The image processing device according to any one of items 14 to 18, wherein the image processing device is part of the head-mounted display device.
(Item 21)
further comprising acquisition means for acquiring VR image data representing the virtual environment;
the generating means generates the image data from the VR image;
21. The image processing apparatus according to any one of items 14 to 20, characterized by:
(Item 22)
the acquisition means further acquires main subject information and/or line-of-sight information obtained when the VR image is captured;
22. An image processing apparatus according to item 21, wherein the generation means determines the feature area to which the processing is applied based on the main subject information or the line-of-sight information.
(Item 23)
An image processing method executed by an image processing device,
a generation step of generating image data for display on a head-mounted display device;
In the generating step, the image data is generated by applying a processing process that visually emphasizes a characteristic region according to the type of virtual environment provided to the user through the display device from other regions.
An image processing method characterized by:
(Item 24)
A program for causing a computer to function as each unit included in the image processing apparatus according to any one of items 8 to 22.

The present invention is not limited to the content of the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

DESCRIPTION OF SYMBOLS 1... Imaging device 22... Imaging part 24... Image processing part 28... Display part 50... System control circuit 70... Operation part 100... Main body 150... Lens unit

Claims (24)

An imaging device,
a detection means capable of detecting a gaze position of a user in an image displayed by the imaging device;
and generating means for generating image data for the display,
The generation means applies processing to the image data generated when the detection means is active to visually emphasize a characteristic region over other regions,
wherein the characteristic region is a region of a subject of a type determined based on settings of the imaging device;
An imaging device characterized by: the settings are settings for shooting a specific scene or a specific subject;
2. The imaging apparatus according to claim 1, wherein the characteristic area is an area of a subject of a type corresponding to the specific scene or an area of the specific subject. The processing treatment is
A process of processing other areas inconspicuously without processing the characteristic area,
processing that emphasizes the characteristic region and does not process other regions;
A process of emphasizing the characteristic region and processing other regions inconspicuously,
A process of processing the entire image including the characteristic region to emphasize the characteristic region;
2. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is any one of: 2. The imaging apparatus according to claim 1, wherein said generating means generates said image data as image data for live view display. 2. The imaging apparatus according to claim 1, further comprising setting means for setting a focus detection area based on the gaze position detected by said detection means. A method executed by an imaging device having detection means capable of detecting a gaze position of a user in a displayed image,
a generating step of generating image data for the display;
In the generating step,
Applying processing to the image data generated when the detection means is active to visually emphasize the characteristic region from other regions,
The image data generated when the detection means is not effective is not processed to visually emphasize the characteristic region from other regions,
wherein the characteristic region is a region of a subject of a type determined based on settings of the imaging device;
A method characterized by: A program for causing a computer of an imaging device to function as each means of the imaging device according to claim 1 . having generating means for generating image data to be displayed on a head-mounted display device;
The generation means generates the image data by applying a processing process that visually emphasizes a characteristic region according to the type of virtual environment provided to the user through the display device from other regions.
An image processing apparatus characterized by: The processing treatment is
A process of processing other areas inconspicuously without processing the characteristic area,
processing that emphasizes the characteristic region and does not process other regions;
A process of emphasizing the characteristic region and processing other regions inconspicuously,
A process of processing the entire image including the characteristic region to emphasize the characteristic region;
9. The image processing apparatus according to claim 8, wherein the image processing apparatus is any one of: Furthermore, it has a detection means capable of detecting a user's gaze position in the image displayed by the display device,
9. The image data according to claim 8, wherein said generating means generates said image data by applying further processing based on said gaze position detected by said detecting means after applying said processing. Image processing device. 11. The image processing apparatus according to claim 10, wherein the further processing is processing for visually emphasizing a characteristic region including the gaze position among the characteristic regions more than other characteristic regions. . 11. The image processing apparatus according to claim 10, wherein said further processing is a processing of superimposing accompanying information relating to a feature area including said gaze position among said feature areas. 11. The image processing apparatus according to claim 10, wherein the further processing is processing for visually emphasizing a characteristic region existing in the moving direction of the gaze position. 9. The method according to claim 8, wherein for each type of said virtual environment, a type of characteristic region to which said processing is applicable and a type of characteristic region to which said processing is applied by default are associated with each other. image processing device. 15. The image processing according to claim 14, wherein said generating means applies said processing based on a type specified by said user from among types of feature regions associated with the virtual environment being provided to the user. Device. The generating means, if not specified by the user, applies the processing based on the type of characteristic region to which the processing is applied by default, which is associated with the virtual environment being provided to the user. 15. The image processing apparatus according to claim 14. Furthermore, it has a detection means capable of detecting a user's gaze position in the image displayed by the display device,
15. The image processing apparatus according to claim 14, wherein said generation means applies said processing to the characteristic region based on said gaze position detected by said detection means. 15. The image processing according to claim 14, wherein, when the generated image data does not include the characteristic region, the generation means includes an index indicating a direction in which the characteristic region exists in the image data. Device. 15. The image processing apparatus according to claim 14, wherein the head-mounted display device is an external device capable of communicating with the image processing apparatus. 15. The image processing device according to claim 14, wherein said image processing device is part of said head-mounted display device. further comprising acquisition means for acquiring VR image data representing the virtual environment;
the generating means generates the image data from the VR image;
15. The image processing apparatus according to claim 14, characterized by: the acquisition means further acquires main subject information and/or line-of-sight information obtained when the VR image is captured;
22. The image processing apparatus according to claim 21, wherein said generating means determines said feature area to which said processing is applied, based on said main subject information or said line-of-sight information. An image processing method executed by an image processing device,
a generation step of generating image data for display on a head-mounted display device;
In the generating step, the image data is generated by applying a processing process that visually emphasizes a characteristic region according to the type of virtual environment provided to the user through the display device from other regions.
An image processing method characterized by: A program for causing a computer to function as each means of the image processing apparatus according to claim 8 .

Images