JP2011023898A - Display device, display method, and integrated circuit - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a display device which can easily specify a candidate area without being affected by the movement of the subject. <P>SOLUTION: A video camera 100 includes a detection part 130, a generation part 1620, a synthesis part 1640, an operation part 180, and a selection part 1650. The detection part 130 detects a candidate area from image data. The generation part 1620 generates candidate image data for identifying each candidate area detected by the detection part 130. The synthesis part 1640 generates synthetic image data by synthesizing the candidate image data and the image data. A display part 160 displays the synthetic image data as a video. The operation part 180 receives input of specification information which specifies a specific candidate image among the candidate images which appear in the video. The selection part 1650 selects the candidate area corresponding to the candidate image data specified through the operation part 180 as a reference area based on the specification information input to the operation part 180. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a display device, a display method, and an integrated circuit for displaying an image.

An imaging device such as a video camera or a digital camera often photographs a moving person or object. The imaging device is equipped with a display device for displaying an image, and an image of the subject is displayed on the display device in real time at the time of shooting. The user takes a still image or a moving image while viewing the video displayed on the display device.
On the other hand, a technique has been proposed in which a subject such as a human being displayed in a video is automatically detected as a candidate area. As this type of technique, for example, a face detection technique is known. In a display device using such a technique, an area (hereinafter also referred to as a main subject area) that is automatically detected and corresponds to a subject (hereinafter also referred to as a main subject) that is most interesting to the user among candidate areas. ) Can be selected. When the main subject area is selected, automatic exposure control (AE) and automatic focus control (AF) are performed based on the selected main subject area (see, for example, Patent Document 1). ).

JP 2006-101186 A

However, in the conventional display device, the automatically detected candidate area is directly specified to determine the main subject area. Therefore, it is difficult to specify the candidate area when the main subject is moving in the video. In addition to the display device provided in the imaging device, there is such a problem as long as the device has a function of selecting a specific area from the video.
An object of the present invention is to provide a display device, a display method, and an integrated circuit that can easily specify a candidate area without being affected by the movement of a subject.

A display device according to the present invention is a device for displaying a plurality of image data as video, and includes a detection unit, a generation unit, a synthesis unit, an operation unit, and a selection unit. The detection unit detects at least one candidate region having a specific visual feature from the image data. The generation unit generates at least one candidate identification information for identifying each candidate region detected by the detection unit. The combining unit generates combined image data in which the candidate identification information and the image data are combined. The display unit displays the composite image data as a video. The operation unit receives input of designation information for designating specific candidate identification information among candidate identification information appearing in the video. The selection unit selects a candidate region corresponding to the candidate identification information specified via the operation unit as a reference region based on the designation information input to the operation unit.
In this display device, a candidate area having a specific visual feature is detected from the image data by the detection unit, and candidate identification information corresponding to the detected candidate area is generated by the generation unit. When the candidate identification information is generated, synthesized image data obtained by synthesizing the generated candidate identification information and the image data is generated by the synthesizing unit, and a plurality of synthesized image data is displayed as a video on the display unit. Since the video displayed on the display unit displays not only image data but also candidate identification information, the user can visually recognize the candidate identification information through the video displayed on the display unit.

Furthermore, when the user inputs designation information using the operation unit in order to designate specific candidate identification information among candidate identification information appearing in the video, the designated candidate identification is performed based on the inputted designation information. A candidate area corresponding to the information is selected as a reference area by the selection unit.
Thus, in this display device, candidate identification information corresponding to the candidate area can be displayed separately from the candidate area, and the candidate area can be selected indirectly using the candidate identification information. Even if the displayed candidate area is moving, the selection operation of the candidate area can be easily performed.
A display method according to the present invention is a display method for displaying a plurality of image data as video, and includes a detection step, a generation step, a synthesis step, a display step, an operation step, and a selection step. I have. In the detecting step, at least one candidate region having a specific visual feature is detected from the image data. In the generation step, at least one candidate identification information for identifying each candidate region detected in the detection step is generated. In the synthesis step, synthesized image data in which the candidate identification information and the image data are synthesized is generated. In the display step, the composite image data is displayed as a video. In the operation step, input of designation information for designating specific candidate identification information among candidate identification information appearing in the video is accepted. In the selection step, based on the designation information input to the operation unit, a candidate region corresponding to the candidate identification information designated via the operation unit is selected as a reference region.

In this display method, candidate areas having specific visual features are detected from image data, and candidate identification information corresponding to the detected candidate areas is generated. When the candidate identification information is generated, composite image data obtained by combining the generated candidate identification information and image data is generated, and a plurality of composite image data is displayed as a video. In the displayed video, not only image data but also candidate identification information appears, so that the user can visually recognize the candidate identification information through the displayed video.
Further, when the user inputs the designation information in the operation step in order to designate the specific candidate identification information among the candidate identification information appearing in the video, the designated candidate identification information is changed based on the inputted designation information. The corresponding candidate area is selected as the reference area.
Thus, in this display method, candidate identification information corresponding to the candidate area can be displayed separately from the candidate area, and the candidate area can be indirectly selected using the candidate identification information. Even when the displayed candidate area is moving, the candidate area can be easily selected.

An integrated circuit according to the present invention is an integrated circuit for displaying a plurality of image data as video, and includes a detection unit, a generation unit, a synthesis unit, an operation unit, and a selection unit. The detection unit detects at least one candidate region having a specific visual feature from the image data. The generation unit generates at least one candidate identification information for identifying each candidate region detected by the detection unit. The combining unit generates combined image data in which the candidate identification information and the image data are combined. The display unit displays the composite image data as a video. The operation unit receives input of designation information for designating specific candidate identification information among candidate identification information appearing in the video. The selection unit selects a candidate region corresponding to the candidate identification information specified via the operation unit as a reference region based on the designation information input to the operation unit.
In this integrated circuit, candidate regions having specific visual features are detected from the image data by the detection unit, and candidate identification information corresponding to the detected candidate regions is generated by the generation unit. When the candidate identification information is generated, synthesized image data obtained by synthesizing the generated candidate identification information and the image data is generated by the synthesizing unit, and a plurality of synthesized image data is displayed as a video on the display unit. Since the video displayed on the display unit displays not only image data but also candidate identification information, the user can visually recognize the candidate identification information through the video displayed on the display unit.

Furthermore, when the user inputs designation information using the operation unit in order to designate specific candidate identification information among candidate identification information appearing in the video, the designated candidate identification is performed based on the inputted designation information. A candidate area corresponding to the information is selected as a reference area by the selection unit.
Thus, in this integrated circuit, the candidate identification information corresponding to the candidate area can be displayed separately from the candidate area, and the candidate area can be indirectly selected using the candidate identification information. Even when the displayed candidate area is moving, the candidate area can be easily selected.
Here, as the display device, a device having a function of displaying a plurality of image data as a video and detecting a specific area appearing in the video can be considered. Specifically, as the display device, for example, a display device mounted on an imaging device such as a video camera, a digital camera, and a mobile phone having a camera function can be considered, and further, a television, a car navigation system, a game machine, etc. A display device mounted on an electronic device is also conceivable.

  In addition, examples of the “candidate area having a specific visual feature” include an area representing all or part of the human body (for example, a face area) and an area having a specific color.

  As described above, according to the display device, the display method, and the integrated circuit according to the present invention, the candidate area can be easily specified without being affected by the movement of the subject.

Block diagram of video camera 100 Block diagram of detector 130 Block diagram of system controller 157 and its surroundings Display example of display unit 160 Flow chart of face detection processing Flow chart of face detection processing Flow chart of face detection processing Block diagram of detector 230 and its surroundings Block diagram of system controller 257 and its surroundings Flow chart of registration mode (second embodiment) Flowchart of face detection process (second embodiment) Flowchart of face detection process (second embodiment) Display example of display unit 160 (second embodiment) Example of input screen for inputting character string (third embodiment) Flow chart of registration mode (third embodiment) Display example of display unit 160 (third embodiment) Display example of display unit 160 (another embodiment)

[First Embodiment]
<Overview of video camera>
The video camera 100 will be described with reference to FIGS.
FIG. 1 is a block diagram of the video camera 100. In FIG. 1, a range surrounded by a broken line represents a video camera 100 (an example of an imaging apparatus).
As shown in FIG. 1, the video camera 100 mainly includes an optical system 105, a video signal generation unit 70, a video signal analysis unit 71, a system control unit 157, an optical mechanism control unit 72, and a recording processing unit. 73, a display unit 160, an operation unit 180, a buffer memory 135, and a system bus 134. For example, the video signal analysis unit 71, the system control unit 157, the optical mechanism control unit 72, the recording processing unit 73, the display unit 160, and the operation unit 180 constitute a display device that displays a plurality of image data as video.

<Optical system>
The optical system 105 forms an optical image of the subject and guides the optical image to the image sensor 110. Specifically, the optical system 105 includes a zoom lens group (not shown), a focus lens (not shown), a diaphragm unit (not shown), and a shutter unit (not shown). ing.
The zoom lens group has a plurality of lenses, and is supported by a plurality of lens frames so as to be movable in the optical axis direction. The plurality of lenses are driven along the optical axis by a zoom drive unit (not shown). When zooming the optical image of the subject, each lens is driven along the optical axis by the zoom drive unit. The focus lens is a lens for performing focusing (adjustment of the in-focus state), and is driven by a focus motor (not shown). The focus motor is controlled by a lens control unit 170 (described later). By driving the focus lens in the optical axis direction, the distance from the subject imaged on the image sensor 110 to the video camera 100 can be adjusted, and the in-focus state can be adjusted. The aperture unit is a unit for performing exposure control, and is controlled by an exposure control unit 165 (described later).

<Video signal generation unit>
The video signal generation unit 70 is a unit that generates a digital video signal based on an optical image of a subject. The image sensor 110, the A / D conversion unit 115, the video signal processing unit 120, and the Y / C conversion unit 125. And have.
The image sensor 110 converts an optical image of a subject formed by the optical system 105 into an electric signal (video signal). As the image sensor 110, for example, a charge coupled device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) image sensor are used. The A / D converter 115 converts the analog video signal output from the image sensor 110 into a digital video signal in RGB format.

The video signal processing unit 120 performs video signal processing such as gain adjustment processing, noise removal processing, gamma correction processing, aperture processing, and knee processing on the digital video signal output from the A / D conversion unit 115. The video signal processing unit 120 outputs the digital video signal subjected to the video signal processing to a Y / C conversion unit 125, a detection unit 130, and an image information extraction unit 132 (described later). The Y / C conversion unit 125 converts the digital video signal processed by the video signal processing unit 120 from the RGB format to the Y / C format. The digital video signal output from the video signal processing unit 120 and the Y / C conversion unit 125 is information representing the video of the subject, and is composed of a plurality of frames of image data.
<Buffer memory>
The buffer memory 135 acquires the digital video signal converted into the Y / C format by the Y / C conversion unit 125 through the system bus 134 and temporarily stores it as a plurality of frames of image data (hereinafter also referred to as reference image data). Store. The system control unit 157 (described later) performs a reduction process on the digital video signal stored in the buffer memory 135 to a size suitable for display on the display unit 160 (described later). The reduced digital video signal is stored in the buffer memory 135 as a display video signal. The display video signal is composed of a plurality of frames of image data. The image data constituting the display video signal subjected to the reduction process by the system control unit 157 is also referred to as display image data hereinafter.

<Recording processing unit>
The recording processing unit 73 is a unit that stores the digital video signal generated by the video signal generating unit 70 in an external recording medium, and includes a CODEC 140, a recording I / F unit 145, and a socket 150. . The CODEC 140 performs irreversible compression processing on the Y / C format digital video signal stored in the buffer memory 135. The recording I / F unit 145 acquires the digital video signal subjected to the irreversible compression processing by the CODEC 140 via the system bus 134 and records it on the recording medium 155. The recording medium 155 can be attached to the socket 150, and the recording medium 155 is electrically connected to the video camera 100. An example of the recording medium 155 is a memory card.
<Video signal analysis unit>
The video signal analysis unit 71 is a unit that acquires necessary information from the input video signal, and includes a detection unit 130 and an image information extraction unit 132.

(1) Detection Unit The detection unit 130 receives an input of a digital video signal output from the video signal processing unit 120, and detects a candidate region having a specific visual feature from the input digital video signal. Specifically, as illustrated in FIG. 2, the detection unit 130 includes an image processing unit 200, a volatile memory 210, and a face detection unit 220. In this embodiment, the candidate area is an area representing a human face.
The image processing unit 200 performs a reduction process on the digital video signal input from the video signal processing unit 120. More specifically, the image processing unit 200 sequentially performs a reduction process on a plurality of frames of image data constituting a digital video signal. Hereinafter, the image data subjected to the reduction process by the image processing unit 200 is also referred to as detection image data. Since the reduction magnification at the time of the reduction processing is the same as the reduction magnification of the display image data with respect to the reference image data, the coordinate information on the detection image data can be used as the coordinate information on the display image data. The volatile memory 210 sequentially stores the detection image data output from the image processing unit 200.

The face detection unit 220 detects a face area (an example of a candidate area) from one frame of detection image data stored in the volatile memory 210. The ROM (described later) of the system control unit 157 stores in advance a reference feature amount (an example of reference information) indicating general features related to a human face. The reference feature amount is data indicating the feature of the face appearing in the image. The face detection unit 220 acquires the reference feature amount from the ROM, compares the sequentially input detection image data and the reference feature amount by a method such as pattern matching, and the region that satisfies the reference feature amount is set as a candidate region. It detects from the detection image data of each frame.
Further, the face detection unit 220 generates coordinate information indicating the position of the detected face area in the image and size information indicating the size of the face area, and uses the coordinate information and size information (hereinafter, these information as candidate information). Is also transmitted to the system control unit 157. When there are a plurality of face areas in one frame of detection image data, the face detection unit 220 transmits coordinate information and size information corresponding to each face area to the system control unit 157. Since the detection of the face area by the face detection unit 220 is performed on the detection image data of each frame, the coordinate information and size information of the face area are sent to the system control unit 157 for each frame.

(2) Image Information Extraction Unit The image information extraction unit 132 extracts various information included in the digital video signal. The various information is, for example, a luminance level value and a high frequency component in a specific area on the image. In the image processing in the image information extraction unit 132, an image that has not been subjected to reduction processing that constitutes a digital video signal input from the video signal processing unit 120 in consideration of the accuracy of automatic exposure control and automatic focusing control. Data is used. In the present embodiment, a main subject area (described later) selected via the operation unit 180 is used as the specific area on the image.
The extracted brightness level value is used for automatic exposure control. Specifically, the image information extraction unit 132 calculates the luminance level value of the main subject area from the sequentially input image data, and outputs the luminance level value to the system control unit 157.

The system control unit 157 calculates an optimum aperture value based on the brightness level value calculated by the image information extraction unit 132, and transmits the calculated aperture value to the exposure control unit 165. Based on the aperture value calculated by the system controller 157, the exposure controller 165 adjusts the aperture of the aperture unit of the optical system 105. The luminance level value is calculated for the image data of each frame.
The high frequency component is used for automatic focusing control of a contrast detection method known as “mountain climbing method”. The high frequency component included in the digital video signal changes according to the degree of focusing, and takes a maximum value in a state where the focus lens is disposed at the focusing position. By determining the maximum value of the high frequency component of a specific region while moving the focus lens included in the optical system 105 in the optical axis direction, the position of the focus lens that can focus on that region can be determined. .

Specifically, while the focus lens is driven in the optical axis direction by the lens control unit 170, the image information extraction unit 132 calculates a high frequency component of the main subject region from sequentially input image data, The high frequency component is output to the system control unit 157. The system control unit 157 acquires the high frequency component calculated by the image information extraction unit 132 from the image information extraction unit 132, and at the same time, obtains the position information of the focus lens when the high frequency component is calculated as the lens. Obtained from the control unit 170.
Based on the sequentially acquired information, the system control unit 157 calculates the maximum value of the high frequency component and the position (focus position) of the focus lens corresponding to the maximum value, and moves the focus lens to the focus position. A control signal for driving up to is generated. Based on this control signal, the lens controller 170 drives the focus lens to the in-focus position, and the above-described in-focus operation is repeated. Thereby, it is possible to always focus on the face area of the main subject.

<Optical mechanism control unit>
The optical mechanism control unit 72 is a unit that controls the focus motor, the shutter unit, and the aperture unit, and includes an exposure control unit 165 and a lens control unit 170.
The exposure control unit 165 drives the aperture unit provided in the optical system 105 based on the aperture value calculated by the system control unit 157. The lens control unit 170 controls the focus motor based on the target position of the focus lens calculated by the system control unit 157, and the focus lens included in the optical system 105 is driven to the target position by the focus motor.
<Display section>
The display unit 160 is a device that displays a digital video signal, and is, for example, a small LCD panel. The display unit 160 displays the display video signal stored on the buffer memory 135 as a video.

As will be described later, when the face area detected by the detection unit 130 exists, the display video signal is combined with the candidate image generated based on the detected face area, and further indicates the current operation status. It is combined with an image showing various information such as an icon, recording time and remaining battery time. The display video signal in which these images are combined is transmitted to the display unit 160 through the system bus 134 and displayed as a video by the display unit 160.
Here, a display example of the display unit 160 will be described. As shown in FIG. 4, for example, two humans (subjects) are displayed on the display unit 160, and the faces of both subjects are detected as face areas by the detection unit 130, respectively. The right subject's face 400a is surrounded by a first detection frame 430a, and the left subject's face 400b is surrounded by a second detection frame 430b. The area surrounded by the first detection frame 430a is the face area of the face 400a, and the area surrounded by the second detection frame 430b is the face area of the face 400b. As will be described later, the face detection process is performed on the detection image data of each frame, and the face area is tracked even when the subject moves, so the first detection frame 430a and the first detection frame 2 The detection frame 430b moves in accordance with the movement of the subject.

Further, a horizontally long candidate display area 410 is displayed on the lower side of the screen, and this candidate display area 410 is stationary in the screen. In the candidate display area 410, four partition areas 410a to 410d, a left operation area 420a, and a right operation area 420b are displayed.
Images of the detected face area are displayed in the divided areas 410a to 410d. For example, a face image (image of the right subject's face 400a) in the first detection frame 430a is displayed in the partition area 410a, and a face image (left subject in the second detection frame 430b) is displayed in the partition area 410b. Is displayed). When five or more face areas are detected, the four divided areas 410a to 410d cannot be displayed, and four face images out of the five are displayed in the divided areas 410a to 410d, and the remaining one face image is displayed. Is not displayed, but by touching the left operation area 420a and the right operation area 420b, the face images displayed in the partition areas 410a to 410d can be scrolled left and right, and the face images that are not displayed are displayed. Can do.

  Further, the first detection frame 430a and the second detection frame 430b have different visual characteristics. Specifically, the first detection frame 430a is drawn with a broken line with a narrow interval, and the second detection frame 430b is drawn with a broken line with a wider interval than the first detection frame 430a. In addition, a first decorative frame 411a having the same visual characteristics as the first detection frame 430a is added to the partitioned area 410a where the face image of the first detection frame 430a is displayed, and the second detection frame 430b A second decorative frame 411b having the same visual characteristics as the second detection frame 430b is added to the partitioned area 410b where the face image is displayed. Further, the detection frame and decoration frame of the face area selected as the main subject are displayed with thicker lines than other frames, for example, so that the face area currently selected as the main subject becomes clear. It has become. In FIG. 4, since the face area of the second detection frame 430b is selected as the main subject, the second detection frame 430b and the second decoration frame 411b are displayed in bold lines.

In the present embodiment, the face area (the first detection frame 430a and the second detection frame 430b) and the partition areas 410a to 410d are square, and have similar shapes even if the sizes are different. For this reason, as will be described later, the size of the image of the face region can be easily adjusted according to the size of the partition region.
<Operation unit>
The operation unit 180 includes various operation members for inputting operation information to the system control unit 157. For example, the operation unit 180 includes various buttons such as a release button (not shown), an operation dial, and a touch panel unit. ing. The buttons, operation dial, and touch panel unit are electrically connected to the system control unit 157. The release button outputs a shooting start signal to the system control unit 157.

The touch panel unit of the operation unit 180 is mounted on the screen of the display unit 160 and detects a touch position (an example of a contacted position) on the screen of the display unit 160. When the user touches the screen, the touch panel unit generates designation information indicating the touch position, and outputs the designation information to the system control unit 157. The designation information is used when a specific face image is designated from among the face images appearing in the video and when the displayed face image is scrolled.
<System controller>
The system control unit 157 controls each unit of the video camera 100. The system control unit 157 includes an MPU (Micro Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). A program is stored in the ROM. For example, when the power is turned on, the program stored in the ROM is expanded in the RAM. The system control unit 157 can realize various functions by causing the MPU to sequentially execute the program expanded in the RAM.

Specifically, as illustrated in FIG. 3, the system control unit 157 includes a list management unit 1610, a generation unit 1620, and a function block realized by cooperation of a predetermined program stored in the ROM and the MPU. A synthesis unit 1640 and a selection unit 1650 are included.
(1) List Management Unit The list management unit 1610 manages each piece of information on the face area detected by the face detection unit 220 of the detection unit 130. The list management unit 1610 creates and updates a candidate information list based on the coordinate information and size information transmitted from the face detection unit 220. Here, creating the candidate information list means that the coordinate information and the size information are associated with each other and stored at a predetermined address. Each piece of information in the candidate information list is stored on the RAM 1630, for example.

The list management unit 1610 determines whether a candidate information list exists on the RAM 1630. When the candidate information list does not exist on the RAM 1630, the list management unit 1610 creates a candidate information list. The coordinate information and size information of the face area are sent from the face detection unit 220 to the list management unit 1610 for each frame. The list management unit 1610 updates the coordinate information and size information stored as the candidate information list to the latest information using the coordinate information and size information sent from the face detection unit 220.
(2) Generation Unit The generation unit 1620 generates candidate identification information related to the candidate area detected by the detection unit 130 based on the display video signal stored in the buffer memory 135. Specifically, based on the coordinate information and size information of each face area detected by the detection unit 130, the generation unit 1620 has the same timing as the detection image data that is the target of face detection. An image in each face area is extracted as candidate image data (an example of candidate identification information) from certain display image data, and the extracted candidate image data is associated with the coordinate information and size information of the face area on the buffer memory 135. Store at a predetermined address. Candidate image data for each face area is extracted sequentially for each frame. In this embodiment, since the candidate area is a face area, the image represented by the candidate image data (hereinafter referred to as a candidate image) is a human face image.

In addition, since the size of the face area usually varies, the size of the extracted candidate image is often different for each face area. Therefore, before associating the candidate image data with the coordinate information and the size information, the generation unit 1620 compares the size of the preset partition areas 410a to 410d with the size of the candidate image, and based on the comparison result, the candidate image Adjust the size of.
For example, if the candidate image is smaller than the partition area, the generation unit 1620 performs an enlargement process on the candidate image data according to the partition area, and associates the candidate image data subjected to the enlargement process with the coordinate information and the size information as a buffer. The data is stored at a predetermined address on the memory 135. In addition, when the candidate image is larger than the partitioned area, the generation unit 1620 performs a reduction process on the candidate image data according to the partitioned area, and associates the candidate image data subjected to the reduced process with the coordinate information and the size information as a buffer. The data is stored at a predetermined address on the memory 135. Note that the generation unit 1620 does not perform the reduction process and the enlargement process on the candidate image when the candidate image has the same size as the partition area.

(3) Combining Unit The combining unit 1640 combines the candidate image data generated by the generating unit 1620 and the display image data on the buffer memory 135, and generates combined image data obtained by combining the two. More specifically, the composition unit 1640 assigns candidate images to the partition regions 410a to 410d, and sets the candidate image data and display image data so that the candidate images are arranged at specific positions on the screen of the display unit 160. Synthesize.
In the present embodiment, as shown in FIG. 4, a plurality of candidate images are arranged in the horizontal direction on the screen of the display unit 160. Further, candidate images are arranged in the lower half region in the vertical direction in the video displayed on the display unit 160. Here, the “vertical direction” means the vertical direction in the image displayed on the display unit 160. The composite image data is stored in the buffer memory 135. With such an arrangement, it is possible to prevent the face of the subject from being hidden in the candidate image.

Further, the synthesis unit 1640 synthesizes the candidate image data and the display image data so that the candidate image is superimposed on an image generated based on the image data while the video is displayed on the display unit 160. . In the partitioned areas 410c and 410d where no candidate image is displayed, only the frame is displayed, so that the back image can be seen as it is.
As shown in FIG. 4, the area of the candidate display area 410 (an example of the first display area) on which the candidate image is displayed on the screen of the display unit 160 is an area 420 (second display area) other than the candidate display area 410. It is set smaller than the area of one example of the display area.
Further, when the candidate image data and the display image data are combined, the combining unit 1640 adds a decoration frame (an example of decoration information) indicating a face area to each partition area. In the example of FIG. 4, a first decoration frame 411a (an example of first decoration information) corresponding to the first detection frame 430a is added to the partition area 410a, and a second decoration frame 411b (first frame) corresponding to the second detection frame 430b is added. 2 (an example of decoration information) is added to the partition area 410b. The first decorative frame 411a has the same visual features as the first detection frame 430a, and the second decorative frame 411b has the same visual features as the second detection frame 430b. Specifically, as described above, the first detection frame 430a and the first decoration frame 411a are drawn with a narrow broken line, and the second detection frame 430b and the second decoration frame 411b are the first detection frame 430a and It is drawn with a broken line that is wider than the first decorative frame 411a.

Furthermore, the detection frame and the decoration frame added to the face area (an example of the reference area) selected as the main subject have different visual characteristics from the other detection frames and the decoration frame. For example, when the face 400b is selected as the main subject, as shown in FIG. 4, the second detection frame 430b and the second decoration frame 411b are drawn with thicker lines than the first detection frame 430a and the first decoration frame 411a. It is.
(4) Selection Unit The selection unit 1650 performs automatic exposure control and automatic focusing control on the face area corresponding to the candidate image specified via the operation unit 180 based on the designation information input to the operation unit 180. It is selected as a main subject area (an example of a reference area) that serves as a reference. Specifically, the selection unit 1650 identifies the candidate image displayed at the position indicated by the designation information based on the designation information input via the operation unit 180, and the face corresponding to the identified candidate image. An area is specified, and the specified face area is selected as a main subject area. The selection unit 1650 stores the coordinate information and size information of the face area corresponding to the candidate image specified based on the designation information in the RAM 1630 as main subject area information. When a different candidate image is designated via the operation unit 180, the information on the main subject area stored in the RAM 1630 is replaced with the coordinate information and size information of the face area specified based on the new designation information. The area information is updated. This main subject area information is used when the image information extraction unit 132 calculates the luminance level value and the high frequency component.

  Further, as described above, the system control unit 157 controls the image information extraction unit 132, the exposure control unit 165, and the lens control unit 170, so that automatic exposure control and automatic focusing control for the main subject region are performed. To do. Specifically, the system control unit 157 notifies the image information extraction unit 132 of coordinate information and size information of the face area selected as the main subject area by the selection unit 1650. The image information extraction unit 132 calculates the luminance level value and high frequency component of the main subject area based on the coordinate information and the size information, and notifies the system control unit 157 of the calculated luminance level value and high frequency component. . The system control unit 157 calculates an optimum aperture value based on the luminance level value, and notifies the exposure control unit 165 of the aperture value. Further, the system control unit 157 calculates position information of the focus lens corresponding to the in-focus state based on the high frequency component, and notifies the lens control unit 170 of the focus position of the focus lens. Based on these pieces of information, automatic exposure control and automatic focusing control are performed.

<Operation of the video camera>
The operation of the video camera 100 described above will be described with reference to the flowcharts shown in FIGS. In the flowcharts shown in FIGS. 5 to 7, the video camera 100 is already turned on, and the operation mode is set to the recording mode in which video can be recorded using the operation unit 180. To do.
(1) Generation of Candidate Image Data As shown in FIG. 5, first, in step S300, image data constituting a digital video signal is sequentially input to the detection unit 130 and the image information extraction unit 132. Specifically, a subject image formed through the optical system 105 is converted into an electrical signal by the image sensor 110, and an analog video signal output from the image sensor 110 is converted into a digital video signal by the A / D converter 115. The The digital video signal output from the A / D conversion unit 115 is subjected to the above-described processing by the video signal processing unit 120 and the Y / C conversion unit 125, and the processed digital video signal has a plurality of frames. This is a collection of image data, and the image data of each frame is sequentially stored in the buffer memory 135.

In addition, the image data constituting the digital video signal output from the Y / C conversion unit 125 is reduced to a size that can be displayed on the display unit 160 by the system control unit 157, and is also stored on the buffer memory 135. Are sequentially stored as display image data. Also, the image data subjected to the reduction process by the image processing unit 200 is sequentially stored in the volatile memory 210 in the detection unit 130 as detection image data.
In step S <b> 310, the face detection unit 220 detects a face area based on the detection image data stored in the volatile memory 210 in the detection unit 130. Specifically, the face detection unit 220 compares the reference feature amount indicating the feature of the human face with the detection image data, and detects a face region that satisfies the reference feature amount from the detection image data as a candidate region. Is done. The face detection unit 220 calculates coordinate information and size information of the detected face area, and outputs the coordinate information and size information from the face detection unit 220 to the system control unit 157.

The processing after step S311 is executed by the system control unit 157. Specifically, in step S311, the list management unit 1610 determines whether a face area exists on the candidate information list stored in the RAM 1630. For example, if the list management unit 1610 determines that there is a candidate information list, the process proceeds to step S313.
On the other hand, for example, since the candidate information list does not exist immediately after switching to the recording mode, when the face area is detected from the detection image data by the detection unit 130 in step S312, the list management unit 1610 creates the candidate information list. The coordinate information and the size information are stored at a predetermined address on the RAM 1630, and the process proceeds to step S313.
In step S313, in order to track the detected face area, the coordinate information of the face area detected from the detection image data is collated with each coordinate information on the candidate information list by the list management unit 1610.

  Here, a process for tracking the detected face area will be described. The tracking process of the detected face area is performed by determining whether or not the face area detected from the detection image data for the current frame matches the face area detected from the detection image data for the previous frame. Done. The simplest method of this determination is a method of comparing the coordinates of face areas. Generally, because 30 or 60 frames of image data are input per second, the face area detected in the image data of the previous frame is far away from the previous face area in the image data of the current frame. It is unlikely to be detected at the position. Therefore, the difference between the coordinate information of the face area detected from the detection image data of the previous frame and the coordinate information of the face area detected from the detection image data of the current frame is within a predetermined range. In this case, it can be determined that the two face areas are face areas of the same subject.

In this way, by collating the coordinate information of the face area detected from the two adjacent detection image data in time series, the face detection process is performed while identifying that the face of the moving subject is the same face area. The face area can be tracked. In addition, when a face area that does not match the face area detected from the detection image data of the previous frame is detected from the detection image data of the current frame, the face of the subject newly appearing in the screen (framed in) is displayed. It can be determined that the face area is shown. Furthermore, when the face area detected from the image data for detection in the previous frame is not detected in the image data for detection in the current frame, the face area indicates the face of the subject that has disappeared (out of frame) from the screen. Can be determined.
In step S313, the list management unit 1610 collates face areas detected from two detection image data adjacent in time series. Specifically, the coordinate information of each face area detected in step S310 is compared with the coordinate information of each face area existing on the candidate information list. As a result of the comparison, it can be determined that the two face areas in which the difference in the coordinate information is within a predetermined range are areas detected from the face of the same subject, and therefore, the latest face area detected in step S310. Coordinate information and size information are registered in the candidate information list, and the candidate information list is updated.

In step S320, if there is a face area registered in the candidate information list but not detected in step S310, the list management unit 1610 determines that the face area is the face area of the subject that is out of frame. The If there is a frame-out face, in step S330, the information corresponding to the face area of the frame-out subject is deleted from the candidate information list by the list management unit 1610, and the process proceeds to step S340. If the list management unit 1610 determines in step S320 that there is no face area of the framed subject, the process proceeds to step S340.
In step S340, if the information corresponding to the face area detected in step S310 does not exist on the candidate information list, the list management unit 1610 determines that the face area is a face area of a new subject framed in. . If the face area of the new subject exists, as shown in FIG. 6, the coordinate information and size information of the face area of the new subject are added to the candidate information list by the list management unit 1610 in step S350, and then the process is performed. The process proceeds to step S351. If there is no new subject face area, the process proceeds to step S351.

In step S351, the generation unit 1620 acquires coordinate information and size information of each face area from the updated candidate information list. In subsequent step S360, based on the acquired coordinate information and size information, the generation unit 1620 generates candidate image data corresponding to each face area.
Specifically, as shown in FIG. 6, the image within the area specified by the coordinate information and the size information is extracted as candidate image data from the display image data stored in the buffer memory 135 by the generation unit 1620. Further, in step S1520, the candidate images generated from the extracted candidate image data are compared with the sizes of the partitioned areas 410a to 410d. When the candidate image is smaller than the partitioned areas 410a to 410d and when the candidate image is larger than the partitioned areas 410a to 410d, the generation unit 1620 determines that the size needs to be changed in step S1530. If a size change is necessary, the size of the extracted candidate image is changed in step S1540. Specifically, when the candidate image is larger than the partitioned areas 410a to 410d, the candidate image is calculated based on the size information and the size of the partitioned areas 410a to 410d so that the candidate image fits within the partitioned areas 410a to 410d. The generation unit 1620 performs reduction processing on the candidate image data at the same magnification. Conversely, when the extracted candidate image is smaller than the partitioned areas 410a to 410d, the size information and the partitioned areas 410a to 410d are displayed so that the candidate image is displayed as large as possible within the partitioned areas 410a to 410d. The generation unit 1620 performs an enlargement process on the candidate image data at a magnification based on the size of the image. The candidate image data generated by the generation unit 1620 is stored in a predetermined address on the buffer memory 135 in association with the corresponding coordinate information and size information. Candidate image data is generated for each frame, and the candidate image data is sequentially updated.

(2) Generation of Composite Image Data As shown in FIG. 6, in subsequent step S370, the system controller 157 combines the candidate image data with the display image data stored in the buffer memory 135. Image data is generated. The candidate image data and the image data are combined by the combining unit 1640 so that the candidate image is arranged at a specific position on the screen. In the present embodiment, the candidate image data and the image data are combined by the combining unit 1640 so that the candidate image is arranged on the lower side in the vertical direction in the image.
The step S370 will be described in detail. As shown in FIG. 6, in step S1550, the composition unit 1640 assigns candidate image data to the four partitioned areas 410a to 410d. At the time of assignment, candidate image data is assigned by the compositing unit 1640 from the left partition area 410a in the order registered in the candidate information list. Referring to the display example of FIG. 4, when face areas are registered in the candidate information list in the order of the face 400a and the face 400b, the candidate image of the face 400a is displayed in the section area 410a, and the candidate image of the face 400b is sectioned. It is displayed in the area 410b. The result of allocation by the synthesis unit 1640 is sent from the synthesis unit 1640 to the list management unit 1610. Specifically, the display position information of each candidate image data is sent from the synthesis unit 1640 to the list management unit 1610, and the candidate information list is displayed in a state where the display position information is associated with the corresponding candidate image data by the list management unit 1610. Registered in

In the next step S1560, the candidate image is combined with the display image data by the combining unit 1640. At this time, the candidate image data and the display image data are combined by the combining unit 1640 so that the candidate image is displayed at the position assigned in step S1550. In addition, since the size of the candidate image is adjusted by the generation unit 1620, the candidate image has the same size as the partition regions 410a to 410d on the screen of the display unit 160 regardless of the size of the detected face region. It is displayed.
In this embodiment, in step S1570, in order to clarify the correspondence between the face area and the candidate image on the screen of the display unit 160, the decoration information is combined with the display image data by the combining unit 1640. The Specifically, decoration data indicating the first detection frame 430a, the second detection frame 430b, the first decoration frame 411a, and the second decoration frame 411b as shown in FIG. 4 is added to the display image data. At this time, although omitted in FIG. 4, in order to realize an OSD (On Screen Display) function, icons such as a remaining recordable time and a remaining battery level are also superimposed on the display image data by the combining unit 1640. .

In the next step S380, the composite image data is displayed on the display unit 160. As shown in the display example in FIG. 4, two people are displayed on the display unit 160, each face is detected as a face area, and a candidate image is further displayed in the candidate display area 410.
Since the processing from step S300 to step S380 is sequentially performed on the display image data of each frame, a composite video composed of the composite image data is displayed on the display unit 160. Since the face detection process is executed so as to track the face of the subject, the face of the same subject can be continuously detected even if the subject is moving. Furthermore, since the candidate images are sequentially extracted from the display image data of each frame, the moving image in the detected face area is displayed in each of the divided areas 410a to 410d.
(3) Designation of main subject area Next, processing when a main subject area is selected as a reference area when performing AE and AF will be described with reference to a flowchart shown in FIG. The main subject area is specified by the selection unit 1650 based on the operation information input from the operation unit 180.

Specifically, as shown in FIG. 7, in step S600, it is determined whether the candidate image displayed on the display unit 160 has been touched. When the screen of the display unit 160 is touched by the user, the touch panel unit of the operation unit 180 generates touch position coordinate information (designated information) on the display unit 160, and the touch position coordinate information is selected from the operation unit 180 by the selection unit. 1650 is notified. In step S601, the selection unit 1650 compares the display position information registered in the candidate information list with the coordinate information of the touch position, and specifies the display position information including the touch position. Further, candidate image data associated with the specified display position information is specified by the selection unit 1650.
In subsequent step S602, the face area corresponding to the identified candidate image data is identified by the selection unit 1650 based on the candidate information list. Specifically, the face area associated with the candidate image data identified in step S601 is selected as the main subject area. In the subsequent step S610, for example, a flag is set in the face area selected as the main subject area in the candidate information list. By referring to the flag in the candidate information list, the face area selected as the main subject area can be specified.

On the other hand, if the candidate image displayed on the display unit 160 is not touched in step S600, the process proceeds to step S630. In step S630, selection unit 1650 determines whether the main subject area has already been determined. Specifically, if the flag indicating the main subject area is set in the candidate information list, the selection unit 1650 determines that the main subject area has been determined, and the process proceeds to step S640. On the other hand, if the flag is not set in the candidate information list, the selection unit 1650 determines that the main subject area has not been determined, and the process proceeds to step S650.
Here, the main subject area that has already been determined in step S630 is a main subject area that has been determined based on at least the composite image data before the currently displayed composite image data. In such a case, since the time has elapsed since the main subject area was selected, there is a concern that the main subject area does not exist in the image. For example, a case where the main subject goes out of the frame or is hidden behind a shielding object and is not visible can be assumed.

However, in steps S313 to S350 shown in FIG. 5, since the subject that has been framed in and out is managed, if the flag set in the candidate information list is confirmed, the main subject region is displayed on the screen. There is no need to determine whether or not.
In step S640, automatic focusing control and automatic exposure control are performed on the main subject area by the system control unit 157. Specifically, the coordinate information and size information of the main subject region are sent to the image information extraction unit 132 by the system control unit 157, and automatic exposure control and automatic focus control are performed based on the image data in the main subject region. It is executed by the extraction unit 132, the system control unit 157, the lens control unit 170, and the exposure control unit 165.
On the other hand, if it is determined in step S630 that the main subject area does not exist, or if it is determined in step S620 that the main subject area does not exist in the screen, for example, a central area of the screen is set in advance. Normal automatic focusing control and automatic exposure control for the area are executed by the image information extraction unit 132, the system control unit 157, the lens control unit 170, and the exposure control unit 165.

In step S650, the presence or absence of a recording mode end instruction is confirmed by the system control unit 157. If there is a recording mode end instruction, the process ends. If there is no end instruction, the process returns to step S300.
Although not shown in the flow, as shown in FIG. 4, the candidate display area 410 is provided with a left operation area 420 a and a right operation area 420 b for switching candidate images displayed in the candidate display area 410. It has been. When the user touches the left operation area 420a or the right operation area 420b, the system control unit 157 has touched the position corresponding to the switching instruction area 420 based on the touch coordinates obtained through the touch panel constituting the operation unit 180. And the candidate image displayed in the candidate display area 410 can be scrolled to the right or left. In this case, the list management unit 1610 updates the correspondence between the candidate image data and the display position information in accordance with the operation status of the left operation area 420a and the right operation area 420b.

<Features>
The characteristics of the video camera 100 described above are summarized below.
(1)
In the video camera 100, a face area (candidate area) is detected from the detection image data by the detection unit 130, and candidate image data corresponding to the detected candidate area is generated by the generation unit 1620. When the candidate image data is generated, composite image data obtained by combining the generated candidate image data and display image data is generated by the combining unit 1640, and the combined image data is displayed on the display unit 160 as a video. In the video displayed on the display unit 160, not only the video generated from the display image data but also the candidate image generated from the candidate image data is displayed, and therefore, through the video displayed on the display unit 160, The user can visually recognize the candidate image corresponding to each candidate area.

Further, when a user designates a candidate image using the operation unit 180 in order to designate a specific candidate image among candidate images appearing in the video, designation information is generated by the operation unit 180, and the input designation information is input. Based on the above, the face area corresponding to the designated candidate image is selected by the selection unit 1650 as the main subject area.
Thus, in this video camera 100, candidate images corresponding to the candidate areas can be displayed separately from the candidate areas, and the candidate areas can be indirectly selected using the displayed candidate images. Even when the subject displayed on the display unit 160 is moving, the candidate region can be easily selected.
(2)
When the composite image data is displayed as a video on the display unit 160, the candidate image is arranged at a specific position on the screen of the display unit 160. Therefore, unlike the detected candidate region, the subject is moving. Even in this case, the candidate image is displayed at a certain position on the screen. Therefore, it is possible to easily specify a desired candidate area via the candidate image without being affected by the movement of the subject, compared to the case where the candidate area is directly specified.

Further, since the combining unit 1640 combines the candidate image data and the display image data so that a plurality of candidate images are arranged on the screen of the display unit 160, the combined image data is displayed on the display unit 160 as a video. In this case, a plurality of candidate images are displayed side by side. For this reason, it becomes easy to confirm a plurality of candidate images, and the operability is further improved.
Further, since the candidate images are arranged in the lower half region in the vertical direction in the video displayed on the display unit 160, the display of the video is hardly hindered by the candidate images.
In particular, when photographing the human upper body or the whole body, the human face tends to concentrate on the upper half of the screen in the vertical direction. Area display is less likely to be disturbed.
(3)
Since the candidate image overlaps the image of the display image data, the image generated from the display image data can be displayed large on the screen of the display unit 160, and the screen of the display unit 160 can be used effectively. it can.

In addition, since the area of the candidate display area 410 where the candidate image is displayed on the screen of the display unit 160 is smaller than the area of the area 420 other than the candidate display area 410, the display image data displayed on the display unit 160 It becomes easier to check the video.
(4)
Image data in each face area detected from one image data is extracted by the generation unit 1620, and the extracted image data is set as candidate image data by the generation unit 1620. For this reason, the correspondence between the candidate image displayed on the display unit 160 and the face area is visually clarified, and the face area can be easily selected.
Further, the size of the partition areas 410a to 410d and the size of the candidate image are compared by the generation unit 1620, and the size of the candidate image is adjusted by the generation unit 1620 based on the comparison result.

Specifically, when the candidate image is smaller than the partition areas 410a to 410d, the generation unit 1620 performs the enlargement process on the candidate image according to the partition areas 410a to 410d, and identifies the candidate image that has been subjected to the expansion process as a candidate identification Set as information.
In addition, when the candidate image is larger than the divided areas 410a to 410d, the generation unit 1620 performs a reduction process on the candidate image according to the divided areas 410a to 410d, and sets the candidate image subjected to the reduction process as candidate identification information. To do.
In this case, for example, even if there is a variation in the size of the candidate image, the size of the candidate image can be matched with the size of the partitioned areas 410a to 410d. Even if the candidate image is smaller than the divided areas 410a to 410d, the candidate image is enlarged to the size of the divided areas 410a to 410d, so that the user can easily confirm the candidate image. Further, even if the candidate image is larger than the partitioned areas 410a to 410d, the candidate image is reduced to the size of the partitioned areas 410a to 410d, so that the user can easily confirm the candidate image.

(5)
When the synthesis unit 1640 synthesizes the candidate image and the image data, the first detection frame 430a indicating the candidate area of the face 400a is added to the candidate area of the face 400a, and the candidate corresponding to the candidate area of the face 400a A first decorative frame 411a indicating that the image is an image is added to the partitioned area where the candidate image is displayed. For this reason, the correspondence between the candidate image displayed on the display unit 160 and the candidate area is further clarified visually, and the candidate area can be easily selected.
For example, as shown in FIG. 4, the first detection frame 430a has a different visual characteristic from the second detection frame 430b, and the first decoration frame 411a has a different visual characteristic from the second decoration frame 411b. Have. Further, the first decorative frame 411a has substantially the same visual characteristics as the first detection frame 430a, and the second decorative frame 411b has substantially the same visual characteristics as the second detection frame 430b. is doing. With these configurations, it becomes easy to identify a plurality of candidate regions, and the correspondence between the candidate regions and the candidate images is visually clarified. Therefore, the candidate area can be easily selected using the displayed candidate image.

[Second Embodiment]
In the first embodiment, all detected face areas are set as candidate images. However, as a modified example, only a face that matches a face registered in advance may be determined as a face area. In the following description, components having substantially the same functions as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.
<Configuration of video camera>
In the video camera 500 according to the second embodiment, the detected face area is registered, and only the face area having the same visual characteristics as the registered face area is detected.
Specifically, as shown in FIGS. 8 and 9, the video camera 500 is different from the first embodiment in the configuration of the detection unit 130 and the configuration of the system control unit 157 in the configuration of the video camera 100. A detection unit 230 and a system control unit 257 are provided. When detecting the face area from the detection image data, the face detection section 220 of the detection section 230 calculates the feature amount of the detected face area. This feature amount is used as data for identifying the detected face area.

As shown in FIG. 8, the detection unit 230 has a recognition unit 700. The recognition unit 700 acquires the feature amount of the face area from the face detection unit 220, the face feature amount (registered feature amount) registered in the volatile memory 210 in advance, and the acquired face feature amount (detection feature amount) And the function of determining whether or not the faces indicated by both are the same. Further, the recognizing unit 700 specifies a registered feature amount that matches the detected feature amount, and sends information about the specified registered feature amount to the list management unit 1610. The list management unit 1610 acquires registered image data associated with the registered feature amount from the buffer memory 135. The recognition unit 700 and the list management unit 1610 constitute a registration information acquisition unit.
FIG. 9 shows the configuration of the system control unit 257 of the video camera 500 in the second embodiment. As illustrated in FIG. 9, the system control unit 257 further includes a registration unit 1660 in addition to the functional blocks similar to those in FIG. 3. The registration unit 1660 stores the registered feature amount as registration information in the volatile memory 210 via the recognition unit 700, and stores the registered image data on the buffer memory 135 in association with the registered feature amount. Also, the registration unit 1660 associates the registered image data with the registered feature amount corresponding thereto, and records it as registration information on the recording medium 155 via the system bus 134 and the recording I / F unit 145. As a result, registration information is not lost even when the power of the video camera 500 is turned off.

<Operation of the video camera>
(1) Registration of face area A method for registering a subject will be described. When a mode change dial (not shown) forming the operation unit 180 is set to the subject registration mode, the video camera 500 shifts to the subject registration mode. When the registration mode is started, the subject image is displayed on the display unit 160 in real time. At this time, as in the first embodiment, candidate images corresponding to the detected face area are displayed in the candidate display area 410.
As illustrated in FIG. 10, in step S800, the operation unit 180 detects a touch position on the touch panel and determines whether any of the partition areas 410a to 410d is touched. When the screen of the display unit 160 is touched, designation information indicating the touch position is generated by the operation unit 180, and the process proceeds to step S801. In step S801, the display position information of each candidate image data registered in the candidate information list and the coordinate information of the touch position are compared by the selection unit 1650, and the candidate image data displayed at the touched position is specified. The process proceeds to step S802.

  In step S802, as in step S602 of the first embodiment, the face area corresponding to the specified candidate image data is specified by the selection unit 1650 based on the candidate information list. In step S810, an image in the face area is extracted as registered image data by the generation unit 1620 based on the coordinate information and size information of the face area specified in step S802. When generating the registered image data, the size of the partitioned areas 410a to 410d and the size of the registered image generated by the registered image data are compared by the generating unit 1620, as in the first embodiment, and based on the comparison result. The size of the registered image is adjusted by the generation unit 1620. In subsequent step S820, the feature amount of the face area specified in step S802 is acquired as a registered feature amount by the detection unit 230. Thereafter, the process proceeds to step S830, where the registration unit 1660 stores the acquired registration feature quantity as registration information in the volatile memory 210 via the recognition unit 700, and the registration image data generated by the generation unit 1620 is registered. It is stored in the buffer memory 135 in association with the feature amount.

Lastly, in step S840, the registration unit 1660 associates the registered image data with the corresponding registered feature value, and is recorded as registration information on the recording medium 155 via the system bus 134 and the recording I / F unit 145. The In step S850, the system control unit 257 checks whether there is an end instruction from the user. Specifically, the system control unit 257 confirms whether or not there is a mode change by a mode change dial that forms the operation unit 180. If the mode change dial is set to a mode other than the registration mode, the registration mode ends. If the mode change dial remains set to the registration mode, the processes in steps S800 to S840 are repeated.
Note that information such as registered image data and registered feature values is not lost even when the power of the video camera 500 is turned off by the processing in step S840.

(2) Display Processing FIG. 11 is a flowchart showing the flow of main object candidate image creation and display processing for specifying a main subject. Since there are steps having the same processing contents as those in the first embodiment shown in FIG.
When the power of the video camera 500 is turned on, in step S1400, the registered feature amount and registered image recorded on the recording medium 155 by the system control unit 257 via the system bus 134 and the recording I / F unit 145. Data loading starts. The read registered feature amount is sent from the system control unit 257 to the recognition unit 700. The registered feature amount is temporarily stored in the volatile memory 210 by the recognition unit 700. The read registered image data is stored in the buffer memory 135 by the system control unit 257, and is used as a candidate image displayed in the partitioned areas 410a to 410d.

For example, in the moving image shooting mode, the video of the subject is displayed on the display unit 160 in real time. Specifically, as in the first embodiment, in steps S300 and S310, detection image data is input to the detection unit 230, and the face detection unit 220 converts the detection image data into the volatile memory 210. Based on this, a face area is detected. At this time, as in the first embodiment, the coordinate information and size information of the face area detected by the detection unit 230 are acquired from the detection image data.
In addition to the face area detection in step S310, in the present embodiment, calculation of feature amounts used for face recognition processing is also performed. Specifically, in step S1410, the feature amount of the face area detected in step S310 is acquired from the detection image data as the detected feature amount. The detected feature amount is data in which human face features are digitized in the same manner as the reference feature amount. However, unlike the reference feature amount, the detected feature amount is data specific to the face area. Therefore, the identity of the face area detected by the detection unit 230 can be determined using the detected feature amount. The detected feature amount detected by the face detection unit 220 is temporarily stored in the volatile memory 210. The detected feature amount is calculated for each face area detected by one frame of detection image data.

In steps S311 to S350, the candidate information list is created and updated as described above. At this time, the list management unit 1610 stores the detected feature amount in association with the coordinate information and the size information on the candidate information list.
Thereafter, in step S1420, the recognition unit 700 compares each registered feature amount stored in the volatile memory 210 with the detected feature amount of the candidate information list. In step S1430, when the recognition unit 700 determines that there is a registered feature amount that matches the detected feature amount, in step S1440, the registered feature amount that matches the detected feature amount is specified by the recognition unit 700, and the specified registration is performed. Information about the feature amount is sent from the recognition unit 700 to the list management unit 1610, and registered image data associated with the specified registered feature amount is acquired from the buffer memory 135 by the list management unit 1610.

In step S1450, the list management unit 1610 associates the registered image data with the detected feature amount. In step S1370, similar to the candidate image data of the first embodiment, the registered image data is combined with the display image data by the combining unit 1640, and the combined image data is generated by the combining unit 1640.
Specifically, as shown in FIG. 12, in step S1371, the composition unit 1640 assigns candidate image data to the four partitioned areas 410a to 410d. At the time of assignment, the registration unit 1640 assigns registered image data from the left partition area 410a in the order registered in the candidate information list. Referring to the display example of FIG. 13, when face areas are registered in the candidate information list in the order of the face 400a and the face 400b, the registered image of the face 400a is displayed in the partitioned area 410a, and the registered image of the face 400b is partitioned. It is displayed in the area 410b. For the face 400c for which no registered image data exists, neither a detection frame nor a registered image is displayed.

The result of allocation by the synthesis unit 1640 is sent from the synthesis unit 1640 to the list management unit 1610. Specifically, display position information of each registered image data is sent from the synthesizing unit 1640 to the list management unit 1610, and the candidate information list is displayed in a state where the display position information is associated with the corresponding registered image data by the list management unit 1610. Registered in
In the next step S 1372, the registration image is combined with the display image data by the combining unit 1640. At this time, the registration unit 1640 synthesizes the registered image data and the display image data so that the registered image is displayed as a candidate image at the position assigned in step S 1371. In addition, since the size of the candidate image is adjusted by the generation unit 1620 when the registered image data is generated, the partition regions 410a to 410d are displayed on the screen of the display unit 160 regardless of the size of the detected face region. Candidate images are displayed at the same size.

Similarly to the first embodiment, in step S1373, in order to clarify the correspondence between the face area and the registered image on the screen of the display unit 160, the composition unit 1640 performs decoration information on the display image data. Is synthesized. Specifically, decoration data indicating the first detection frame 430a, the second detection frame 430b, the first decoration frame 411a, and the second decoration frame 411b as shown in FIG. 13 is added to the display image data.
In the subsequent step S380, the generated composite image data is displayed on the display unit 160. Thus, as shown in FIG. 13, for the registered face areas (faces 400a and 400b) among the detected face areas, the registered images are displayed in the partitioned areas 410a and 410b using the registered image data, and registered. Nothing is displayed for the face area (face 400c) that has not been displayed. Since the registered image is a still image acquired at the time of registration, the orientation and expression are different from the current faces 400a and 400b, and the registered image displayed in the partitioned areas 410a and 410b changes even if the orientation of the subject changes. do not do.

Since the subsequent processing is substantially the same as the flow shown in FIG. 7, detailed description thereof is omitted.
In this case, the feature amount detected by the detection unit 230 is recorded on the recording medium 155 via the registration unit 1660 as a registered feature amount in association with the corresponding candidate identification information. When the detection unit 230 generates the detection feature amount based on the input image data, the generated detection feature amount is collated with the registration feature amount by the recognition unit 700, and is associated with the registration feature amount that substantially matches the detection feature amount. The registered image data is specified by the list management unit 1610.
Further, the registration image data specified by the list management unit 1610 and the display image data are combined by the combining unit 1640 to generate combined image data. In other words, if there is a face region having a feature amount that roughly matches the registered feature amount, candidate identification information associated with the registered feature amount is displayed on the display unit 160. Therefore, if the registered feature quantity is registered in association with the candidate identification information, the same registered image data is displayed on the display unit 160 for the face area having the same feature quantity, and the face area and the corresponding registered image data are displayed. It becomes easy to recognize.

[Third Embodiment]
This embodiment is a modification of the second embodiment. In this embodiment, character information for identifying a face area is registered instead of registered image data, and the registered character information is displayed in the candidate display area 410 of the display unit 160. As the character information, the name of the subject can be considered.
In the registration mode described above, character information indicating a registration name is input via the operation unit 180 when generating registration image data. An example of an input screen for inputting character information is shown in FIG.
In FIG. 14, a registered image 1000 is a face image to be registered. The soft keyboard 1010 is an area for inputting character information, and includes a character type switching key area, a character deletion key area, and a determination key area in addition to the character key area. When a touch panel provided on the display unit 160 and constituting the operation unit 180 is touched by the user, the system control unit 257 determines which key area of the soft keyboard 1010 indicates the touched coordinates, and It is assumed that the character corresponding to the key to be input has been input. The input characters are displayed in the input area 1020.

FIG. 15 is a flowchart for registering a subject in this embodiment. Steps similar to those in FIG. 10 are denoted by the same reference numerals, and detailed description thereof is omitted.
The difference from the flowchart in FIG. 10 shown in the second embodiment is Step S1100. In Step S1100, as described above, a character string for identifying a main subject candidate is input. At this time, the registered image data extracted in step S810 is displayed on the display unit 160 as a registered image 1000.
Further, the registration information recorded on the recording medium 155 in step S840 includes the character information input in step S1100 in addition to the registered image data and the registration feature amount described in the second embodiment. At this time, the character information is associated with the registered feature value by the registration unit 1660 and recorded in the buffer memory 135, for example.

FIG. 16 shows a display example of this embodiment. Constituent elements similar to those in FIG. In FIG. 16, the candidate display area 410 is provided with partitioned areas 1200a to 1200d corresponding to the partitioned areas 410a to 410d described above. Identification character strings 1202a and 1202b for identifying the face area are displayed, the identification character string 1202a corresponds to the right face 400a, and the identification character string 1202b corresponds to the left face 400b.
In addition, since the identification character string 1202a corresponds to the face 400a, the first decorative frame 1201a surrounding the partition area 1200a is displayed with a broken line of the same type as the first detection frame 430a. Since the identification character string 1202b corresponds to the face 400b, the second decorative frame 1201b surrounding the partitioned area 1200b is displayed with a broken line of the same type as the second detection frame 430b. Further, the detection frame and the decoration frame of the face area selected as the main subject area are displayed with bold lines. In the display example shown in FIG. 16, since the region surrounded by the second detection frame 430b is selected as the main subject region, the second detection frame 430b and the second decoration frame 1201b are displayed with bold lines.

Similar to the first and second embodiments described above, by touching the identification character string 1202a, the face area 430a corresponding to the identification character string 1200a is selected as the main subject area.
In this case, basically the same effect as in the case of the second embodiment can be obtained, but in addition to this, it is possible to easily determine who the face area is by using character information.
[Other Embodiments]
The present invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the scope of the present invention.
(1)
In the above-described embodiment, the display device has been described by taking the operation at the time of recording of the video camera as an example. However, the above-described technique can also be applied to the live view mode at the time of still image shooting and the playback of the recorded video. It is. For example, when it is desired to enlarge a subject of interest during playback, it is necessary to specify the subject to be enlarged. Even in such a case, the present invention is applicable.

In addition, the present invention is applicable to any device provided with an image pickup device other than a video camera, such as a digital camera or a mobile phone equipped with a camera, or a display device such as a television, a car navigation system, or a game machine. is there.
(2)
In the above-described embodiment, it is assumed that one main subject area is selected, but a plurality of main subject areas may be selectable.
(3)
In the above-described embodiment, the main subject area is selected using the touch panel. However, the main subject area may be selected using another type of operation unit. For example, the main subject area may be selected using a cursor key that can indicate a direction. Even in this case, the main subject candidate whose position relationship is always changed is not directly selected, but the main candidate images arranged on the screen are selected, so that a desired main subject region can be surely selected. .

(4)
In the above-described embodiment, the candidate image, the registered image, and the character information are displayed superimposed on the display image. However, different display methods may be used. For example, the candidate image may be made translucent so that the display image can be seen through even in the region where the candidate image is displayed. Alternatively, the display image may be reduced so that the display image does not overlap the candidate image, and the candidate image and the display image may be displayed on the display unit 160 without being superimposed.
(5)
In the above-described embodiment, a human face is detected as a candidate area. However, there is no problem even if a human body, a pet, or the like is detected by the detection unit 130 and set as a candidate area. Alternatively, not only a specific object but also a moving region obtained by taking a difference between input images or a region including a specific color component is detected by the detection unit 130 and configured as a candidate region. Also good. When a candidate region is detected by taking a difference between images, the detection unit 130 recognizes a moving subject by comparing two image data acquired at different timings, and has a specific visual feature. Candidate areas are detected.

(6)
In the above-described embodiment, the system control unit 157 and the detection unit 130 are configured separately, but the processing performed by the detection unit 130 may be performed by the system control unit 157. The program is stored in the ROM, but the program may be read from an external recording medium.
(7)
The area detected by the detection unit 130 is not limited to an area representing a human face. For example, the detection unit 130 may detect an area representing all or part of a human body, or may be an area including a specific color component.
(8)
In the above-described embodiment, the generation unit 1620 performs the enlargement process and the reduction process on the candidate image and the registered image in accordance with the size of the partitioned areas 410a to 410d, but the candidate image and the registered image before the processing are the partitioned area 410a. As long as it falls within the range of ~ 410d, a configuration in which enlargement processing is not performed may be used.

Further, although the candidate display area 410 is arranged on the lower side of the screen, the arrangement of the candidate display area 410 is not limited to the above-described embodiment.
(9)
In the first embodiment described above, the first decorative frame 411a and the second decorative frame 411b are added to the candidate image. However, as shown in FIG. 17, the decorative elements such as the first decorative frame 411a and the second decorative frame 411b are used. An embodiment without a frame is also conceivable. Only the detection frame of the main subject region may have a visual feature different from the other detection frames so that the face region selected as the main subject region can be easily identified.
The first detection frame 430a and the first decoration frame 411a have the same visual characteristics. However, if it can be recognized that both the frames 430a and 411a correspond, the substantially same visual characteristics are obtained. You may have it. For example, in FIG. 4, the first decorative frame 411a and the second decorative frame 411b have different broken line intervals, but the frame color may be changed.

(10)
In the above-described embodiment, the function of tracking the face area based on the coordinate information of the face area is realized. However, the movement amount of the face area is stored as a motion vector, and there are a plurality of candidate images. You may use the method of raising the determination precision in a case. Further, since it is considered that the size of the face does not change greatly between the image data, it may be added as a condition that the amount of change in the size of the face area is within a certain range between the image data.
(11)
The display example shown in FIG. 16 is an example in the present embodiment. For example, the registered image data may be displayed on the display unit 160 together with character information. That is, an embodiment in which the second embodiment and the third embodiment are combined is also conceivable.

(12)
In the flowchart shown in FIG. 5, if it is determined in step S320 that the face area has disappeared from the screen, information regarding the face area determined to have disappeared from the candidate information list in step S330 is immediately deleted. However, the information may be deleted after a predetermined time has elapsed. In this case, even if a short-time occlusion or frame-out occurs in the face area, the candidate images displayed in the partition areas 410a to 410d do not change. It is possible to avoid a situation in which the display content in 410 changes and selection of the main subject area fails. In addition, since a main subject area that does not exist in the screen can be selected, when the main subject enters the screen, automatic focusing control and automatic exposure control are quickly performed on the main subject area. An effect is also obtained.

(13)
In the flowchart shown in FIG. 7, when it is determined that the main subject does not exist in the screen, normal automatic focusing control and automatic exposure control are performed immediately. Focus control and automatic exposure control may be performed. Various control methods of automatic focusing control and automatic exposure control performed until a predetermined time elapses are conceivable. For example, automatic focusing control and automatic exposure control may be stopped. Automatic focus control and automatic exposure control do not follow the short occlusion and frame out of the main subject, making it difficult for screen blur and screen brightness to change suddenly, resulting in good shot images Can be obtained.
(14)
In the above-described embodiment, the display order of the candidate images is the order of the candidate information list. However, the display position of the candidate images may be determined in accordance with the position of the face area. For example, in the display example shown in FIG. 4, the candidate image of the left face 400b is displayed in the partitioned area 410b, and the candidate image of the right face 400a is displayed in the partitioned area 410a on the left side of the partitioned area 410b. The candidate image of the face 400b may be displayed in the partitioned area 410a and the candidate image of the face 400a may be displayed in the partitioned area 410b in accordance with the arrangement of the faces 400a and 400b. In this case, since the positional relationship between the face area and the positional relationship between the candidate images correspond, the user can more easily understand the correspondence between the candidate image and the face area.

(15)
In the second embodiment described above, even if an unregistered face is detected, the candidate image data is not displayed in the candidate display area 410. However, as in the first embodiment, an unregistered face is detected. Alternatively, candidate image data may be generated from the detection image data and displayed in the candidate display area 410. That is, there may be a configuration in which the first embodiment and the second embodiment are combined.
In addition, when registering registration information, a registration feature amount for recognition processing is not stored in the recording medium 155 but is stored in a non-volatile memory (not shown) provided in the video camera 500. Also good. In this case, even if the recording medium 155 is replaced with another recording medium, the registration information can be read from the nonvolatile memory, and the convenience of the device is improved. As the non-volatile memory, a flash memory and a hard disk drive can be considered.

(16)
Each functional block may be individually made into one chip by a semiconductor device such as an LSI, or may be made into one chip so as to include a part or the whole.
Here, although LSI is used, it may be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.
Further, the method of circuit integration is not limited to LSI, and implementation with a dedicated circuit or a general-purpose processor is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.
Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied as a possibility.

(17)
Each process of the above-described embodiment may be realized by hardware, may be realized by software, or may be realized by a mixed process of software and hardware.

  The display device according to the present invention makes it easy to specify a main subject area, and thus the present invention is useful in the field of an apparatus that performs focus control and exposure control for a specific area on an image. is there.

DESCRIPTION OF SYMBOLS 100 Video camera 130 Detection part 157 System control part 160 Display part 180 Operation part 220 Face detection part 700 Recognition part 1620 List management part 1620 Generation part 1640 Composition part 1650 Selection part 1660 Registration part

Claims (24)

A display device for displaying a plurality of image data as video,
A detection unit for detecting at least one candidate region having a specific visual feature from the image data;
A generating unit that generates at least one candidate identification information for identifying each candidate area detected by the detecting unit;
A combining unit that generates combined image data in which the candidate identification information and the image data are combined;
A display unit for displaying the composite image data as the video;
An operation unit for receiving input of designation information for designating specific candidate identification information among the candidate identification information appearing in the video;
Based on the designation information input to the operation unit, a selection unit that selects the candidate region corresponding to the candidate identification information designated through the operation unit as a reference region;
A display device comprising: The combining unit combines the candidate identification information and the image data so that the candidate identification information is arranged at a specific position on the screen of the display unit.
The display device according to claim 1. The combining unit combines the candidate identification information and the image data so that a plurality of the candidate identification information are arranged on the screen of the display unit.
The display device according to claim 1. The synthesizing unit synthesizes the candidate identification information and the image data so that the candidate identification information is arranged in a lower half region in the vertical direction of the video displayed on the display unit;
The display device according to claim 1. The composition unit combines the candidate identification information and the image data so that the candidate identification information is superimposed on an image generated based on the image data in a state where the video is displayed on the display unit. To synthesize,
The display device according to claim 1. The area of the first display area where the candidate identification information is displayed on the screen of the display unit is smaller than the area of the second display area other than the first display area,
The display device according to claim 1. The generation unit extracts candidate image data in each candidate area detected from one image data, and sets the extracted candidate image data as the candidate identification information.
7. The display device according to any one of 1 to 6. The generation unit compares the size of the partition area in which one candidate image is displayed as the candidate identification information with the size of the candidate image, and adjusts the size of the candidate image based on a comparison result.
The display device according to claim 7. When the candidate image is smaller than the partition area, the generation unit performs an enlargement process on the candidate image according to the partition area, and sets the candidate image subjected to the enlargement process as the candidate identification information ,
The display device according to claim 8. When the candidate image is larger than the partition area, the generation unit performs a reduction process on the candidate image according to the partition area, and sets the candidate image subjected to the reduction process as the candidate identification information ,
The display device according to claim 8 or 9. When combining the candidate identification information and the image data, the combining unit adds first decoration information indicating the candidate area to each candidate area, and the candidate identification information corresponding to the candidate area 2nd decoration information indicating that is added to each area where the candidate identification information is displayed,
The display device according to claim 1. The first decoration information added to the reference area has a visual feature different from the other first decoration information,
The second decoration information added to the candidate identification information corresponding to the reference area is a visual feature different from the other second decoration information and is substantially the same as the corresponding first decoration information. Have visual features,
The display device according to claim 11. The detection unit compares reference information representing the specific visual feature with the image data, and detects a region that satisfies the criteria information as the candidate region.
The display device according to claim 1. The candidate region having the specific visual feature is a region representing all or part of a human body;
The display device according to claim 1. The candidate area having the specific visual feature is an area representing a human face;
The display device according to claim 1. The candidate region having the specific visual feature is a region including a specific color component;
The display device according to claim 1. The detection unit detects the candidate region having the specific visual feature by comparing two image data acquired at different timings.
The display device according to claim 1. The designation information indicates a position on the screen of the display unit,
The selection unit identifies the candidate identification information displayed at the position indicated by the designation information based on the designation information input via the operation unit, and corresponds to the identified candidate identification information Selecting the candidate region as the reference region;
The display device according to claim 1. The operation unit includes a touch panel unit that detects a contact position on the screen of the display unit and generates the designation information indicating the contact position.
The display device according to claim 1. The detection unit generates a detection feature amount indicating a visual feature of the candidate area for each detected candidate area.
The display device according to claim 1. A registration unit that associates the detected feature amount generated by the detection unit with the candidate identification information corresponding to the candidate region from which the detected feature amount is generated;
A registered information acquisition unit that matches the detected feature value with the registered feature value and identifies the candidate identification information associated with the registered feature value that matches the detected feature value;
The synthesizing unit synthesizes the candidate identification information identified by the registration information acquisition unit and the image data to generate the synthesized image data;
The display device according to claim 20. The operation unit is adapted to accept input of character information,
The registration unit associates the character information with the registered feature amount;
The display device according to claim 21. A display method for displaying a plurality of image data as video,
Detecting at least one candidate region having a specific visual feature from the image data;
Generating at least one candidate identification information for identifying each candidate area detected in the detecting step;
Generating a composite image data in which the candidate identification information and the image data are combined;
A display step of displaying the composite image data as the video;
An operation step of receiving input of designation information for designating the specific candidate identification information among the candidate identification information appearing in the video;
A selection step of selecting, as a reference area, the candidate area corresponding to the candidate identification information specified via the operation unit based on the designation information input to the operation unit;
Display method with. An integrated circuit for displaying a plurality of image data as video,
A detection unit for detecting at least one candidate region having a specific visual feature from the image data;
A generating unit that generates at least one candidate identification information for identifying each candidate area detected by the detecting unit;
A combining unit that generates combined image data in which the candidate identification information and the image data are combined;
A display unit for displaying the composite image data as the video;
An operation unit for receiving input of designation information for designating specific candidate identification information among the candidate identification information appearing in the video;
Based on the designation information input to the operation unit, a selection unit that selects the candidate region corresponding to the candidate identification information designated through the operation unit as a reference region;
Integrated circuit with.

Images