JP2022135674A - Electronic apparatus, information processing apparatus, control method, learning method, and program - Google Patents

Abstract

To appropriately discriminate a main object person and a person having a conversation with the main object person from a photographed image.SOLUTION: An electronic apparatus has: person detection means that detects persons from a picked-up image; main object selection means that selects a person to be a main object from the persons detected by the person detection means; person motion detection means that detects motions of the persons detected by the person detection means; person motion counting means that counts the number of times of the motions detected by the person motion detection means; and grouping means that, when the number of times of motions of a predetermined person detected within a predetermined first time from when the utterance of the main object person is detected reaches a predetermined number of times within the predetermined first time, groups the predetermined person as a person having a conversation with the main object.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for recording conversational voices of people when shooting.

When recording conversational voices of people, such as when shooting movies, the voices of people may be recorded at a volume that is difficult to hear depending on the shooting environment. Techniques for emphasizing and recording are known. As a technique for discriminating a person who is having a conversation, Patent Document 1 proposes a method for recognizing a person who is a speaker based on characteristics of lip movements and voice. In addition, Japanese Patent Laid-Open No. 2002-200002 proposes a method of grouping people in an image based on the directions and positions of two faces that are adjacent to each other.

JP 2010-192956 A JP 2015-056701 A

However, in Japanese Patent Application Laid-Open No. 2002-200013, when recording conversational voices of people at the time of photographing, the spoken voices of people who are not interested in the user among the people who are having a conversation are also recorded. The utterance voice that the user is interested in is the conversation voice of the person who is speaking (the main target) and the conversation voice of the person who is the conversation partner of the main target, and the utterance voices of other people are recorded. Further, in Patent Document 2, if the orientation of the face cannot be specified, the person who is having a conversation cannot be properly identified.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object thereof is to realize a technique capable of appropriately distinguishing between a main target person and a person conversing with the main target person from a photographed image.

In order to solve the above problems and achieve the object, an electronic device according to the present invention includes person detection means for detecting a person from a captured image; a main target selecting means for selecting a person to be a person; a human motion detecting means for detecting the human motion detected by said human motion detecting means; and a human motion counting means for counting the number of motions detected by said human motion detecting means. and, when the number of actions of a predetermined person detected within a predetermined first time after the utterance of the main target person is detected reaches a predetermined number of times within the predetermined first time, the predetermined and a grouping means for grouping the person as a person who is conversing with the main subject.

According to the present invention, a main target person and a person conversing with the main target person can be appropriately discriminated from a photographed image.

FIG. 2 is a block diagram showing the device configuration of the embodiment; FIG. 2 is a block diagram showing the configurations of an imaging processing unit and an audio processing unit according to the embodiment; Explanatory drawing of the main target selection method of this embodiment. FIG. 2 is an explanatory diagram of an assumed scene 1 of the first embodiment; 4 is a flow chart showing processing of assumed scene 1 and assumed scene 3 of the first embodiment. FIG. 4 is an explanatory diagram of an assumed scene 2 of the first embodiment; 4 is a flowchart showing processing of assumed scene 2 of the first embodiment; FIG. 3 is an explanatory diagram of an assumed scene 3 of the first embodiment; FIG. 4 is an explanatory diagram of an assumed scene 4 of the first embodiment; 10 is a flowchart showing processing of assumed scene 4 according to the first embodiment; FIG. 8 is a hardware configuration diagram of a server that executes machine learning according to the second embodiment; FIG. 10 is a software configuration diagram for executing estimation processing of the imaging device and learning processing of the server according to the second embodiment; FIG. 10 is a conceptual diagram of a trained model of Embodiment 2; 9 is a flowchart showing learning processing according to the second embodiment; 9 is a flowchart showing estimation processing according to the second embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

[Embodiment 1]
Embodiments in which the electronic device of the present invention is applied to an imaging device such as a digital camera will be described below.

First, with reference to FIGS. 1 to 3, the configuration and functions of the audio processing device included in the imaging device of this embodiment will be described.

FIG. 1 is a block diagram showing the configuration of an imaging device 100 according to Embodiment 1. As shown in FIG.

The imaging unit 101 includes a lens unit that forms an imaging optical system, an image sensor that converts an optical image of a subject formed by the lens unit into an electrical signal, an A/D conversion unit, and the like. The imaging unit 101 converts an analog image signal generated by capturing an optical image of a subject into a digital signal, and outputs the digital signal to the image processing unit 102 . The image processing unit 102 performs various image processing on the digital signal generated by the imaging unit 101 to generate image data. Note that the lens unit may be a lens built into the imaging device 100 or a detachable lens.

The audio input unit 103 collects sounds around the imaging device 100 using one or a plurality of microphones built into the imaging device 100 or connected via an audio terminal, and converts the sounds into electrical signals. The audio input unit 103 converts analog audio signals generated by collecting sounds around the imaging device 100 into digital signals and outputs the digital signals to the audio processing unit 104 . The audio processing unit 104 performs various audio processing on the digital signal generated by the audio input unit 103 to generate audio data. The microphones may be directional or non-directional.

A memory 105 temporarily stores image data generated by the image processing unit 102 and audio data generated by the audio processing unit 104 . Also, the memory 105 is used as a work memory that stores programs executed by the control unit 111, parameters, and the like.

The display control unit 106 displays the video of the image data generated by the image processing unit 102, the operation screen of the imaging device 100, the menu screen, etc. on the display unit 107 or an external display via a video terminal (not shown). . The display unit 107 has a touch panel function, and by detecting a touch operation by the user, setting operation of the imaging apparatus 100, menu screen operation, subject selection operation, and the like are possible.

The encoding processing unit 108 reads image data and audio data temporarily stored in the memory 105 and performs predetermined encoding to generate compressed image data, compressed audio data, and the like. Note that the audio data may not be compressed. Compressed image data is, for example, MPEG2 or H.264. 264/MPEG4-AVC or any other compression method. Also, the compressed audio data may be compressed by a compression method such as AC3(A)AC, ATRAC, ADPCM, or the like.

The recording/reproducing unit 109 records various data such as compressed image data, compressed audio data, or audio data generated by the encoding processing unit 108 on the recording medium 110 , or records them on the recording medium 110 . and read various data that has been stored. The recording medium 110 includes all recording media such as magnetic disks, optical disks, semiconductor memories, etc., which can record image data, audio data, and the like.

The control unit 111 includes an arithmetic processing unit such as a CPU for executing various controls of the imaging apparatus 100 by controlling each component of the imaging apparatus 100, and a memory such as a ROM for storing programs for executing various controls. It includes an external storage device connected to the device and imaging device 100 .

The operation unit 112 includes operation members such as buttons and dials that accept user operations, and outputs operation signals corresponding to user operations to the control unit 111 . In this embodiment, the operation unit 112 includes a power button for instructing power on/off, a shooting button for instructing start/end of moving image recording, and optically or electronically zooming an image. Including a zoom lever for pointing. Further, the operation unit 112 includes a mode switch for switching operation modes, a cross key for performing various settings, an enter key, and the like.

The audio output unit 113 outputs audio data or compressed audio data reproduced by the recording/reproducing unit 109 or audio data output by the control unit 111 to the speaker 114, an audio terminal, or the like. The external output unit 115 outputs the compressed video data, compressed audio data, audio data, etc. reproduced by the recording/reproducing unit 109 to an external device. A data bus 116 transmits and receives various data such as audio data, image data, and control signals between the components of the imaging apparatus 100 .

Next, functions of the imaging device 100 of this embodiment will be described.

In the imaging apparatus 100 of the present embodiment, when the user operates the power supply included in the operation unit 112 to give an instruction to turn on the power, a power supply unit (not shown) supplies power to each component of the imaging apparatus 100 . be done.

When power is supplied, the control unit 111 determines the operation mode based on the operation signal of the mode changeover switch included in the operation unit 112 . Operation modes include, for example, a still image shooting mode, a moving image recording mode, a playback mode, and the like. In the moving image recording mode, image data captured by the imaging unit 101 and generated by the image processing unit 102 and audio data collected by the audio input unit 103 and generated by the audio processing unit 104 are recorded as one file. In the playback mode, the compressed image data recorded on the recording medium 110 is played back by the recording/playback unit 109 and displayed on the display unit 107 .

In the moving image recording mode, the control unit 111 transmits a control signal to each component of the imaging device 100 so as to shift to the shooting standby state, and each component performs the following operations. The image processing unit 102 performs various types of image processing on the digital image signal generated by the imaging unit 101 to generate image data, and the display control unit 106 displays an image (live view) of the image data on the display unit 107. indicate. The user can prepare for photographing such as automatic focus adjustment (AF) processing and automatic exposure (AE) processing while viewing the live view image. Further, the audio processing unit 104 performs various audio processing on the digital audio signal generated by the audio input unit 103 to generate audio data, and the audio output unit 113 outputs the audio data to the speaker 114 or an earphone (not shown). Output from The user can make preparations for recording such as adjusting the volume while listening to the voice.

Next, when the user operates the shooting button included in the operation unit 112 to give an instruction to start shooting, the control unit 111 sends a control signal instructing each component of the imaging device 100 to start shooting. , and each component operates as follows. Image data generated by the imaging unit 101 and the image processing unit 102 is temporarily stored in the memory 105 . Audio data generated by the audio input unit 103 and the audio processing unit 104 (when there is one microphone) or multi-channel audio data (when there are multiple microphones) is temporarily stored in the memory 105 .

The encoding processing unit 108 reads image data and audio data temporarily stored in the memory 105 and performs predetermined encoding processing to generate compressed image data, compressed audio data, and the like. The control unit 111 synthesizes the compressed image data and compressed audio data generated by the encoding processing unit 108 , forms a data stream, and outputs the data stream to the recording/playback unit 109 . When the audio data is not compressed, the control unit 111 synthesizes the audio data stored in the memory 105 and the compressed image data, forms a data stream, and outputs the data stream to the recording/playback unit 109 .

The recording/playback unit 109 writes the data stream as one moving image file to the recording medium 110 based on a predetermined file system such as UDF or FAT. The control unit 111 continues such operations during shooting.

Next, when the user operates the shooting button included in the operation unit 112 to give an instruction to end shooting, the control unit 111 transmits a control signal instructing each component of the imaging device 100 to stop shooting, Each component operates as follows.

The imaging unit 101, the image processing unit 102, the audio input unit 103, and the audio processing unit 104 stop generating image data and audio data, respectively. The encoding processing unit 108 reads the remaining image data and audio data stored in the memory 105, performs predetermined encoding processing, and stops processing after generating compressed image data, compressed audio data, and the like. If the audio data is not to be compressed, the processing is stopped after the generation of the compressed image data is completed. The control unit 111 synthesizes the finally generated compressed image data with the compressed audio data or audio data, forms a data stream, and outputs the data stream to the recording/playback unit 109 .

The recording/reproducing unit 109 writes the data stream as one moving image file to the recording medium 110 based on a predetermined file system such as UDF or FAT, and completes the moving image file when the supply of the data stream is stopped. Stop.

When the recording/reproducing unit 109 stops processing, the control unit 111 transmits a control signal to each component of the imaging apparatus 100 so as to shift to the shooting standby state, and returns to the shooting standby state.

Next, when the user operates the mode changeover switch included in the operation unit 112 to give an instruction to switch to the playback mode, the control unit 111 sends a control signal instructing each component of the imaging device 100 to transition to the playback state. , and each component operates as follows. The recording/reproducing unit 109 reads a moving image file composed of compressed image data and compressed audio data recorded on the recording medium 110 and transmits it to the encoding processing unit 108 . The encoding processing unit 108 decodes the compressed image data and the compressed audio data received from the recording/playback unit 109 and transmits the decoded data to the display control unit 106 and the audio output unit 113 . Display control unit 106 displays the decoded image data on display unit 107 . The audio output unit 113 outputs the decoded audio data from an internal speaker or an external speaker connected to an audio terminal.

In this embodiment, when the audio input unit 103 or the audio processing unit 104 generates an audio signal or audio data, level adjustment processing is performed on the audio signal collected by the microphone. The level adjustment process may be always performed after the imaging apparatus 100 is activated, may be performed after the shooting mode is selected, or may be performed after a mode related to audio recording is selected. may Alternatively, in a mode related to audio recording, it may be performed in response to the start of audio recording. In this embodiment, it is assumed that processing is performed at the timing when shooting of a moving image is started.

As described above, the imaging apparatus 100 of this embodiment can record and reproduce images and sounds.

Next, detailed configurations and functions of the imaging unit 101, the image processing unit 102, the audio input unit 103, and the audio processing unit 104 will be described with reference to FIG.

The imaging unit 101 includes a lens unit 201 that forms an imaging optical system, and an image sensor 202 that includes a photoelectric conversion element such as a CCD or CMOS that converts an optical image of a subject captured by the lens unit 201 into an electrical signal. The imaging unit 101 also includes a lens control unit 203 having a driving mechanism such as a position sensor and a motor for driving the lens unit 201 . In this embodiment, the imaging unit 101 incorporates the lens unit 201 and the lens control unit 203 , but a lens that is detachable from the imaging apparatus 100 may be used. For example, when the user operates the operation unit 112 to input an instruction such as zoom operation or focus adjustment, the control unit 111 transmits a control signal (driving signal) for moving the lens to the lens control unit 203 . The lens control unit 203 confirms the position of the lens unit 201 with a position sensor and moves the lens unit 201 with a motor or the like according to the control signal.

The image processing unit 102 includes an image adjustment unit 221 that performs various image quality adjustment processes on image signals generated by the image sensor 202 . Image data that has undergone image adjustment processing is transmitted to memory 105 via data bus 116 . The control unit 111 performs AF processing, AE processing, and the like using image data that has been subjected to image quality adjustment processing by the image adjustment unit 221 . Also, the image processing unit 102 includes various functional units to be described later. The person detection unit 222 extracts feature points of a person's face, such as eyes, nose, and mouth, from the image data subjected to image quality adjustment processing by the image adjustment unit 221, and determines the position and size of the person's face in the image data. and so on. In addition, the person detection unit 222 detects utterances, movements of the body such as lips, head, and hands in utterances, movements of the line of sight and the face, affirmative actions of shaking the face vertically or nodding, and negative actions of twisting or shaking the face sideways. to detect The human motion detection unit 223 detects changes in facial expressions such as emotions. The human motion counting unit 224 counts the number of times the human motion detecting unit 223 has detected. The main target selection unit 225 selects a person as a main target from the persons detected by the person detection unit 222 . Selection by the main target selection unit 225 is performed based on conditions determined by the control unit 111 . A method of selecting a main target by the main target selection unit 225 will be described later.

Furthermore, the image processing unit 102 has a conversation grouping unit 226 . The conversation grouping unit 226 determines a person who is having a conversation with the main target person selected by the main target selection unit 225 from the persons detected by the person detection unit 222, and groups them as a main target conversation group. conduct. The details of the processing by conversation grouping unit 226 will be described later.

Next, the voice input section 103 and the voice processing section 104 will be described.

The audio input unit 103 includes a microphone 211 that converts audio vibrations into electrical signals and outputs them as audio signals. In this embodiment, the microphone 211 is of a stereo system configured with left and right Lch/Rch channels, but may be of a single channel monaural system or may be configured to hold a plurality of microphones of two or more channels. The A/D converter 212 converts the analog audio signal generated by the microphone 211 into a digital audio signal.

The audio processing unit 104 is a circuit that performs various audio processing on the audio signal converted by the audio input unit 103 . In this embodiment, the audio processor 104 has an audio extractor 213 , an audio adjuster 215 and an audio synthesizer 217 . The voice extraction unit 213 can separate and extract human voices and other voices (non-human voices). It is possible to separate the voice of each person into the voice of each person. The audio adjustment unit 215 can individually perform audio processing such as level adjustment and an equalizer on the audio separated and extracted by the audio extraction unit 213 . Based on the result of determination by the conversation grouping unit 226, the conversation voice adjustment unit 216 performs adjustment processing for emphasizing the separated and extracted voice to make it easier to hear, or making it less audible and more conservative. Details of the adjustment processing by the conversation voice adjustment unit 216 will be described later. The voice synthesizing unit 217 synthesizes the individual separated and extracted voices adjusted by the voice adjusting unit 215, and restores them to one voice signal again. An ALC (Auto Level Controller) 219 adjusts the amplitude of the audio signal synthesized by the audio synthesizing section 217 to a predetermined level.

With the above configuration, the audio processing unit 104 performs predetermined processing on the audio signal and transmits the generated audio data to the memory 105 .

In addition, as described below, the main subject selection unit 225 uses image data and audio data temporarily stored in the memory 105 to select a target person for audio processing using a series of recorded image information. It becomes possible to select.

Next, a method of selecting a main target by the main target selection unit 225 will be described with reference to FIG.

Since the main target in this embodiment indicates a person to whom the user pays attention, various methods are conceivable for the selection condition. For example, in the case of FIG. 3A, the person 301 having a conversation corresponds to the subject to which the AF frame 302 of the imaging apparatus 100 is aligned, and the AF frame 302 and the person 301 having a conversation match. , the imaging apparatus 100 determines that the subject with which the AF frame 302 is aligned is the main subject. The main subject can be selected in this way.

There is also a method using a pre-registered face image as shown in FIG. 3(b). An affirmative action of shaking the face vertically or nodding and a negative action of twisting or shaking the face sideways are registered in the memory 105 in advance as the face image 303, and a person who matches the face image 303 may be selected as the main target. can.

There is also a method of determining the main object according to the user's intention, as shown in FIG. 3(c). When the display unit 107 has a touch panel function, the user can also select a main target by performing a touch operation 304 on a desired person in the displayed image.

There is also a method of selecting a main target from recorded moving image data as shown in FIG. 3(d). For example, the person 305 who appears most frequently in the recorded moving image data 306 and has the highest frequency of focus can be selected as the main subject.

<Grouping Processing> Next, the grouping processing by the image processing unit 102 of the present embodiment will be described with reference to FIGS. 4 to 10. FIG.

5, 7 and 10, the CPU of the control unit 111 or the like executes a program stored in a storage device such as a ROM, and each component of the image processing unit 102 shown in FIG. It is realized by controlling.

<Assumed Scene 1> First, processing in the assumed scene 1 of the present embodiment will be described with reference to FIGS. 4 and 5. FIG.

FIG. 4 is an explanatory diagram of an assumed scene 1 of this embodiment. FIG. 5 is a flow chart showing the processing of the assumed scene 1. As shown in FIG. Below, the flow of processing in FIG. 5 will be described along the assumed scene 1 illustrated in FIG.

FIG. 4A illustrates a state in which person A401, person B402, and person C403 are having a conversation as a first group, and person D404 and person E405 are having a conversation as a second group. In FIG. 4A, it is assumed that the person whom the user wants to be the main target is person A401.

S501: Selection of Main Subject The main subject selection unit 225 selects the person A401 as the main subject by the main subject selection method described above. The user's intention is to mainly select a person to be focused on as the main object.

S502: Detection of Movement of Main Subject When the person A401 is selected as the main subject and the person A401 as the main subject speaks as shown in FIG. Detects that A401 has spoken.

S503: Detection of motion of person other than main target The human motion detection unit 223 detects the motion of a person other than the main target in the image within a predetermined first time after the person A401, who is the main target, speaks. determine whether or not For example, within a predetermined first time period after the main target person A401 speaks, as shown in FIG. The unit 223 detects that the person B402 turned his or her line of sight or face toward the main target person A401.

If the human motion detection unit 223 determines that the person in the image has performed the motion within the predetermined first time period after the person A401, who is the main target, has spoken, the process proceeds to S504. If the human motion detection unit 223 determines that the person in the image does not move within the predetermined first period of time after the main target person A401 speaks, the process returns to S502. .

S504, S505: Identifying Person, Counting Number of Actions The human action detection unit 223 directs the line of sight or face to the main target person A401 within a predetermined first time after the main target person A401 speaks. Person B402 is identified as the person who Then, the human motion counting unit 224 counts the number of motions of the human B402 to 1 and starts counting a predetermined first time.

S506-S508: Determining the number of actions of persons other than the main target, grouping After that, within a predetermined first time after the person A401 who is the main target speaks, as shown in FIG. 4D, the person B402 For example, when the person B402 makes an affirmative action, the person action detection unit 223 determines that the person B402 has made an affirmative action. Then, the person detection unit 222 detects that the person who made the affirmative action is person B402, who is the same person as the person detected in S503. up to 2.

The human motion counting unit 224 determines whether the counted number of motions of the person B402 who has made a motion within a predetermined first period of time after the person A401, who is the main target, has reached a predetermined number of times (for example, three times). determine whether or not When the human motion counting unit 224 determines that the counted number of motions of the person B402 has reached the predetermined number (YES in S506), the process proceeds to S507. If the human motion counting unit 224 determines that the counted number of motions of the person B402 has not reached the predetermined number (NO in S506), the process returns to S502 to continue detecting the motion of the person B402.

When returning to S502, when the person B402 speaks as shown in FIG. A speech is detected (YES in S503), the person B402 is specified in S504, and the person movement counting unit 224 increases the count number of the number of movements of the person B402 to 3 in S505.

After that, in S506, the human motion counting unit 224 determines whether or not the counted number of motions of the person B402 who has made a motion within a predetermined first period of time after the person A401, who is the main target, has reached a predetermined number of times. determine whether When the human motion counting unit 224 determines that the counted number of motions of the person B402 has reached the predetermined number (YES in S506), the process proceeds to S507.

In S507, the conversation grouping unit 226 determines that the person B402 is having a conversation with the main target person A401, and in S508 groups the person B402 into the first group of the main target person A401. I do.

As described above, the grouping is performed for the person who is having a conversation with the person A401 who is the main target.

<Assumed Scene 2> Next, processing in the assumed scene 2 of the present embodiment will be described with reference to FIGS. 6 and 7. FIG.

FIG. 6 is an explanatory diagram of an assumed scene 2 of this embodiment. FIG. 7 is a flow chart showing the processing of the assumed scene 2. As shown in FIG. Below, the flow of processing in FIG. 7 will be described along with assumed scene 2 illustrated in FIG.

In FIG. 6A, as in FIG. 3A, person A401, person B402, and person C403 are having a conversation as a first group, and person D404 and person E405 are having a conversation as a second group. exemplifies the state. It is assumed that the person A401 and the person B402, who are the main objects, have already been grouped.

S701: Detect main target and grouped person The conversation grouping unit 226 detects the main target, person A401, and grouped person B402, as shown in FIG. 6(a).

S702: Detecting Action of Predetermined Person Other than Main Subject in Group of Main Subjects As shown in FIG. The detection unit 223 detects that a person B402 who is not the main target and who has been grouped with the person A401 who is the main target speaks.

S703: Detecting motions of persons other than a person who is the main target and other than a predetermined person in the main target group, and counting the number of motions As shown in FIG. When a person C403 who is not in the group of the person A401 who is the main object turns his/her gaze or face toward the person B402 within a predetermined first time after the person B402 who is not the object speaks, the person action detection unit 223 It is detected that the person C403 turned his or her line of sight or face toward the person A401, who is the main object, and the person B402, which is not the main object.

S704, S705: Identifying Persons, Counting the Number of Actions In S704, the human movement detection unit 223 identifies the person C403 as the person who directed his gaze and face to the person A401 who is the main target and the person B402 who is not the main target. . In S705, the human motion counting unit 224 counts the number of motions of the human C403 to 1 and starts counting a predetermined first time.

S706-S708: Determining the number of actions of the same person, grouping As shown in FIG. If the person C403 performs, for example, an affirmative action within this period, the human action detection unit 223 determines that the person C403 has made an affirmative action. The person detection unit 222 detects that the person who made the affirmative action is the person C403 who is the same person as the person detected in S503. Up.

The human motion counting unit 224 counts the number of motions of the person C 403 who has made a motion within a predetermined first time period after the main target person A 401 and the non-main target person B 402 uttered. , three times). When the human motion counting unit 224 determines that the counted number of motions of the person C403 has not reached the predetermined number (NO in S706), the process returns to S702 to continue motion detection of the person C403.

In the case of returning to S702, within a predetermined first time after the person B402, who is not the main object and who has been grouped with the person A401 who is the main object, speaks, in the main object group, as shown in FIG. In S704, the human motion detection unit 223 detects that the person C403 has spoken. up to.

The human motion counting unit 224 counts the number of motions of the person C403, who has been grouped with the main target person A401 and has made a motion within a predetermined first time after the person B402, who is not the main target, has spoken. Determine whether or not the number of times has been reached. When the human motion counting unit 224 determines that the counted number of motions of the person C403 has reached the predetermined number (YES in S706), the process proceeds to S707. In S707, the conversation grouping unit 226 determines that the person C403 is grouped with the main target person A401 and is having a conversation with the non-main target person B402. In S708, the conversation grouping unit 226 groups the person C403 into the first group of the main target person A401.

As described above, the grouping is performed for the person who is already grouped with the person A401 who is the main target and is having a conversation with the person B402 who is not the main target.

<Assumed Scene 3> Next, processing in the assumed scene 3 of the present embodiment will be described with reference to FIGS. 5 and 8. FIG.

FIG. 8 is an explanatory diagram of an assumed scene 3 of this embodiment. The processing flow of the assumed scene 3 of the third embodiment is the same as that shown in FIG. Below, the flow of processing in FIG. 5 will be described along with assumed scene 3 illustrated in FIG.

In FIG. 8A, person A401, person B402, and person C403 are having a conversation as a first group, and person D404 and person E405 are having a conversation as a second group. The person whom the user wants to be the main target is person A401.

S501: Selection of Main Subject The main subject selection unit 225 selects the person A401 as the main subject. The user's intention is to mainly select a person to be focused on as a main target.

S502: Detection of Action of Main Subject When the person A401, who is the main subject, speaks in a state in which the person A401 is selected as the main subject, as shown in FIG. Detects that A401 has spoken.

S503: Detection of motion of person other than main target The human motion detection unit 223 detects whether a person other than the main target in the image has made a motion within a predetermined first time after the person A401, who is the main target, has spoken. determine whether or not For example, if a person D404, who is not in conversation with the main target person A401, speaks within a predetermined first period of time after the main target person A401 speaks, as shown in FIG. The human motion detection unit 223 detects that the person D404 has spoken.

If the human motion detection unit 223 determines that the person D404 who is not speaking with the main target person A401 speaks within a predetermined first time after the main target person A401 speaks, , the process proceeds to S504. In addition, the human motion detection unit 223 determines that the person D404, who is not conversing with the main target person A401, has not spoken within a predetermined first time period since the main target person A401 has spoken. If so, the process returns to S502.

S504-S505: Identification of Person, Counting of Number of Actions In S504, the person detection unit 222 identifies person D404 as a person who has spoken within a predetermined first period of time after person A401, who is the main target, has spoken. In S505, the human motion counting unit 224 counts the number of motions of the human D404 to 1 and starts counting a predetermined first time.

S506-S508: Determining the Number of Actions of Persons Other than the Main Subject, Grouping Within a predetermined first time after the person A401, who is the main subject, has spoken, the person D404 is the main subject as shown in FIG. 8(d). When the person D404 directs his or her line of sight or face to the person A401 who is the main target, the human motion detection unit 223 determines that the person D404 turns his or her line of sight or face to the person A401, which is the main target. Then, the person detection unit 222 detects that it is the person D404, who is the same person as the person detected in S503, who turned his gaze and face toward the person A401, who is the main object, and the person action counting unit 224 The count number of the number of actions of the person D404 is increased to two.

The human motion counting unit 224 determines whether the counted number of motions of the person D404, who has made a motion within a predetermined first period of time after the person A401, who is the main target, has reached a predetermined number of times (for example, three times). determine whether or not When the human motion counting unit 224 determines that the counted number of motions of the person D404 has reached the predetermined number (YES in S506), the process proceeds to S507. If the human motion counting unit 224 determines that the number of motions of the person D404 has not reached the predetermined number (NO in S506), the process returns to S502 to continue detecting the motion of the person D404.

When the human motion counting unit 224 determines that the counted number of motions of the person D404 has reached the predetermined number (YES in S506), the process proceeds to S507.

In S507, the conversation grouping unit 226 determines that the person D404 is having a conversation with the main target person A401, and in S508 groups the person D404 into the first group of the main target person A401. I do.

When the human motion counting unit 224 determines that the counted number of motions of the person D404 has not reached the predetermined number (NO in S506), for example, as shown in FIG. If the motion of person D404 is not detected within a predetermined first time after A401 speaks, the process returns to S502.

As described above, the grouping is performed for the person who is having a conversation with the person A401 who is the main target.

<Assumed Scene 4> Next, processing in the assumed scene 4 of the present embodiment will be described with reference to FIGS. 9 and 10. FIG.

FIG. 9 is an explanatory diagram of an assumed scene 4 of this embodiment. FIG. 10 is a flow chart showing processing of assumed scene 1 of the fourth embodiment. Below, the flow of processing in FIG. 10 will be described along with assumed scene 4 illustrated in FIG.

In FIG. 9, person A401, person B402, and person C403 are having a conversation as a first group, and person D404 and person E405 are having a conversation as a second group.

S1001: Detect Main Target and Grouped Persons As shown in FIG. 9A, the main targets, Person A401, Person B402, and Person C403, have already been grouped. In this state, as shown in FIG. 9B, the already grouped person C403 is removed from the first group in conversation, and the person D404 and person E405 join the second group in conversation. Suppose you enter

S1002: Detecting a person who has not been grouped with the main target When the main target person A401 or the main target person A401 and the non-main target person B402 who has been grouped speaks, the human motion detection unit 223 It is detected that a certain person A401 or a person B402 who is not the main target has spoken.

S1003 to S1006: Human Action Detection, Grouping Maintenance or Release , determine whether or not a person other than the person A401 who is the main object in the image and the person other than the person B402 who is not the main object has performed an action.

Here, as shown in FIG. 9B, when the person C403 leaves the first group in which the person A401 who is the main subject and the person B402 who is not the main subject are having a conversation, the person A401 who is the main subject or the person B402 who is not the main subject Even if the person B402, who is not the target, speaks, the person C403 does not make a predetermined number of actions in response to it within a predetermined first period of time. Therefore, the conversation grouping unit 226, within a predetermined first period of time after the person A401 who is the main subject or the person B402 who is not the main subject speaks, utters a speech other than the person A401 who is the main subject in the image and the person who is not the main subject in the image. It is determined that the action of the person C403 other than the person B402 is not detected (NO in S1003), and the process proceeds to S1005.

In S1005, the conversation grouping unit 226 selects the main subject in the image within a predetermined second time longer than the predetermined first time after the person A401 who is the main subject or the person B402 who is not the main subject has spoken. It is determined whether or not a person C403 other than the person A401 who is the main target and the person C403 other than the person B402 who is not the main object has performed an action.

In this case, as shown in FIG. 9C, the person C403 is excluded from the first group in which the main object person A401 and the non-main object person B402 are having a conversation. The action of the person C403 is not performed a predetermined number of times even within a predetermined second time period longer than the first predetermined time period with respect to the utterance of the person B402 who is not the object. Therefore, the conversation grouping unit 226 determines whether the person A 401, who is the main subject, or the person B 402, who is not the main subject, has spoken within a predetermined second time period longer than the first predetermined time period. It is determined that the action of a person C403 other than a person A401 and person B402 who is not the main target is not detected (NO in S1005), and the grouping of the person C403 is released in S1006.

In addition, the conversation grouping unit 226 selects the main target person A401 or the non-main target person B402 as the main target in the image within a predetermined second time longer than the predetermined first time after speaking. If it is determined that the action of a person C403 other than a certain person A401 and person B402 who is not the main subject has been detected (YES in S1005), the process returns to S1003, and the person A401 who is the main subject or the person B402 who is not the main subject is detected. It is determined again whether or not the person C403 other than the person A401 who is the main object in the image and the person B402 who is not the main object has performed an action within a predetermined first time after speaking.

Then, the conversation grouping unit 226 selects, within a predetermined first period of time after the main target person A401 or the non-main target person B402 speaks, the main target person A401 and non-main target person A401 in the image. If it is determined that the person C403 other than the person B402 has performed the action (YES in S1003), the grouping of the person C403 is maintained in S1004 (S1004).

As described above, it is determined whether grouping is to be maintained or canceled for the person C403 who has already been grouped, and grouping is performed again.

The voice processing unit 104 performs voice processing on the voices of the persons grouped as described above, in which they are conversing.

As an example of speech processing, existing speech separation technology is used to separate the speech of each individual person, to emphasize the speech of only grouped persons, or to reduce the speech of ungrouped persons. do. Enhancement processing includes processing to amplify the gain of the voice of the grouped person, processing to attenuate the gain of the voice of the person other than the grouped person, and processing of the voice of the grouped person. At least one of processing for amplifying the gain of the band is included. The processing method is not limited to this as long as the processing makes it easier to hear the voices of the grouped persons.

It should be noted that in the present embodiment, the number of groups that are actually having a conversation and the number of persons in the groups may be acceptable.

In addition, in the present embodiment, the number of times of operation is counted as 3 to determine that the group is having a conversation, but this is not the only option.

In addition, in the present embodiment, utterance, nodding, and the motion of directing the line of sight or face to the speaker are shown, but the present invention is not limited to these examples. good. For example, utterance, body movements such as lips, head, and hands in utterance, gaze and face movements, and even positive actions such as shaking the face vertically or nodding, or negative actions such as twisting or shaking the face sideways. good. Furthermore, changes in facial expressions of emotions and the like may be used.

Further, when the imaging device 100 has a function of tracking a specific subject included in an image as a main subject, when the imaging device 100 displays a live view, the main target person (face) in the image or the main target A tracking frame may be displayed with a person (face) grouped in the same group as the main subject, and subject to AF processing, AE processing, or zoom processing.

As described above, according to the present embodiment, it is possible to appropriately distinguish between a main target person and a person conversing with the main target person from a photographed image. Grouping can be performed appropriately as a group of By separating, emphasizing, and recording conversational voices of people grouped in the same group, the spoken voices of people the user is not interested in are recorded, and the voices of people who are having a conversation become difficult to hear. It is possible to improve the situation where

[Embodiment 2] Next, Embodiment 2 will be described.

In the first embodiment, it is necessary to detect the number of motions of a person at least a predetermined number of times. takes time. Therefore, in the second embodiment, a method of performing machine learning and estimating whether or not the main subject and another person will start a conversation before the conversation starts will be described.

Embodiments in which the electronic device of the present invention is applied to an imaging device such as a digital camera, and an information processing device is applied to a server will be described below.

FIG. 11 illustrates the hardware configuration of a server 1101 that executes machine learning according to the second embodiment.

The CPU 1102 is an arithmetic processing unit that executes programs stored in the storage device 1104 and controls each component of the server 1101 . The CPU 1102 also executes various calculations for machine learning.

A RAM 1103 is a memory that temporarily stores program data, video data, and audio data stored in the storage device 1104 .

The storage device 1104 includes a magnetic disk, an optical disk, a semiconductor memory, and the like, and stores program data for performing machine learning, video data and audio data for generating input data for learning, and learning obtained by machine learning. stored model.

The GPU 1105 is an arithmetic processing unit for image rendering that executes programs stored in the storage device 1104 and executes various operations for machine learning in cooperation with the CPU 1102 .

A bus 1106 is a data transmission path that allows data to be exchanged between components of the server 1101 .

FIG. 12 illustrates a software configuration for executing the estimation processing of the imaging device 100 and the learning processing of the server 1101 of the second embodiment.

The software 1201 that executes the estimation processing of the imaging device 1211 is implemented by a program stored in a storage device such as a ROM by an arithmetic processing device such as the CPU of the control unit 111 in FIG. The software 1211 that executes the learning process of the server 1101 is implemented by a program stored in the storage device 1104 by the CPU 1102 or GPU 1105 of FIG.

Since the GPU 1105 can perform efficient calculations by processing more data in parallel, it is effective to perform calculations on the GPU 1105 when learning multiple times using a learning model such as machine learning. is. Therefore, in the second embodiment, the processing by the learning unit 1206 uses the GPU 1105 in addition to the CPU 1102 . Specifically, when a learning program including a learning model is executed, the CPU 1102 and the GPU 1105 cooperate to perform calculations for learning. Note that the processing of the learning unit 1206 may be performed only by the CPU 1102 or the GPU 1105 . Functions of the software 1201 and 1211 in FIG. 12 will be described later with reference to FIGS. 14 and 15. FIG.

FIG. 13 is a conceptual diagram of a trained model according to the second embodiment. The learned model in FIG. 13 is generated by machine learning performed by the learning unit 1206 of the server in FIG. 12, and the estimating unit 1218 of the imaging device 100 performs estimation processing using the learned model.

As input data for the trained model 1303, image data including a person and person designation information for designating a person to be determined are input. Image data including a person may be a still image, or may be a moving image including a plurality of still images.

Also, the person designation information is, for example, coordinate information indicating where the faces of at least two persons whose faces are to be determined whether to start a conversation are located in the image, which is the input data. However, the person designation information is not limited to coordinate information, and feature information of a face image or the like may be used. Also, the number of persons to be designated as determination targets may be other than two.

As output data of the trained model 1303, an estimation result as to whether or not the two persons to be determined specified by the input data will start a conversation is output. In the second embodiment, even if the two persons to be determined do not directly start a conversation, it may be determined to start a conversation if the two persons start a conversation with a common third person. Also, even if the input data is an image after the conversation has already started, it may be determined that the conversation will start.

When two people start a conversation, there are visual characteristics such as facing the other person, approaching the other person, touching the other person's body, and changing facial expressions immediately before or after starting the conversation. There is a correlation between "image data including a person", which is input data, and "determination result as to whether or not to start a conversation", which is output data.

FIG. 14 is a flow chart showing learning processing by the software 1201 in FIG. 14 is realized by the CPU 1102 and GPU 1105 of the server 1101 shown in FIG. 11 working together to execute the software 1201 of the server shown in FIG.

FIG. 14(a) is a flow chart showing a learning process for a correct scene (a scene predicted to start a conversation).

In S<b>1401 , the learning data generation unit 1203 reads moving image data showing a person from the data storage unit 1202 . A data storage unit 1202 copies and stores in advance moving image data captured by another device. Alternatively, the data storage unit 1202 may pre-download and store moving image data stored in another device via a network.

In S1402, the main subject selection unit 1204 selects a person as a main subject from the moving image data read in S1401. In the second embodiment, the criteria for selecting a person to be the main object are not limited, but, for example, a person whose face is positioned closest to the center in the image is selected. Alternatively, a person is selected so that the face is arranged at the frontmost position in the image.

In S1403, the conversation scene determination unit 1205 determines a scene in which the main subject selected in S1402 is having a conversation with another person from the moving image data read in S1401. It also identifies a person who is having a conversation with the main subject. In the second embodiment, there are no restrictions on the method of determining a conversation scene or the method of determining a person who is having a conversation with a main subject. By doing so, the conversation scene and the person who is talking with the main subject are discriminated.

In S1404, the learning data generation unit 1203 extracts the image data immediately before the start of the conversation scene determined in S1403 as learning input data. The image data extracted here may be a still image, or may be a moving image consisting of a plurality of still images. In addition, the data is not limited to the data immediately before the conversation, and may include an image after the conversation starts.

In S1405, the learning data generation unit 1203 generates, as learning input data, person designation information designating the main subject selected in S1402 and a person who is having a conversation with the main subject determined in S1403. Although the format of the person designation information is not limited, in the second embodiment, coordinate information indicating the position of the main target and the face of one person who is having a conversation with the main target in the image extracted in S1404.

In S1406, the learning unit 1206 uses, as input data, the image extracted in S1404 and the coordinate information of the main subject generated in S1405 and the one person who is conversing with the main subject, and obtains information on the scene where the conversation starts. Learning is performed as teacher data. A learned model generated as a learning result is stored in the learning model storage unit 1207 . Specific algorithms of machine learning include nearest neighbor method, naive Bayes method, decision tree, support vector machine, and the like. Another example is deep learning, in which neural networks are used to generate feature quantities and connection weighting coefficients for learning. As appropriate, any of the above algorithms that can be used can be used and applied to the present embodiment. The learning section 1206 may include an error detection section and an update section. The error detection unit obtains an error between the output data output from the output layer of the neural network according to the input data input to the input layer and the teacher data. The error detector may use a loss function to calculate the error between the output data from the neural network and the teacher data. Based on the error generated by the error detection unit, the update unit updates the weighting coefficients for coupling between nodes of the neural network so that the error is reduced. The updating unit updates the coupling weighting coefficients and the like using, for example, the error backpropagation method. The error backpropagation method is a technique for adjusting the connection weighting coefficients and the like between nodes of each neural network so as to reduce the above error.

A correct scene (a scene predicted to start a conversation) is learned by repeatedly executing the process of FIG.

FIG. 14(b) is a flow chart showing learning processing of an incorrect scene (a scene predicted not to start a conversation).

The processing of S1411, S1412, S1413, and S1414 in FIG. 14B is the same as S1401, S1402, S1403, and S1404 in FIG. 14A.

In S1415, the learning data generation unit 1203 generates, as learning input data, person designation information designating one person other than the main subject selected in S1412 and the person who is having a conversation with the main subject determined in S1413. do. Although the format of the person designation information is not limited, in the image extracted in S1414, coordinate information indicating the position of the face of the main target and one person other than the person who is conversing with the main target is used in the second embodiment. do.

In S1416, the learning unit 1206 uses, as input data, the image extracted in S1414 and the coordinate information of the main target generated in S1415 and one person other than the person who is having a conversation with the main target, to create a scene in which a conversation does not start. Machine learning is performed using information as teacher data. A learned model generated as a learning result is stored in the learning model storage unit 1207 . A specific algorithm for machine learning is the same as in S1406.

In S1417, the learning data generation unit 1203 extracts, as learning input data, an image of a scene separated by a predetermined time from the conversation scene determined in S1413. The image data extracted here may be a still image, or may be a moving image consisting of a plurality of still images.

In S1418, the learning data generation unit 1203 generates the main target selected in S1412 and person designation information designating one person other than the main target as learning input data. Although the format of the person designation information is not limited, in the second embodiment, coordinate information indicating the face positions of the main target and one person other than the main target in the image extracted in S1417 is used as input data for learning. .

In S1419, the learning unit 1206 uses the image extracted in S1417, the main target generated in S1418, and the coordinate information of one person other than the main target as input data, and the information of the scene in which the conversation does not start as training data. do the learning. A learned model generated as a learning result is stored in the learning model storage unit 1207 . A specific algorithm for machine learning is the same as in S1406.

By repeatedly executing the processing of FIG. 14B using a large number of moving image data, the learning of incorrect scenes (scenes predicted not to start conversation) is performed.

FIG. 15 is a flow chart showing estimation processing by the software 1211 in FIG. The processing in FIG. 15 is implemented by the control unit 111 of the imaging device 100 shown in FIG. 1 executing the software 1211 of the imaging device shown in FIG.

In S1501, the imaging data acquisition unit 1213 acquires image data generated by the imaging unit 101 in FIG.

In S1502, the main subject selection unit 1212 selects a main subject in the image data acquired in S1501 using the function of the main subject selection unit 225 in FIG.

In S1503, the input data generation unit 1216 determines whether or not the main object selected in S1502 is a person. , the process returns to S1501.

In S1504, the input data generation unit 1216 generates the main object selected in S1502 and information designating the person in the image as input data for estimation performed in S1505. Although the format of the information specifying a person is not limited, in the second embodiment, coordinate information indicating the positions of the faces of the main subject and one person other than the main subject in the image acquired in S1501.

In S1505, the estimating unit 1218 uses the image acquired in S1501 and the coordinate information indicating the position of the face of the main target and one person other than the main target generated in S1504 as input data, and the main target and the designated main target. It is estimated whether the person who has A learned model stored in the learning model storage unit 1217 is used for estimation. Note that the learning model storage unit 1217 copies the learned model stored in the learning model storage unit 1207 in advance and stores it in a storage device such as a ROM included in the control unit 111 in FIG.

In S1506, the input data generation unit 1216 determines whether the estimation process in S1505 has been performed for all persons in the image acquired in S1501. If the input data generation unit 1216 determines that the estimation process has been performed for all persons, the process proceeds to S1507, and if it determines that the estimation process has not been performed for all persons, the process returns to S1504. .

In S1507, the estimation unit 1218 refers to the estimation result in S1505 and determines whether or not the main subject will start a conversation with one or more persons in the image acquired in S1501. If the estimating unit 1218 determines to start the conversation, the process advances to S1508, and if it determines not to start the conversation, the process returns to S1501.

In S1508, the recording setting control unit 1214 changes the characteristics of the voice input unit 103 in FIG. 1 so that the voice of the person determined to start a conversation in S1505 becomes easier to hear. For example, the sensitivity, frequency characteristics, and directivity of the microphone are changed according to the position of the person determined to start talking, or the gain and band are changed so as to emphasize the voice of the person. Alternatively, the audio processing unit 104 may be used to perform audio processing that makes it easier to hear the person's voice, such as emphasizing the person's voice band.

In S1509, the imaging setting control unit 1215 changes the imaging settings of the imaging unit 101 in FIG. 1 so that the person determined to start a conversation in S1505 can be easily visually recognized. For example, the focus position is adjusted by controlling the lens unit 201 in FIG. 2 so that the person determined to start talking in S1505 is in focus. Alternatively, the angle of view is adjusted by controlling the focal length of the lens unit 201 in FIG. 2 so that the person determined to start a conversation in S1505 is appropriately arranged within the angle of view.

Of the processing steps in FIG. 15, S1505 is executed using a machine-learned trained model, but rule-based processing such as a lookup table (LUT) may also be executed. In that case, for example, the relationship between the input data and the output data is created in advance as an LUT. Then, it is preferable to store the created LUT in a storage device such as a ROM included in the control unit 111 of FIG. When performing the processing of S1505, the output data can be acquired by referring to this stored LUT. That is, the LUT is a program for performing processing equivalent to the processing steps described above, and performs processing by operating in cooperation with the CPU or GPU.

14 and 15, it is possible to estimate whether or not the main subject will start a conversation with the person in the image before (or immediately after) the conversation. As a result, recording settings and shooting settings suitable for shooting conversation scenes can be performed in real time during shooting. By performing recording settings and shooting settings suitable for conversation scenes in real time, it is possible to record conversation scenes with higher quality than in the case of processing voices and captured images by post-processing. In addition, even when viewing recorded voices and captured images in real time at the time of shooting, it is possible to provide easy-to-hear conversational voices and images that are easy for the person speaking to visually recognize.
[Other embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to this embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100... imaging device (hardware, 102... image processing unit, 104... voice processing unit, 222... person detection unit, 223... person action detection unit, 224... person utterance detection unit, 225... main subject selection unit, 226... conversation Grouping part

Claims (16)

person detection means for detecting a person from the imaged image;
a main target selection means for selecting a main target person from among the persons detected by the person detection means;
human motion detection means for detecting the motion of the person detected by the human detection means;
human motion counting means for counting the number of motions detected by the human motion detecting means;
When the number of actions of the predetermined person detected within a predetermined first time after the utterance of the main target person is detected reaches a predetermined number of times within the predetermined first time, the predetermined person as a person who is conversing with the main subject. The grouping means is adapted to detect utterances of the main target person or a predetermined person grouped with the main target person, and the predetermined person other than the main target and the predetermined person detected within a predetermined first time. When the number of actions of a person other than the person reaches a predetermined number of times within the predetermined first time period, the other person is grouped as a person who is conversing with the main target. The electronic device according to claim 1. The grouping means determines that the number of actions of the other person who is not in conversation with the main target person detected within the predetermined first time after the main target person's utterance is detected is the predetermined number of times. 2. The electronic device according to claim 1, wherein when a predetermined number of times has been reached within the first period of time, the other person is grouped as a person who is conversing with the main subject. The grouping means detects an action of a person other than the main target and the predetermined person within a predetermined first time period after the utterance of the main target or the predetermined person is detected. , maintaining the grouping of said other persons;
Actions of persons other than the main target and the predetermined person are not detected within a predetermined second time longer than the first time after the utterance of the main target or the predetermined person is detected. 4. The electronic device according to claim 2 or 3, wherein the grouping of the other person is canceled if the grouping of the other person is canceled. The human action detection means detects the actions of the person detected by the person detection means, such as speaking, nodding, shaking the face or neck vertically, shaking the face or neck sideways, and directing the gaze or the face to the person who is speaking. 5. The electronic device according to any one of claims 1 to 4, wherein at least one of a motion and a motion of expressing emotions is detected. 6. The apparatus according to any one of claims 1 to 5, further comprising voice processing means for performing voice processing on voices uttered by persons in the group based on information on the persons grouped by said grouping means. 1. The electronic device according to item 1. The audio processing means performs a process of amplifying the gain of the voice of the person for whom the grouping has been performed, or a process of attenuating the gain of the voice of the person other than the person who has been grouped, or 7. The electronic device according to claim 6, wherein at least one of a process of amplifying the gain of the voice band of the person subjected to said grouping is performed. imaging means;
a person detection means for detecting a person from the image generated by the imaging means;
a person designation means for designating a main target person and a person other than the main target from the persons detected by the person detection means;
an image including the person specified by the person specifying means and the person specified by the person specifying means based on a trained model for estimating whether or not the person specified by the person specifying means will start a conversation; and an estimating means for estimating whether or not the person specified by the person specifying means will start a conversation, using information about the person as input data. an audio input means for inputting audio during shooting of moving image data; and a control means for controlling the audio input means;
9. The electronic device according to claim 8, wherein said control means changes any one of frequency characteristics, directivity and gain of said voice input means according to the estimation result of said estimation means. further comprising audio processing means for processing the audio input by the audio input means;
10. The electronic device according to claim 9, wherein said control means controls said audio processing means so as to adjust the gain of the audio of a scene in which a person is speaking according to the estimation result of said estimation means. machine. further comprising a lens unit for capturing an optical image of a subject into the imaging unit; and a lens control unit for controlling the lens unit;
9. The electronic device according to claim 8, wherein the lens control means changes the focal length or the in-focus position of the lens unit according to the estimation result of the estimation means. An information processing device that generates a trained model for estimating whether a specific person in an image and another person will start a conversation,
a storage means for storing moving image data generated by the imaging means;
a person detection means for detecting a person from the video data stored in the storage means;
a selection means for selecting a specific person from among the persons detected by the person detection means;
Conversation scene discrimination means for discriminating a scene in which the specific person is having a conversation with another person from the video data stored in the storage means;
Using an image immediately before or after the conversation scene extracted by the conversation scene determination means as input data for learning a scene in which conversation is predicted to start,
learning data generation means for using an image of a scene separated by a predetermined time from the conversation scene extracted by the conversation scene determination means as input data for learning a scene in which conversation is predicted not to start. and information processing equipment. A method of controlling an electronic device, comprising:
a person detection step of detecting a person from the captured image;
a main target selection step of selecting a main target person from among the persons detected by the person detection step;
a human motion detection step of detecting the motion of the person detected by the human detection step;
a human motion counting step for counting the number of motions detected by the human motion detection step;
When the number of actions of the predetermined person detected within a predetermined first time after the utterance of the main target person is detected reaches a predetermined number of times within the predetermined first time, the predetermined person as a person conversing with said main subject. A method of controlling an imaging device, comprising:
a person detection step of detecting a person from the image generated by the imaging means;
a person designation step of designating a main target person and a person other than the main target from the persons detected by the person detection step;
An image including the person specified by the person specifying step and information of the person specified by the person specifying step based on a trained model for estimating whether the person specified by the person specifying step will initiate a conversation. as input data, an estimation step of estimating whether the person specified by said person specifying step will start a conversation. A method for generating a trained model for estimating whether a specific person in an image and another person will start a conversation, comprising:
a person detection step of detecting a person from the recorded video data;
a selection step of selecting a specific person from among the persons detected by the person detection step;
a conversation scene determination step of determining a scene in which the specific person is having a conversation with another person from the moving image data;
An image immediately before or after the conversation scene extracted by the conversation scene determination step is used as input data for learning a scene in which conversation is predicted to start,
and a learning data generation step of using an image of a scene separated by a predetermined time from the conversation scene extracted by the conversation scene determination step as input data for learning a scene in which conversation is predicted not to start. and how to. A computer readable program for executing a method according to any one of claims 13 to 15 on a computer.

Images