JP7532963B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method.

Images (including still images and videos) including audio from within a vehicle cabin are captured by an on-board camera and an on-board microphone and are stored as a record of a journey. When reviewing such images stored as a record, it is effective to add effects such as subtitles to the images in advance.
For example, Patent Document 1 discloses a display mode determination device that extracts words from sounds included in an image and determines the display mode of subtitles depending on the frequency of use of the words.
The same applies not only to images of the interior of a vehicle, but also to images of the interior of various mobile objects or images of other spaces that are stored as records of events.

JP 2019-062332 A

However, the images mentioned above have a problem in that they contain a lot of redundant parts and are not suitable for reviewing as a record of events after shooting. This problem is particularly noticeable when the playback time is long.

The present invention has been made to solve the above-mentioned problems, and provides an image processing device and an image processing method that allow users to look back on important scenes captured during shooting in an easy-to-understand digest image after shooting.

An image processing device according to one aspect of the present invention includes an acquisition unit that acquires voice data and image data of a target space, a voice recognition unit that converts the voice data into text data, a situation evaluation unit that evaluates the situation of the target space based on the image data and calculates a situation evaluation value of the situation in the target space, and an image processing unit that, when an event occurs indicating a situation in which the situation evaluation value satisfies a predetermined condition, generates superimposed image data in which at least one of an image based on the text data and an image related to the situation is superimposed on an image of the image data for a predetermined period including the period during which the event occurs.

An image processing method according to one aspect of the present invention includes the steps of acquiring voice data and image data of a target space, converting the voice data into text data, evaluating a situation in the target space based on the image data and calculating a situation evaluation value of the situation in the target space, and, when an event occurs indicating a situation in which the situation evaluation value satisfies a predetermined condition, generating superimposed image data in which at least one of an image based on the text data and an image related to the situation is superimposed on an image of the image data for a predetermined period including the period during which the event occurs.

The present invention provides an image processing device and an image processing method that allow users to review important scenes captured during shooting in an easy-to-understand digest image after shooting.

1 is a block diagram showing an example of a configuration of an image processing apparatus according to a first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of a status table according to the first embodiment. 5 is a diagram illustrating an example of a data structure of an image processing mode table according to the first embodiment. FIG. 4 is a flowchart showing an outline of processing performed by the image processing device according to the first embodiment. 5 is a flowchart showing a first example of processing of the image processing device according to the first embodiment. 10 is a flowchart showing a second example of the process of the image processing device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image processing mode of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image processing mode of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image processing mode of the image processing apparatus according to the first embodiment. FIG. 11 is a block diagram showing an example of a configuration of an image processing device according to a second embodiment. FIG. 11 is a diagram illustrating an example of a data structure of a status table according to the second embodiment. FIG. 11 is a diagram illustrating an example of a data structure of an image processing mode table according to the second embodiment. 10 is a flowchart showing a process of an image processing device according to a second embodiment. FIG. 11 is a diagram showing an example of an image processing mode of an image processing apparatus according to a second embodiment.

The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In addition, the same elements are given the same reference numerals in each drawing, and duplicate explanations have been omitted as necessary.

<Embodiment 1>
First, a first embodiment of the present invention will be described with reference to FIGS. 1 to 9. FIG. 1 is a block diagram showing an example of the configuration of an image processing system 1 according to the first embodiment. The image processing system 1 is a computer system that performs image processing such as adding subtitles to an image captured of a target space including a person, and generates a digest image. In the first embodiment, the target space indicates an interior space of a vehicle in which an occupant is present (hereinafter, sometimes referred to as a vehicle interior). However, the target space is not limited to this, and may be an interior space of another moving body such as a ship or airplane in which an occupant is present, or an indoor or outdoor space of a building in which a person is present.

The image processing system 1 includes an image processing device 10, a camera 30, and a microphone 40. In this embodiment 1, the image processing device 10, the camera 30, and the microphone 40 configure an in-vehicle equipment system including in-vehicle equipment such as a drive recorder, a driver monitor, or an in-cabin monitor.

The camera 30 is mounted at any position in the vehicle, captures the scenery inside the vehicle cabin, and generates image data of the captured image. As an example, the camera 30 captures an image of an occupant sitting in a seat in the vehicle. The camera 30 is communicatively connected to the image processing device 10. The camera 30 generates image data at, for example, 30 frames per second (30 fps), and supplies the generated image data to the image processing device 10 every 1/30th of a second. In addition to the scenery inside the vehicle cabin, the camera 30 may also capture the scenery outside the vehicle and supply the image data to the image processing device 10.

The microphone 40 is mounted at any position in the vehicle, and captures the voices within the vehicle cabin to generate voice data. As an example, the microphone 40 captures the voices of passengers sitting in the vehicle seats. The microphone 40 is also communicatively connected to the image processing device 10. The microphone 40 supplies the generated voice data to the image processing device 10 in the same manner as the camera 30. The microphone 40 is not limited to capturing voices within the vehicle cabin, and may capture voices outside the vehicle and supply them to the image processing device 10.

The image processing device 10 is a computer or hardware installed in association with an in-vehicle device. The image processing device 10 may be built into the in-vehicle device or may be arranged externally. The image processing device 10 may also be installed at a location away from the vehicle and communicate with the camera 30 and microphone 40 via wireless communication means. The image processing device 10 performs image processing, such as adding subtitles, on the image and audio data acquired from the camera 30 and microphone 40, respectively, to generate a digest image. The image processing device 10 may be configured to include the camera 30 and the microphone 40.

The image processing device 10 has an acquisition unit 11, a situation evaluation unit 12, a voice recognition unit 15, an image processing unit 16, an image recording unit 17, and a table storage unit 18.

The acquisition unit 11 acquires voice data and image data of the vehicle interior to be photographed from the microphone 40 and the camera 30, respectively. The acquisition unit 11 may be an interface such as an input terminal or an input circuit that is connected to the microphone 40 and the camera 30 by wire, or may be a communication receiving unit that is connected to the microphone 40 and the camera 30 by wireless. The acquisition unit 11 supplies the acquired voice data and image data to the situation evaluation unit 12 and the voice recognition unit 15.

The situation evaluation unit 12 evaluates the situation inside the vehicle cabin. Here, the situation inside the vehicle cabin refers to a typified state of the facial expressions or speech patterns of one or more people (occupants) in the space. For example, the situation may include a state in which there is a smiling occupant, a state in which there is a sleeping occupant, or a state in which there is a mixture of angry occupants and smiling occupants.

At this time, the situation evaluation unit 12 calculates a situation evaluation value for the situation to be evaluated. The situation to be evaluated refers to a situation that is the subject of evaluation, and for example refers to a situation designated as the subject of evaluation among the situations shown in the situation table 19 described below. The situation to be evaluated may be all or some of the situations shown in the situation table 19.

The situation evaluation value is calculated for each situation being evaluated and indicates the level to which the situation inside the vehicle is related to the situation being evaluated.

Here, the situation evaluation unit 12 includes an image evaluation unit 13 .
The image evaluation unit 13 calculates a facial expression evaluation statistic for each facial expression related to the evaluation target situation based on the image data acquired by the acquisition unit 11. More specifically, the image evaluation unit 13 performs image recognition processing on the image data to recognize people included in the image data, and also recognizes the facial expressions of the recognized people and evaluates them as types of facial expressions such as "laughter,""sadness,""anger," and "sleepiness." The image recognition processing uses known techniques such as machine learning.

The facial expression evaluation statistic is one of the situation evaluation indexes included in the situation evaluation value. The facial expression evaluation statistic is a statistical value of facial expression evaluation values for a population of all occupants captured in image data.
Here, the facial expression evaluation value is determined according to the type of facial expression, and is an index showing the degree to which the facial expression of each occupant is related to the determined facial expression. The facial expression evaluation value is calculated for each occupant appearing in the image data. For example, the facial expression evaluation value may be calculated based on the difference between the actual value obtained from the image data and a predetermined reference value for the size, position, shape, or angle of each part of the face or the face or neck. The facial expression evaluation value may also be calculated by a facial expression evaluation model that has learned images of people with various facial expressions as subjects using as teacher data. The facial expression evaluation statistics are calculated based on the facial expression evaluation values of all occupants appearing in the image data. The facial expression evaluation statistics may be the number of occupants having a facial expression evaluation value equal to or greater than a threshold value, or may be the proportion of occupants having a facial expression evaluation value equal to or greater than a threshold value among all occupants. The facial expression evaluation statistics may also be a determination result of whether or not the number or proportion of occupants having a facial expression evaluation value equal to or greater than a threshold value exceeds a predetermined threshold value. The facial expression evaluation statistics may be the sum or average value of the facial expression evaluation values. For example, the facial expression evaluation statistics of "laughing" may have a higher value as the number of occupants having a high facial expression evaluation value of "laughing" increases. By using the facial expression evaluation statistics as the situation evaluation value in this manner, it is possible to more accurately evaluate the overall situation inside the vehicle cabin.

In this embodiment 1, a facial expression evaluation statistic is used as the situation evaluation value. That is, in this embodiment 1, the image evaluation unit 13 evaluates the facial expression of the person captured based on the image data, calculates a facial expression evaluation statistic of the facial expression related to the situation to be evaluated in the vehicle cabin, and sets the facial expression evaluation statistic as the situation evaluation value.

The situation evaluation unit 12 then uses the situation table 19 to determine whether the situation evaluation value satisfies a predetermined condition, and if so, recognizes that an event of the situation to be evaluated has occurred. In this embodiment 1, the situation evaluation unit 12 determines whether each facial expression evaluation statistical value related to the situation to be evaluated is equal to or greater than a predetermined threshold value corresponding to that facial expression evaluation value, and determines whether an event has occurred based on these determination results. If the situation evaluation unit 12 recognizes that an event has occurred, it notifies the voice recognition unit 15 and the image processing unit 16.

The voice recognition unit 15 converts the voice data into text data including character strings. The voice recognition unit 15 may use a known voice data text conversion technology.

The image processing unit 16 performs image processing according to the evaluation target situation on the image of the image data during the event occurrence period using the image processing mode table 20. The event occurrence period refers to a period including the period during which the event actually occurs and a predetermined period before and after the event occurrence. The event occurrence period may include either the period before the event occurrence or the period after the event occurrence instead of the period before and after the event occurrence, or may not include these periods. The image processing unit 16 generates superimposed image data by superimposing at least one of an image based on text data and an image related to the evaluation target situation on the image of the image data during the event occurrence period. An example of the image based on text data is a subtitle image in which the text data is visualized as a character string in a predetermined font. For example, when the text data is Japanese, the image based on text data may be converted into kanji or katakana and then converted into a subtitle image. The image based on text data may be in a form in which part of the text data is omitted or predetermined text data is added. The image based on text data may be a mark image that is previously associated with the text data by a database not shown. The image related to the evaluation target situation is a mark image that is previously associated with a situation ID by the image processing mode table 20 described later. For example, if the situation to be evaluated has a situation ID that includes "sadness," the image associated with the situation to be evaluated may be a teardrop-shaped mark image.

The image processing unit 16 then records the generated superimposed image data in the image recording unit 17. The image recording unit 17 is a storage medium that stores the superimposed image data generated by the image processing unit 16.

At this time, the image processing unit 16 preferably records the data in the image recording unit 17 in a recording format that prohibits overwriting. For example, when the image processing device 10 is mounted on a drive recorder, the image processing unit 16 may record the data in the image recording unit 17 in a recording mode different from the recording modes of a normal drive recorder, such as continuous recording (overwriting), abnormality recording (recording with overwriting prohibited when an impact is detected, etc.), and manual recording. The image recording unit 17 is preferably a recording area different from the recording area of the normal recording mode. It is preferable that the image processing unit 16 does not perform the above-mentioned image superimposition process on the image recorded in the abnormality recording recording mode. This makes it possible to easily view the recorded file after driving, distinguishing it from the data recorded in the normal recording mode. In order to further facilitate viewing, it is preferable to identify the recorded file by giving a folder name and file name according to the recording mode, but this is not limited to this. Note that the drive recorder does not need to be equipped with all or part of the above-mentioned recording modes.

The table storage unit 18 is a storage medium that stores the status table 19 and the image processing mode table 20.

FIG. 2 is a diagram illustrating an example of a data structure of the status table 19 according to the first embodiment.
The situation table 19 associates situation identification information (situation ID), the content of the situation, the type of situation evaluation value, and an event threshold value.

In this embodiment 1, the content of the situation refers to facial expressions related to the situation or a combination thereof. For example, the content of the situation with situation IDs 1 to 3 indicates that there are smiling passengers. The content of the situation with situation IDs 4 to 5 indicates that there are both sad and smiling passengers. The content of the situation with situation ID 6 indicates that there are both angry and smiling passengers. The content of the situation with situation ID 7 indicates that there are both sleeping passengers, and the content of the situation with situation ID 8 indicates that there are both sleeping and smiling passengers.

The type of situation evaluation value indicates the type of situation evaluation value corresponding to each item included in the situation content. For example, the situations with situation IDs 1 to 3 have the first facial expression evaluation statistic, which is the facial expression evaluation statistic for "laughter", as the situation evaluation value. Furthermore, the situations with situation IDs 4 to 5 have the second facial expression evaluation statistic, which is the facial expression evaluation statistic for "sadness", and the first facial expression evaluation statistic as the situation evaluation value. Furthermore, the situation with situation ID 6 has the third facial expression evaluation statistic, which is the facial expression evaluation statistic for "anger", and the first facial expression evaluation statistic as the situation evaluation value. Furthermore, the situation with situation ID 7 has the fourth facial expression evaluation statistic, which is the facial expression evaluation statistic for "drowsiness", and the situation with situation ID 8 has the fourth facial expression evaluation statistic and the first facial expression evaluation statistic as the situation evaluation value.

The event threshold is a threshold for the situation evaluation unit 12 to determine the occurrence of an event. For example, situations are classified into three stages, situation IDs 1 to 3, depending on the magnitude of the first evaluation statistical value. If the first evaluation statistical value is _X1 or more, it is determined that an event of situation ID 1 in which all occupants are smiling has occurred. If the first evaluation statistical value is _X2 or more and less than _X1 , it is determined that an event of situation ID 2 in which the majority of occupants are smiling has occurred. If the first evaluation statistical value is _X3 or more and less than _X2 , it is determined that an event of situation ID 3 in which a small number of occupants are smiling has occurred. As a modified example, the first evaluation statistical value may be an evaluation value of the degree of smiling of the occupants, which will be described later, or may be determined as a case where situation ID 1 has a facial expression evaluation value of 5 or more, or situation ID 2 has a facial expression evaluation value of 3 or 4, etc.

Here, a specific example of a method for calculating various facial expression evaluation statistics will be described. First, the image evaluation unit 13 of the situation evaluation unit 12 generates a normalized image of the face of each occupant from the acquired image data in advance. Then, when the facial expression related to the situation is "laughing", the image evaluation unit 13 of the situation evaluation unit 12 calculates the amount of decrease in the width of the eyes of the occupant, the amount of change in the size of the mouth, and the amount of rise in the corners of the mouth of the occupant appearing in the normalized image from the average values of the eye size, the size of the mouth, and the position of the corners of the mouth in the case of a blank expression. The image evaluation unit 13 calculates the facial expression evaluation value of "laughing" as a five-level evaluation value, for example, from 1 (blank expression) to 5 (maximum smile), based on various changes and conditions such as whether teeth are visible. It may be a 0/1 evaluation of whether or not the person is laughing. This evaluation value may be calculated based on parameters such as the change in the size of the mouth and whether teeth are visible, or the degree of smiling may be evaluated using a recognition dictionary of the degree of smiling that has been machine-learned. The same applies to "sadness", "anger", and "drowsiness" described later. The facial expression evaluation value for "laughing" may also be calculated using a facial expression evaluation model trained on training data of images of "laughing" facial expressions of people with various facial expressions.

If the facial expression associated with the situation is "sadness," the image evaluation unit 13 calculates the amount of reduction in the eye width of the occupant in the normalized image and the amount of change in the shape of the mouth and the angle of the neck and face from the average values of the eye size, mouth shape, and neck and face angle in the case of a neutral expression. The image evaluation unit 13 calculates the facial expression evaluation value of "sadness" based on the various amounts of change and conditions such as the presence or absence of tears. The facial expression evaluation value of "sadness" may also be calculated using a facial expression evaluation model trained using images of "sadness" facial expressions of people with various facial expressions as training data.

Furthermore, if the facial expression associated with the situation is "anger," the image evaluation unit 13 calculates the amount of change in the eye size, mouth shape, eyebrow position, and neck and face angle of the occupant appearing in the normalized image from the average values of the eye size, mouth shape, eyebrow position, and neck and face angle in the case of a neutral expression. The image evaluation unit 13 calculates an facial expression evaluation value for "anger" based on the various amounts of change. The facial expression evaluation value for "anger" may also be calculated using a facial expression evaluation model trained using images of "anger" facial expressions of people with various facial expressions as subject data.

If the facial expression related to the situation is "drowsiness," the image evaluation unit 13 determines whether the eyes of the occupant in the normalized image are closed from the average values of the eye size and the angles of the neck and face when the occupant has no expression, and calculates the amount of change in the angle of the neck and face. The image evaluation unit 13 calculates the facial expression evaluation value of "drowsiness" based on the determination result and various amounts of change. The facial expression evaluation value of "drowsiness" may also be calculated by a facial expression evaluation model that has been trained using images of "drowsiness" facial expressions of people with various facial expressions as subject data.

The image evaluation unit 13 then calculates various facial expression evaluation statistics based on the calculated facial expression evaluation values for each occupant.

Figure 3 is a diagram showing an example of the data structure of the image processing mode table 20 according to the first embodiment. The image processing mode table 20 associates a situation ID with an image processing mode. The image processing mode includes matters related to the display mode of the superimposed subtitles, such as the character size, font, color, and subtitle superimposition position of the subtitles, and other image processing modes. In this figure, matters related to the mode of the superimposed subtitles are the character size, font, and subtitle superimposition position. Other image processing modes include adding marks, adding subtitles according to the situation, solid black processing, changing colors, converting color images to monochrome, black and white inversion processing, and mosaic processing. The character size in Figure 3 is shown as five levels from 5 (maximum) to 1 (minimum) as an example, but may be ten levels or may be specified by the actual character size (for example, the number of pixels of the font height).

The image processing unit 16 may determine the display mode of the superimposed subtitles based on the situation evaluation value, that is, in this embodiment 1, the facial expression evaluation statistical value. This allows subtitle presentation according to the degree of excitement in the cabin determined by the statistical value of the facial expressions of the occupants.

FIG. 4 is a flowchart showing an outline of the process of the image processing apparatus 10 according to the first embodiment.
First, in step S10, the acquisition unit 11 of the image processing device 10 acquires voice data and image data in the vehicle cabin from the microphone 40 and the camera 30, respectively. The acquisition unit 11 supplies the acquired voice data and image data to the situation evaluation unit 12 and the voice recognition unit 15.

Next, in step S11, the image evaluation unit 13 of the situation evaluation unit 12 calculates a facial expression evaluation value related to the situation to be evaluated for each recognized facial expression of the occupant based on the image data, and calculates a facial expression evaluation statistical value based on the facial expression evaluation value of each occupant. If there are multiple facial expression evaluation statistical values related to the situation to be evaluated, the image evaluation unit 13 calculates each facial expression evaluation statistical value. Note that the image evaluation unit 13 may use the situation table 19 to check the type of facial expression evaluation statistical value related to the situation to be evaluated.

Then, in step S12, the situation evaluation unit 12 calculates a situation evaluation value for each situation evaluation value type related to the situation to be evaluated. In this embodiment 1, the situation evaluation unit 12 sets the facial expression evaluation statistics calculated in step S11 as the situation evaluation value.

Next, in step S13, the situation evaluation unit 12 uses the situation table 19 to determine whether or not an event of the situation to be evaluated has occurred, based on the situation evaluation value for each situation evaluation value type calculated in step S12. If the situation evaluation unit 12 determines that an event has occurred (Yes in step S13), it notifies the voice recognition unit 15 of this and proceeds to step S14. On the other hand, if the situation evaluation unit 12 does not determine that an event has occurred (No in step S13), it returns the process to step S10.

In step S14, in response to receiving a notification from the situation evaluation unit 12, the voice recognition unit 15 converts the voice data during the event occurrence period into text data, supplies the converted text data to the image processing unit 16, and proceeds to step S15.

In step S15, the image processing unit 16 uses the image processing mode table 20 to generate at least one of an image based on text data corresponding to the situation ID of the situation to be evaluated and an image related to the situation to be evaluated for the image data of the event occurrence period, performs image processing to superimpose the image data, generates superimposed image data, and proceeds to step S16.

Then, in step S16, the image processing unit 16 records the audio data and superimposed image data during the event occurrence period in the image recording unit 17 with overwriting prohibited, and ends the process. Step S16 may be omitted.

Note that step S15 may be executed before step S14, or may be executed in parallel. Also, instead of step S14, the voice recognition unit 15 may convert the voice data into text data in response to acquisition of the voice data, regardless of whether or not a notification is received from the situation evaluation unit 12. Also, the voice recognition unit 15 may convert the voice data into text data only when it is determined that the occupant is speaking based on the movement of the occupant's mouth. Note that the above determination may be made by the image evaluation unit 13.

Next, a specific example of the processing of the image processing device 10 will be described with reference to Figures 5 and 6. Figure 5 is a flowchart showing a first example of the processing of the image processing device 10 according to the first embodiment. In the first example, it is assumed that situation IDs 1 to 3 in the situation table 19 are designated in advance as situations to be evaluated. Note that a description of the processing shown in step S10 in Figure 4 will be omitted.

In step S20, the situation evaluation unit 12 of the image processing device 10 calculates a first facial expression evaluation value for each occupant and determines whether or not there is an occupant whose first facial expression evaluation value is equal to or greater than a predetermined threshold value, that is, whether or not a smiling occupant has been detected. If the image evaluation unit 13 determines that a smiling occupant has been detected (Yes in step S20), the process proceeds to step S21; otherwise (No in step S20), the process shown in step S20 is repeated.

Next, in step S21, the situation evaluation unit 12 calculates a first facial expression evaluation statistic and determines whether or not the first facial expression evaluation statistic satisfies the condition of situation ID 3 shown in the situation table 19, that is, whether or not a predetermined number of occupants are smiling. If the situation evaluation unit 12 determines that a predetermined number of occupants are smiling (Yes in step S21), it determines that an event that gets the conversation going has occurred and proceeds to step S22; if not (No in step S21), it returns to step S20.

In step S22, the situation assessment unit 12 backs up the image data before and after the smiling occupant is detected in a specified storage area.

Next, in step S23, the voice recognition unit 15 converts the voice data before and after detecting the smiling occupant into text data.

Next, in step S24, the situation evaluation unit 12 determines whether the first facial expression evaluation statistical value satisfies the condition of situation ID 1 shown in the situation table 19, that is, whether all occupants are smiling. If the situation evaluation unit 12 determines that all occupants are smiling (Yes in step S24), the process proceeds to step S25; otherwise (No in step S24), the process proceeds to step S26.

In step S25, the image processing unit 16 superimposes an image such as a subtitle onto the image of the backed up image data in the image processing mode of situation ID 1 in the image processing mode table 20. For example, the image processing unit 16 makes the character size larger than normal subtitles and gives the subtitles a more eye-catching font and color. The image processing unit 16 then advances the process to step S29.

In step S26, the situation evaluation unit 12 determines whether the first facial expression evaluation statistic satisfies the condition of situation ID 2 shown in the situation table 19, that is, whether the majority of the occupants are smiling. If the situation evaluation unit 12 determines that the majority of the occupants are smiling (Yes in step S26), the process proceeds to step S27; otherwise (No in step S26), the process proceeds to step S28.

In step S27, the image processing unit 16 superimposes an image such as a subtitle onto the image of the backed up image data in the image processing mode of situation ID 2 in the image processing mode table 20. The image processing unit 16 then advances the process to step S29.

In step S28, the image processing unit 16 superimposes an image such as a subtitle onto the image of the backed up image data in the image processing mode of situation ID 3 in the image processing mode table 20. The image processing unit 16 then advances the process to step S29.

In step S29, the image processing unit 16 records the superimposed image data and the corresponding audio data in the image recording unit 17 with overwrite protection, and ends the process. Step S29 may be omitted.

In this example, the image processing mode is determined by the first facial expression evaluation statistic, which takes into account the number of smiles. Therefore, it is possible to perform subtitle presentation according to the level of excitement determined by the facial expressions of the passengers.

Figure 6 is a flowchart showing a second example of the processing of the image processing device 10 according to the first embodiment. In the second example, it is assumed that situation IDs 7 to 8 in the situation table 19 are designated in advance as situations to be evaluated. Note that the description of the processing shown in step S10 in Figure 4 is omitted.

In step S30, the situation evaluation unit 12 of the image processing device 10 calculates a fourth facial expression evaluation statistical value and determines whether or not the fourth facial expression evaluation statistical value satisfies the conditions of situation IDs 7 to 8 shown in the situation table 19, that is, whether or not a sleeping occupant has been detected. If the situation evaluation unit 12 determines that a sleeping occupant has been detected (Yes in step S30), the process proceeds to step S31; otherwise (No in step S30), the process shown in step S30 is repeated.

In step S31, the situation evaluation unit 12 calculates a first facial expression evaluation statistic and determines whether the first facial expression evaluation statistic satisfies the condition of situation ID 8 shown in the situation table 19, that is, whether a predetermined number of occupants or more are smiling. If the situation evaluation unit 12 determines that a predetermined number of occupants or more are smiling (Yes in step S31), it determines that an event has occurred in which other occupants are amused by the sleeping occupant, and proceeds to step S32. On the other hand, if this is not the case (No in step S31), the situation evaluation unit 12 proceeds to step S35.

In step S32, the image processing unit 16 generates zoomed-in images of the face of the sleeping occupant (target occupant) before and after the sleeping occupant is detected, and backs up the images in a specified recording area.

Next, in step S33, the voice recognition unit 15 converts the voice data before and after detecting the sleeping occupant into text data.

Next, in step S34, the image processing unit 16 performs image processing in which, in the image processing mode of situation ID 8 in the image processing mode table 20, at least one of an image based on the text data converted in step S33 and an image related to the situation to be evaluated is generated for the image of the backed up image data, and the image processing unit 16 superimposes the image data to generate superimposed image data. Then, the image processing unit 16 proceeds to the process of step S37.

In step S35, the situation assessment unit 12 generates a zoomed-in image of the target occupant's face, similar to step S32, and backs it up in a specified recording area.

Next, in step S36, the image processing unit 16 superimposes subtitles and an image related to the situation onto the image of the backed up image data in the image processing mode of situation ID 7 in the image processing mode table 20. The image processing unit 16 then advances the process to step S37.

In step S37, the image processing unit 16 records the superimposed image data and the corresponding audio data in the image recording unit 17 with overwrite protection, and ends the process. Step S37 may be omitted.

Figures 7 to 9 are diagrams showing an example of the image processing mode of the image processing device 10 according to the first embodiment.

FIG. 7 shows an image IMG1 of the superimposed image data that is generated when an event corresponding to situation ID 8 occurs.
The image processing unit 16 generates image data based on the situation ID 8 in the image processing mode table 20, for example, a "zzz" mark image MK1 representing the onomatopoeia of sleep and a nose balloon image MK2, and superimposes them on a zoomed-up image of the face of the target occupant who is sleeping.
The image processor 16 then converts the voice data, which is the speech of the other passenger, into text data to generate subtitle image data such as "Don't wake up," and superimposes SB1 at the bottom of the image.

FIG. 8 shows an image IMG2 of the superimposed image data that is generated when an event corresponding to situation ID 5 occurs.
The image processing unit 16 generates and superimposes a teardrop mark image MK3 representing sadness around the eyes of the zoomed-up image of the sad target occupant's face, and superimposes a vertical diagonal line mark MK4 around the forehead.
The image processing unit 16 then converts the voice data, which is the speech content of the target occupant, into text data to generate a subtitle image SB2 saying "Hiiiiii" and superimposes it on the left side of the occupant.

FIG. 9 shows an image IMG3 of the superimposed image data that is generated when an event corresponding to situation ID 6 occurs.
The image processing unit 16 generates an anger mark image MK5 representing anger around the forehead of the angry target occupant and superimposes it on the zoomed-up image of the face of the angry target occupant.
The image processing unit 16 then converts the voice data, which is the speech content of the target occupant, into text data to generate a subtitle image SB3 saying "What!" and superimposes it to the right of the occupant together with a speech bubble expressing anger.

As described above, according to the first embodiment, the image processing device 10 evaluates the situation of the target space based on information detected from the image data, and generates a digest image of the period around the time when an event corresponding to the situation occurred. This allows the user to look back on important scenes during shooting, particularly exciting scenes, using the digest image after shooting.

The image processing device 10 also applies subtitles to the captured images in accordance with the evaluated situation. This allows the user to look back on their travel memories after capturing the images with easy-to-understand subtitles, like those in a variety show.

To evaluate the situation in the target space, the image processing device 10 may detect the facial expressions of the occupants from the image data and consider combinations of the occupants' facial expressions, rather than just a single facial expression. This makes it possible to record images of not just "laughing" but also complex situations in which opposing facial expressions, such as a combination of "sadness" and "laughing," occur simultaneously among the occupants, as scenes of excitement.

<Embodiment 2>
Next, a second embodiment of the present invention will be described with reference to Figures 10 to 14. The second embodiment is characterized in that the situation assessment unit assesses the situation of the target space based on audio data in addition to image data.

Figure 10 is a block diagram showing an example of the configuration of an image processing device 10a according to the second embodiment. The image processing device 10a according to the second embodiment has basically the same configuration and functions as the image processing device 10 according to the first embodiment. However, the image processing device 10a differs from the image processing device 10 in that it includes a situation evaluation unit 12a, a situation table 19a, and an image processing mode table 20a instead of the situation evaluation unit 12, the situation table 19, and the image processing mode table 20.

The situation evaluation unit 12a has basically the same functions as the situation evaluation unit 12, but in addition to the image evaluation unit 13, it also has an audio evaluation unit 14.

The voice evaluation unit 14 evaluates features obtained from the voice data and calculates a voice evaluation value based on the features. For example, the voice evaluation value includes a first voice evaluation value indicating the level of volume of the speech included in the voice data, and a second voice evaluation value that recognizes the spoken voice, extracts predetermined keywords, and indicates their frequency of occurrence.

The situation evaluation unit 12a evaluates the situation in the target space based on the facial expression evaluation statistics calculated by the image evaluation unit 13 and the audio evaluation value calculated by the audio evaluation unit 14. Therefore, the situation evaluation value includes the audio evaluation value in addition to the facial expression evaluation statistics as a situation evaluation index.

11 is a diagram showing an example of the data structure of the situation table 19a according to the second embodiment. The situation table 19a is a record of situations with situation IDs 9 to 10 in which the voice evaluation value is adopted as one of the situation evaluation values instead of the situation table 19. The situation table 19a may be configured to include the situation ID of the situation table 19.
The contents of the situations with the situation IDs 9 to 10 indicate facial expressions related to the situation, speech patterns, or a combination thereof.
For example, the content of the situation of situation ID 9 indicates that there is a smiling occupant and that the occupant is speaking at a volume (loud voice) equal to or greater than a predetermined value. In this case, the first facial expression evaluation statistical value based on the image data and the first voice evaluation value based on the voice data are set as the situation evaluation value.
The content of the situation of situation ID 10 indicates that there is a high frequency keyword that indicates a word that is repeatedly uttered at a frequency equal to or higher than a predetermined frequency during a period when a smiling occupant is present. In this case, the first facial expression evaluation statistic based on the image data and the second voice evaluation value based on the voice data are set as the situation evaluation value.

FIG. 12 is a diagram showing an example of the data structure of the image processing mode table 20a according to the second embodiment.
The image processing mode table 20a includes, in addition to the image processing mode table 20, records of the situations with the situation IDs 9 to 10.

The image processing unit 16 may determine the style of the subtitles to be superimposed based on the audio evaluation value in addition to or instead of the facial expression evaluation statistics. This allows the subtitles to be displayed in a way that matches the level of excitement.

Figure 13 is a flowchart showing the processing of the image processing device 10a according to the second embodiment. The steps shown in Figure 13 include steps S40 to S43 instead of steps S12, S13, and S15 shown in Figure 4. Explanations of steps that are the same as those shown in Figure 4 will be omitted as appropriate.

The voice data text conversion process shown in step S14 may be performed between steps S10 and S11.

In step S40, the voice evaluation unit 14 of the situation evaluation unit 12a calculates various voice evaluation values from the voice data supplied from the acquisition unit 11. This step may be performed before step S11 or in parallel with step S11.

In step S41, if the situation to be evaluated corresponds to situation ID 9 or 10, the situation evaluation unit 12a sets the voice evaluation value as the situation evaluation value in addition to the facial expression evaluation statistics.

In step S42, the situation evaluation unit 12a judges whether or not an event of the evaluation target situation has occurred by using the situation table 19a. If the situation evaluation unit 12a judges that an event has occurred (Yes in step S42), it supplies information on the situation evaluation value to the image processing unit 16 and proceeds to step S43. On the other hand, if the situation evaluation unit 12a does not judge that an event has occurred (No in step S42), it returns the process to step S10.

In step S43, the image processing unit 16 uses the image processing mode table 20a to perform image processing on the image data during the event occurrence period according to the evaluation target situation, and generates superimposed image data.

Figure 14 is a diagram showing an example of an image processing mode of the image processing device 10a according to the second embodiment. This figure shows an image IMG4 of superimposed image data that is generated when an event corresponding to situation ID 10 occurs. In situation ID 10, when a keyword appears with a frequency equal to or greater than a threshold value in the presence of smiling passengers, the situation evaluation unit 12a determines that the conversation is lively and that the keyword is a key point of the conversation.

The image processing unit 16 then converts the captured image data into, for example, a solid black image, and generates superimposed image data in which the frequently occurring keyword "sushi" is superimposed as subtitle data in a display mode that emphasizes the keyword by increasing the character size, changing the font to be different from other words, changing the color, increasing the color density, etc., and superimposes the superimposed image data on the converted solid black image.
When such an image is reproduced as a video, an image in which the above-mentioned superimposed image data is superimposed on a solid black image is reproduced, and the corresponding audio may be reproduced. At this time, the audio may be reproduced in a manner that emphasizes the frequently occurring keywords by increasing the volume, converting the frequency, etc.

As described above, according to the second embodiment, the image processing device 10a calculates a situation evaluation value based on image data as well as features obtained from voice data such as voice volume and repetition of keywords. This allows the image processing device 10a to accurately detect exciting scenes and perform image presentation with a wide variety of options.

In the above embodiment, the present invention has been described as being configured as hardware, but the present invention is not limited to this. The present invention can also be realized by having a processor execute a computer program to perform any process.

In the above example, the program can be stored and supplied to the computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, DVDs (Digital Versatile Discs), BDs (Blu-ray (registered trademark) Discs), and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can provide the program to the computer via a wired communication path, such as an electric wire or optical fiber, or via a wireless communication path.

In the above-described embodiment, the computer is configured as a computer system including a personal computer, a word processor, etc. However, the computer is not limited to this, and can also be configured as a server of a LAN (local area network), a host for computer (personal computer) communication, a computer system connected to the Internet, etc. It is also possible to distribute functions to each device on the network and configure a computer as a whole network.

The order of execution of each process in the systems and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and may be implemented in any order, as long as the output of a previous process is not used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, this does not mean that it is essential to implement the process in this order.

The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. A part or all of the above-described embodiment can be described as follows, but is not limited to the following.
(Additional Note)
(Appendix 1)
An acquisition unit that acquires audio data and image data of a target space;
a voice recognition unit for converting the voice data into text data;
a situation evaluation unit that evaluates a situation of the target space based on the image data and calculates a situation evaluation value of the situation in the target space;
and an image processing unit that, when an event occurs indicating a situation in which the situation evaluation value satisfies a predetermined condition, generates superimposed image data by superimposing at least one of an image based on the text data and an image related to the situation on an image of the image data for a predetermined period including a period during which the event occurs.
(Appendix 2)
the situation evaluation unit evaluates a facial expression of the person captured based on the image data, and calculates a facial expression evaluation statistic of the facial expression related to the situation in the target space;
The image processing system of claim 1, wherein the situation evaluation value includes the facial expression evaluation statistic.
(Appendix 3)
The image processing system according to claim 1 or 2, wherein the image processing unit determines an aspect of an image based on the text data to be superimposed based on the situation evaluation value.
(Appendix 4)
a voice evaluation unit that calculates a voice evaluation value based on features acquired from the voice data;
The situation evaluation value includes the voice evaluation value.
4. The image processing system according to claim 1 .
(Appendix 5)
acquiring audio data and image data of a target space;
converting the audio data into text data;
evaluating a situation in the target space based on the image data and calculating a situation evaluation value of the situation in the target space;
and when an event occurs indicating a situation in which the situation evaluation value satisfies a predetermined condition, generating superimposed image data by superimposing at least one of an image based on the text data and an image related to the situation on an image of the image data for a predetermined period including a period during which the event occurs.

REFERENCE SIGNS LIST 1, 1a Image processing system 10, 10a Image processing device 11 Acquisition unit 12, 12a Situation evaluation unit 13 Image evaluation unit 14 Voice evaluation unit 15 Voice recognition unit 16 Image processing unit 17 Image recording unit 18 Table storage unit 19, 19a Situation table 20, 20a Image processing mode table 30 Camera 40 Microphone

Claims (5)

An acquisition unit that acquires voice data and image data of a target space, which is an interior space of a moving body ;
a voice recognition unit for converting the voice data into text data;
a situation evaluation unit that evaluates a situation of the target space based on the image data and calculates a situation evaluation value of the situation in the target space;
an image storage unit that stores in advance an image when an abnormality occurs in the moving body;
an image processing unit that, when an event occurs indicating a situation in which the situation evaluation value satisfies a predetermined condition, generates superimposed image data by superimposing at least one of an image based on the text data and an image related to the situation on an image of the image data for a predetermined period including a period in which the event occurs , and records the superimposed image data in the image memory unit separately from images when an abnormality occurs in the moving body that have been previously recorded . The acquisition unit acquires image data of a person present in the target space,
the image processing unit generates an image in which the person is zoomed in from the image data, and generates superimposed image data in which at least one of an image based on the text data and an image related to the situation is superimposed on the zoomed-in image.
The image processing device according to claim 1 . The image processing device according to claim 1 , wherein the image processing unit determines an aspect of an image based on the text data to be superimposed on the image based on the situation evaluation value. a voice evaluation unit that calculates a voice evaluation value based on features acquired from the voice data;
The situation evaluation value includes the voice evaluation value.
The image processing device according to claim 1 . acquiring audio data and image data of a target space, which is a room of a moving body ;
converting the audio data into text data;
evaluating a situation in the target space based on the image data and calculating a situation evaluation value of the situation in the target space;
when an event occurs indicating a situation in which the situation evaluation value satisfies a predetermined condition, generating superimposed image data by superimposing at least one of an image based on the text data and an image related to the situation on an image of the image data for a predetermined period including a period during which the event occurs, and recording the superimposed image data in the image storage unit separately from images when an abnormality occurs in the moving body that have been recorded in advance in the image storage unit .

Images