JP2019125305A - Support device for creating teacher data - Google Patents

Abstract

To provide a teacher data creating support device which simplifies work of accurately creating teacher data containing a correspondence relationship between an image of a human face and expressions of the face.SOLUTION: In a teacher data creating support device, a support screen 300 displayed by a display processing unit includes: a first region 310 in which plural first images are displayed that are extracted from plural still pictures; a second region 320 in which plural second images are displayed that are time-sequential relative to one selected first image among the plural still pictures, and that fall within a predetermined range, when receiving a user's operation for selecting the one first image among the plural first images; and a third region 330 in which plural candidates of expressions are displayed that may be associated with a human face projected in the plural second images respectively.SELECTED DRAWING: Figure 3

Description

  Embodiments of the present invention relate to a teacher data creation support device.

  Conventionally, a learned model that outputs some information (labels) in response to image input is known, and generation of teacher data necessary for generating such a learned model by supervised learning is supported. Various techniques for achieving this have been studied.

JP, 2016-76073, A JP, 2010-91401, A

  However, in order to create the above teacher data accurately, finally, the user looks at the image visually to determine what label is to be associated with the image, and based on the determination result Since it is necessary to manually associate the image with the label, the work for creating teacher data is generally complicated.

  In particular, when correlating the image of a human face with the facial expression of the face, not only one image but also other images in time series before and after that are considered as a general image. Since it is necessary to visually analyze changes in facial expressions that are difficult to extract in processing and to determine what kind of correspondence is to be carried out, the work to create teacher data accurately in this case is very important. It is complicated.

  Therefore, one of the problems in the embodiment is teacher data creation support that can simplify the task of accurately creating teacher data including the correspondence between the image of a human face and the facial expression of the face. It is providing a device.

  The teacher data creation support device as an example of the embodiment is a teacher data creation support device for supporting creation of teacher data used for supervised learning, which is an acquisition unit for acquiring a moving image in which a human face is taken, An extraction unit for extracting a plurality of still images for each frame from a moving image, and a support screen for supporting association between human faces captured in each of the plurality of still images and facial expressions of the face A display processing unit for displaying on the screen, an operation reception unit for receiving an operation of the user via the operation input unit on the support screen, and an association processing unit for performing association in accordance with the user's operation on the support screen; In the support screen, a first region where a plurality of first images extracted from a plurality of still images are displayed, and a user operation for selecting one first image of the plurality of first images are accepted In the case A second region in which a plurality of second images within a predetermined range are displayed in time series with respect to the selected one first image among the still images of the second image and a plurality of second images And a third area in which a plurality of candidates for facial expressions that can be mapped to human faces are displayed.

  According to the above-described teacher data creation support device, it is possible to simplify the task of correctly creating teacher data including the correspondence between the image of a human face and the facial expression of the face by using the support screen.

  In the above-described teacher data creation support device, when a plurality of first images are input with an image in which a human face appears, a plurality of processes using an expression recognition library that outputs estimation results of the facial expression of the human face They are a plurality of images at a plurality of time points when a plurality of still images are extracted in time series and extracted on the basis of a result of execution on a still image in time series, and a change of a predetermined level or more is observed. According to such a configuration, the image and the expression can be associated with each other on the basis of the image that is highly likely to change in expression.

  Further, in the above-described teacher data creation support device, the plurality of first images are a plurality of images arranged in predetermined time intervals by viewing a plurality of still images in time series. According to such a configuration, the first image can be easily extracted.

  In the teacher data creation support device described above, the display processing unit is configured to be able to change the predetermined range for extracting the plurality of second images for each selected first image. According to such a configuration, for example, it is possible to make the number of second images to be extracted different in order to confirm changes in expression including time series before and after the time point corresponding to the first image. Thus, the image and the expression can be associated efficiently.

  In the teacher data creation support device described above, the association processing unit is configured to select one or more second images among the plurality of second images displayed in the second area according to the user's operation. When one of the plurality of candidates displayed in the third area is selected according to the user's operation, the association between the one candidate and one or more second images is collectively performed. carry out. According to such a configuration, it is possible to reduce the work load, unlike, for example, the case of individually associating the plurality of second images.

  In addition, in the teacher data creation support device described above, the acquisition unit acquires the biological information at each time point of the moving image of the human taken in the moving image together with the moving image, and the support screen displays the change in the biological information within a predetermined range Further includes a fourth area to be According to such a configuration, it is possible to more accurately determine the facial expression in consideration of the change in the biological information.

  Further, in the teacher data creation support device described above, the acquisition unit acquires a video inside the vehicle on which a human being is riding as a moving image, and acquires vehicle information including the traveling state of the vehicle at each time of the video. The support screen further includes a fifth area in which a change in vehicle information within a predetermined range is displayed. According to such a configuration, it is possible to more accurately determine the expression by further considering the change in the vehicle information of the vehicle.

FIG. 1 is an exemplary and schematic block diagram showing a hardware configuration of a teacher data creation support device according to the embodiment. FIG. 2 is an exemplary and schematic block diagram showing a functional configuration of the teacher data creation support device according to the embodiment. FIG. 3 is an exemplary and schematic diagram showing a screen configuration of a support screen provided by the teacher data creation support device according to the embodiment. FIG. 4 is an exemplary schematic flow chart showing the flow of processing when the teacher data creation support device according to the embodiment displays the support screen on the display unit. FIG. 5 is an exemplary schematic diagram showing a flow of processing when the teacher data creation support device according to the embodiment displays a detailed image on the support screen. FIG. 6 is an exemplary schematic diagram showing a flow of processing when the teacher data creation support device according to the embodiment carries out the correspondence in response to the operation on the support screen. FIG. 7 is an exemplary and schematic block diagram showing a configuration of a vehicle control system realized by using teacher data created by the teacher data creation support device according to the embodiment.

  Hereinafter, embodiments will be described based on the drawings. The configurations of the embodiments described below, and the operations and results (effects) provided by the configurations are merely examples, and the present invention is not limited to the following description.

  FIG. 1 is an exemplary and schematic block diagram showing the hardware configuration of the teacher data creation support device 100 according to the embodiment. The teacher data creation support device 100 is a device for supporting creation of teacher data used for supervised learning. As described below, the teacher data creation support device 100 has the same hardware configuration as a normal computer such as a PC (personal computer).

  As shown in FIG. 1, the teacher data creation support device 100 has a processor 101, a memory 102, an input / output interface (I / F) 103, and a storage 104 as a hardware configuration. These hardware configurations are communicably connected to one another via a bus 150.

  The processor 101 is formed of, for example, a CPU (central processing unit), and centrally controls the operation of each unit of the teacher data creation support device 100. Memory 102 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), etc., and realizes storage of data necessary for various processes executed by processor 101, provision of a work area of processor 101, etc. Do.

  The input / output interface 103 is an interface capable of connecting an input device for inputting information to the teacher data creation support device 100 and an output device for outputting information output from the teacher data creation support device 100. The input device is, for example, an operation input unit 151 such as a mouse or a keyboard, and the output device is, for example, a display unit 152 such as an LCD (liquid crystal display) or an OELD (organic electroluminescent display). It goes without saying that the configuration connectable to the input / output interface 103 is not limited to the operation input unit 151 and the display unit 152.

  The storage 104 is an auxiliary storage device configured by, for example, a hard disk drive (HDD) or a solid state drive (SSD).

  In the embodiment, teacher data is created by the user operating the teacher data creation support device 100 having the hardware configuration as described above.

  The teacher data according to the embodiment is a learned model that outputs an expression of the face (an emotion appearing in an expression) when information including an image showing an image of a human face on a vehicle is input (specific example) Is used for supervised learning to generate (described later).

  By the way, a learned model that outputs some information (label) associated with the image according to the input of the image is conventionally known, and is necessary when generating such a learned model by supervised learning Various techniques for assisting in the creation of teacher data are being considered.

  However, in order to create the above teacher data accurately, finally, the user looks at the image visually to determine what label is to be associated with the image, and based on the determination result Since it is necessary to manually associate the image with the label, the work for creating teacher data is generally complicated.

  In particular, in the case of an embodiment in which an image showing a human face is associated with an expression of the face, not only one image but also other images before and after it are taken into consideration. Because it is necessary to visually analyze changes in facial expressions that are difficult to extract in general image processing, and to determine what kind of correspondence is to be performed, to create teacher data accurately in this case The work is very complicated.

  Therefore, according to the embodiment, the teacher data creation support device 100 accurately creates teacher data including the correspondence between an image in which a human face appears and an expression of the face by providing the following function. Achieve simplification of work.

  FIG. 2 is an exemplary and schematic block diagram showing a functional configuration of the teacher data creation support device 100 according to the embodiment. The functional modules shown in FIG. 2 are realized as a result of the processor 101 executing software (control program) stored in the memory 102. In the embodiment, part or all of the functional module group shown in FIG. 2 may be realized by dedicated hardware (circuit).

  As illustrated in FIG. 2, the teacher data creation support device 100 has an acquisition unit 201, an extraction unit 202, a display processing unit 203, an operation reception unit 204, and an association processing unit 205 as functional components. have.

  The acquisition unit 201 acquires a moving image in which the face of a person riding a vehicle is taken, which is a source of creation of teacher data. In the embodiment, the acquisition unit 201 is a traveling state of the vehicle at the time of imaging of a moving image, such as human's biological information (heart rate, sweating condition, body temperature, center of gravity, body weight, etc.) Vehicle information including (speed, acceleration, accelerator pedal opening, brake pedal pressing force, time, position, etc.) can also be acquired together. The vehicle information may also include an imaging environment (such as the amount of ambient light) in the vehicle and a situation around the vehicle (such as traffic conditions and weather) at the time of capturing a moving image. These data are externally input to the teacher data creation support device 100 by, for example, wired or wireless communication.

  The extraction unit 202 extracts a plurality of still images from the moving image acquired by the acquisition unit 201 for each frame. That is, since the moving image is configured as a collection of a plurality of still images continuous in time series, the extraction unit 202 extracts a plurality of still images continuous in time series constituting the moving image from the moving image acquired by the acquiring unit 201. Extract the picture.

  The display processing unit 203 controls the display content of the display unit 152 (see FIG. 1). For example, the display processing unit 203 may use a support screen 300 for supporting association of the human face in each of the plurality of still images extracted by the extraction unit 202 with the facial expression of the face (see FIG. 3) is displayed on the display unit 152 (see FIG. 1). Although the details will be described later, the support screen 300 is provided with various GUIs (Graphical User Interfaces) for providing the user with an environment for achieving accurate association with a simple operation.

  The operation accepting unit 204 accepts an operation of the user via the operation input unit 151 (see FIG. 1). For example, the operation receiving unit 204 receives the user's operation on the support screen 300.

  The association processing unit 205 associates the human face captured in each of the plurality of still images with the facial expression of the face according to the user's operation on the support screen 300. In the embodiment, teacher data is created based on the result of the association performed by the association processing unit 205.

  Here, the support screen 300 provided in the embodiment will be specifically described with an example of the screen configuration.

  FIG. 3 is an exemplary and schematic diagram showing the screen configuration of the support screen 300 provided by the teacher data creation support device 100 according to the embodiment.

  As shown in FIG. 3, the support screen 300 according to the embodiment identifies the reproduction display area 301 in which the moving image is reproduced and displayed, and the reproduction section (time) of the moving image reproduced and displayed in the reproduction display area 301. And a playback / pause button 303 for executing playback / pause of the moving image in the playback / display area 301. According to these screen configurations, the user can perform desired association (association of an image and an expression) while visually recognizing the moving image acquired by the acquisition unit 201.

  In addition, the support screen 300 according to the embodiment is a schematic display in which a plurality of schematic images further extracted (spacingly) according to a predetermined standard from the plurality of still images extracted by the extraction unit 202 are displayed. Region 310 is included. The plurality of schematic images represent schematic changes along the time series of the plurality of still images. The outline image is an example of the “first image”, and the outline display area 310 is an example of the “first area”.

  For example, in the example illustrated in FIG. 3, among the plurality of outline images further extracted according to a predetermined standard from the plurality of still images extracted by the extraction unit 202, three images 311 as outline images corresponding to the section 302a. ... 312 are displayed side by side in time series in the schematic display area 310. In the embodiment, the section 302a corresponding to the range of the schematic image displayed in the schematic display area 310 can be switched manually or automatically.

  In the embodiment, the outline image is, for example, an expression recognition library that outputs an estimation result of the facial expression of a human face when an image including a human face is input to a plurality of still images extracted by the extraction unit 202. They are a plurality of images at a plurality of time points when the plurality of still images are viewed in time series and the estimation result of the expression changes, which is extracted based on the result of executing the process using. According to such a configuration, the image and the expression can be associated with each other on the basis of the image that is highly likely to change in expression.

  The expression recognition library used in the embodiment may be known. Further, in the embodiment, a configuration may be adopted in which the outline image is extracted on a certain basis without using the expression recognition library. Therefore, in the embodiment, the outline images may be, for example, a plurality of images arranged in predetermined time intervals by viewing a plurality of still images extracted by the extraction unit 202 in time series.

  Further, in the embodiment, the rough image may be extracted by calculating the amount of change of each feature amount of a plurality of predetermined feature points for a human face without using an expression recognition library. Good. In this configuration, for example, with respect to two still images at two time points adjacent to each other in time series, differences between feature amounts are taken between corresponding feature points, and a certain number of feature points whose value indicating the difference is equal to or more than a threshold By determining whether or not the above exists, it is determined whether or not there is a change in expression between two still images, and a still image determined to have a change in expression is extracted as a schematic image. It can be adopted. The number of feature points to be compared can be set to an arbitrary number such as 100, as long as the number is more than one.

  Furthermore, in the embodiment, when the plurality of extracted outline images do not fit within the outline display area 310, the cursor is superimposed on the area in the section display 302 corresponding to the section 302a, and the cursor is moved according to the movement. A configuration may be employed in which the range of the schematic image displayed in the schematic display area 310 is switched.

  Here, the support screen 300 according to the embodiment includes a screen configuration for confirming in more detail the situation before and after in the time series of each of the plurality of outline images displayed in the outline display area 310. More specifically, the support screen 300 according to the embodiment is extracted by the extraction unit 202 when the operation reception unit 204 receives an operation of the user for selecting one of the plurality of outline images. Among the plurality of still images, it includes a detail display area 320 in which a plurality of detail images within a predetermined range in time series with respect to one selected outline image is displayed. The detailed image is an example of the “second image”, and the detailed display area 320 is an example of the “second area”.

  For example, in the example shown in FIG. 3, in the detail display area 320, nine images 321 to 329 which are positioned before and behind the image 311 as the outline image displayed in the outline display area 310 are detailed It is displayed as an image. Thereby, the user sees the image 311 displayed as the outline image together with the nine images 321 to 329 displayed as the detailed image, thereby changing the expression in time series of the predetermined range before and after the image 311 (transition ), And based on the confirmation result, the expression can be accurately identified. In the embodiment, the number of detail images displayed in the detail display area 320 is not limited to nine as in the example shown in FIG.

  In the embodiment, the predetermined range from which the detailed image is extracted may be different for each selected one of the schematic images. For example, in the embodiment, the range from which the detailed image is extracted may be changed according to the estimation result of the expression corresponding to one selected outline image. According to such a configuration, it is possible to extract more detailed images, for example, as the necessity for confirming changes in facial expressions including the time series before and after in greater detail is increased. The association can be performed efficiently.

  In addition, the support screen 300 according to the embodiment includes a screen configuration for receiving an operation of association by the user. More specifically, in the support screen 300 according to the embodiment, a plurality of expressions of facial expressions that can be associated with human faces captured in each of the plurality of detail images displayed in the detail display area 320 are displayed. The candidate display area 330 is included. The candidate display area 330 is an example of the “third area”.

  For example, in the example shown in FIG. 3, in the candidate display area 330, as a screen configuration representing a plurality of candidates, a button 331 corresponding to an indeterminate emotion (neutral) and a button 332 corresponding to an emotion (happiness) And a button 333 corresponding to the sadness of sadness is displayed. Needless to say, buttons representing emotions other than these may be displayed in the candidate display area 330.

  Here, in the embodiment, the plurality of candidates displayed in the candidate display area 330 in a state where one or more of the plurality of detail images displayed in the detail display area 320 are selected according to the user's operation. Is configured to collectively carry out the correspondence between the selected one candidate and the selected one or more detail images when one of the candidates is selected according to the user's operation. ing.

  For example, in the embodiment, the plurality of detail images displayed in the detail display area 320 are in a state of being selected by default. Then, the user selects, from among the plurality of detail images displayed in the detail display area 320, a detail image that the user does not want to perform the same correspondence as the other images, and the target of the selected detail image By excluding one from the above and selecting one candidate in the candidate display area 330 on that, the association processing unit 205 is made to carry out the collective association. According to such a configuration, it is possible to reduce the work load, unlike, for example, the case of individually associating the plurality of detailed images.

  Furthermore, the support screen 300 according to the embodiment includes a screen configuration for displaying information for assisting the determination of the facial expression. More specifically, the support screen 300 according to the embodiment includes the reference information display area 340 in which changes in the biological information and the vehicle information (such as the traveling state) acquired by the acquisition unit 201 are displayed. The reference information display area 340 is an example of the “fourth area” and an example of the “fifth area”.

  For example, in the example illustrated in FIG. 3, in the reference information display area 340, a heart rate as an example of biological information within a period (predetermined range) corresponding to a plurality of detailed images displayed in the detail display area 320. A graph 341 representing a change and a graph 342 representing a change in vehicle speed as an example of vehicle information (e.g., a traveling state) within the period are displayed. According to such a configuration, it is possible to more accurately determine the facial expression of the human face shown in the detailed image, further considering changes in the biological information and the vehicle information.

  In the example shown in FIG. 3, the change in biological information (graph 341) and the change in vehicle information (graph 342) are both displayed in the same area (reference information display area 340), but in the embodiment, The change in biological information and the change in vehicle information may be displayed in separate areas. Further, in the embodiment, as described above, the acquisition unit 201 can also acquire, as vehicle information, the imaging environment in the car, the situation around the vehicle at the time of imaging a moving image, etc. It can be displayed.

  Based on the above configuration, the teacher data creation support device 100 according to the embodiment executes the following process in order to support the creation of teacher data by the user.

  FIG. 4 is an exemplary schematic flow chart showing the flow of processing when the teacher data creation support device 100 according to the embodiment displays the support screen 300 on the display unit 152. The process flow shown in FIG. 4 is executed, for example, when the user inputs an operation for calling the support screen 300 to create teacher data to the teacher data creation support device 100 via the operation input unit 151. .

  In the processing flow shown in FIG. 4, first, in S401, the teacher data creation support apparatus 100 (for example, the acquisition unit 201) receives an animation of the face of a person riding a vehicle, which is the basis of creation of teacher data. Or, human's biological information captured in the moving image, vehicle information including the traveling state of the vehicle at the time of capturing the moving image, and the like are acquired.

  Then, in S402, the teacher data creation support device 100 (for example, the extraction unit 202) extracts a plurality of still images for each frame from the moving image acquired in S401.

  Then, in S403, the teacher data creation support device 100 (for example, the display processing unit 203) generates a predetermined outline image to be displayed in the outline display area 310 of the support screen 300 described above from the plurality of still images extracted in S402. Extract according to the criteria. As described above, the rough image may be extracted based on the estimation result of the expression by the expression recognition library, or may be extracted based on a certain reference regardless of the estimation result of the expression.

  Then, in S404, the teacher data creation support device 100 (for example, the display processing unit 203) displays, on the display unit 152, the support screen 300 including the schematic display area 310 on which the schematic image extracted in S403 is displayed. Then, the process ends.

  FIG. 5 is an exemplary schematic diagram showing the flow of processing when the teacher data creation support device 100 according to the embodiment displays a detailed image on the support screen 300. As shown in FIG. The process flow shown in FIG. 5 is executed, for example, after the support screen 300 is displayed on the display unit 152 by the series of processes shown in FIG. 4 described above.

  In the processing flow shown in FIG. 5, first, in S501, the teacher data creation support device 100 (for example, the display processing unit 203) receives the user's operation (the operation on the schematic display area 310 of the support screen 300) received by the operation receiving unit 204. It is determined whether or not one of the plurality of outline images displayed in the outline display area 310 is selected according to.

  If it is determined in S501 that one of the outline images is not selected, the process ends. On the other hand, when it is determined in S501 that one of the plurality of outline images is selected, the process proceeds to S502.

  Then, in S502, the teacher data creation support device 100 (for example, the display processing unit 203) causes the one outline image selected in S501 to be within a predetermined range in time series from a plurality of still images constituting the moving image. Extract certain detailed images.

  Then, in S503, the teacher data creation support device 100 (for example, the display processing unit 203) displays the detail image extracted in S502 in the detail display area 320 of the support screen 300.

  Then, in S504, the teacher data creation support device 100 (for example, the display processing unit 203) displays the change of the biological information and the vehicle information in the period corresponding to the detail display area 320 displayed in S503 as the reference information display of the support screen 300. Displayed in the area 340. Then, the process ends.

  FIG. 6 is an exemplary and schematic diagram showing a flow of processing when the teacher data creation support device 100 according to the embodiment performs the correspondence in accordance with the operation on the support screen 300. The processing flow shown in FIG. 6 is executed, for example, after the detail image is displayed in the detail display area 320 by the series of processing shown in FIG. 5 described above.

  In the processing flow shown in FIG. 6, first, in S601, the teacher data creation support apparatus 100 (for example, the association processing unit 205) receives the user's operation (the support screen 300 for the candidate display area 330) received by the operation receiving unit 204. In accordance with the operation), it is determined whether or not the expression candidate is determined (selected).

  If it is determined in S601 that no expression candidate has been determined, the process ends. On the other hand, when it is determined in S601 that the expression candidate is determined, the process proceeds to S602.

  Then, in step S602, the teacher data creation support apparatus 100 (for example, the association processing unit 205) performs, on the basis of the facial expression candidates determined in step S601, collective association with the currently selected detailed image. Note that, as described above, the currently selected detailed image is one of the plurality of detailed images displayed in the detail display area 320 that is not excluded from the target of association by the operation of the user. In the embodiment, teacher data is created based on the result of the matching in S602. Then, the process ends.

  As described above, the teacher data creation support device 100 according to the embodiment includes the display processing unit 203 that displays the support screen 300 including at least the following screen configuration on the display unit 152. The support screen 300 includes a schematic display area 310 in which a plurality of schematic images extracted according to a predetermined standard from a plurality of still images constituting a moving image, and a user's user for selecting one of the plurality of schematic images. Among the plurality of still images, a detail display area 320 displaying a plurality of detail images within a predetermined range in time series with respect to a selected one of the plurality of still images, when the operation is accepted; And a candidate display area 330 in which a plurality of expressions of facial expressions that can be mapped to human faces captured in each of the detail images are displayed.

  According to the above configuration, the support screen 300 accurately creates teacher data including the correspondence between the image of the human face (present in the vehicle) and the facial expression of the face. Can be simplified.

  In the embodiment, the plurality of outline images are extracted based on the result of executing the expression recognition library on the plurality of still images constituting the moving image, and the plurality of still images are viewed in time series It is a plurality of images at a plurality of time points when a change of a predetermined level or more is found in the estimation result of the expression. According to this configuration, the image and the expression can be associated with each other on the basis of the image that is highly likely to change in expression.

  In the embodiment, the plurality of outline images may be a plurality of images which are arranged at predetermined time intervals by viewing a plurality of still images in time series. With this configuration, it is possible to easily extract a schematic image.

  Further, in the embodiment, the display processing unit 203 is configured to be able to change a predetermined range for extracting a plurality of detailed images for each selected one of the outline images. For example, in the embodiment, the range from which the detailed image is extracted may be changed according to the estimation result of the expression corresponding to one selected outline image. This configuration makes it possible to make the number of detailed images to be extracted different in order to confirm changes in expression including the time series before and after the time point corresponding to the outline image, for example. It is possible to extract more detailed images on the basis of the rough image, as the rough image needs to confirm the change of the facial expression including the time series before and after in detail in detail. As a result, it is possible to efficiently associate the image with the expression.

  Further, in the embodiment, the association processing unit 205 displays the candidate display area 330 in a state where one or more of the detail images among the plurality of detail images displayed in the detail display area 320 are selected according to the user's operation. When one candidate among the displayed plurality of candidates is selected according to the user's operation, the association between the one candidate and one or more detail images is collectively performed. According to such a configuration, it is possible to reduce the work load, unlike, for example, the case of individually associating the plurality of detailed images.

  In the embodiment, the support screen 300 further includes a reference information display area 340 in which changes in biological information and vehicle information within a predetermined range corresponding to a plurality of detailed images are displayed. According to this configuration, it is possible to more accurately determine the expression by further considering changes in the biological information and the vehicle information.

  Hereinafter, a technology using teacher data created by the teacher data creation support device 100 according to the embodiment will be briefly described. By using the teacher data created in the embodiment, a vehicle control system 700 as shown in the following FIG. 7 can be configured.

  FIG. 7 is an exemplary and schematic block diagram showing a configuration of a vehicle control system 700 realized by utilizing the teacher data created by the teacher data creation support device 100 according to the embodiment. The vehicle control system 700 is mounted on a vehicle.

  As shown in FIG. 7, the vehicle control system 700 includes a vehicle control device 710, a sensor group 720, and a control target 730.

  Vehicle control device 710 is, for example, an ECU (Electronic Control Unit) mounted on a vehicle. The ECU is generally configured as a microcomputer having hardware similar to that of a normal computer such as a processor and a memory.

  The sensor group 720 is a collection of various sensors provided in the vehicle. In the example illustrated in FIG. 7, the sensor group 720 includes an in-vehicle camera 721 that captures an in-vehicle situation, a biometric information sensor 722 that detects human biometric information present in the vehicle, and a traveling state in which the traveling state of the vehicle is detected. And a sensor 723.

  The control target 730 is various equipment of the vehicle controlled by the vehicle control device 710. The control target 730 provides various services (air conditioning, music, etc.) in the vehicle as well as systems related to vehicle travel such as acceleration systems including accelerators, braking systems including brakes, steering systems including steering etc. May also include a system for

  Here, the vehicle control device 710 includes an information acquisition unit 711, an expression determination unit 712, and a control unit 713. These configurations may be realized as a result of the processor executing software (control program) stored in the memory, or may be realized by dedicated hardware (circuit).

  The information acquisition unit 711 acquires information from the sensor group 720. Then, the facial expression determination unit 712 determines the facial expression of the human face in the car based on the information acquired by the information acquisition unit 711.

  More specifically, the facial expression determination unit 712 includes a learned model 712 a generated by supervised learning based on the teacher data generated by the teacher data generation support device 100 according to the embodiment. Since this learned model 712a uses the technology according to the embodiment, the image (moving image or still image) acquired by the in-vehicle camera 721, the biological information acquired by the biological information sensor 722, and the traveling state sensor According to the input of the traveling state acquired by 723, (the estimation result of) the facial expression of the human face currently existing in the car is output.

  Then, in accordance with the determination result of the expression determination unit 712, the control unit 713 controls the control target 730 so as to realize provision of appropriate traveling and services that match the human facial expression currently existing in the car. Control. This makes it possible to provide a comfortable vehicle in line with human emotions.

  In the example illustrated in FIG. 7, the sensor group 720 includes a configuration for detecting vehicle information other than the traveling state, such as an imaging environment in a car, a situation around the vehicle at the time of imaging a moving image, and the like. It is also good. Then, the facial expression determination unit 712 may be configured to determine the facial expression in consideration of all the vehicle information.

  As mentioned above, although embodiment of this invention was described, embodiment mentioned above is an example to the last, and limiting the scope of invention is not intended. The novel embodiments described above can be implemented in various forms, and various omissions, substitutions, and modifications can be made without departing from the scope of the invention. The above-described embodiments and the modifications thereof are included in the scope and the gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

100 teacher data creation support device 151 operation input unit 152 display unit 201 acquisition unit 202 extraction unit 203 display processing unit 204 operation reception unit 205 association processing unit 300 support screen 310 schematic display area (first area)
320 Detail display area (second area)
330 candidate display area (third area)
340 Reference information display area (fourth area, fifth area)

Claims (7)

A teacher data creation support device for supporting creation of teacher data used for supervised learning, comprising:
An acquisition unit that acquires a moving image of a human face;
An extraction unit for extracting a plurality of still images from the moving image for each frame;
A display processing unit that displays on a display unit a support screen for supporting association of the human face captured in each of the plurality of still images with the facial expression of the face;
An operation receiving unit that receives a user's operation via the operation input unit on the support screen;
An association processing unit that carries out the association according to the user's operation on the support screen;
Equipped with
The support screen is
A first area in which a plurality of first images extracted from the plurality of still images are displayed;
When the user's operation for selecting one of the plurality of first images is accepted, time may be selected with respect to the selected one of the plurality of still images. A second area in which a plurality of second images within a predetermined range in series are displayed;
A third area in which a plurality of candidates for the facial expression that can be associated with the human face captured in each of the plurality of second images are displayed;
including,
Teacher data creation support device. When the plurality of first images receives an image including the human face, processing is performed on the plurality of still images using an expression recognition library that outputs the estimation result of the expression of the human face. It is a plurality of images at a plurality of time points when the plurality of still images extracted based on the result are seen in time series and the estimation result of the expression is changed in a predetermined amount or more.
The teacher data creation support device according to claim 1. The plurality of first images are a plurality of images in which the plurality of still images are viewed in time series and arranged at predetermined time intervals.
The teacher data creation support device according to claim 1. The display processing unit is configured to be able to change the predetermined range for extracting the plurality of second images for each of the selected first images.
The teacher data creation assistance apparatus of any one of Claims 1-3. The association processing unit is displayed in the third area in a state where one or more second images among the plurality of second images displayed in the second area are selected according to the user's operation. When one candidate among the plurality of candidates is selected according to the operation of the user, the association between the one candidate and the one or more second images is collectively performed. ,
The teacher data creation assistance apparatus of any one of Claims 1-4. The acquisition unit acquires, together with the moving image, biological information at each time point of the moving image of the human captured in the moving image,
The support screen further includes a fourth area in which a change in the biological information within the predetermined range is displayed.
The teacher data creation assistance apparatus of any one of Claims 1-5. The acquisition unit acquires, as the moving image, an image of the inside of a vehicle on which the human being is riding, and acquires vehicle information including a traveling state of the vehicle at each time of the image.
The support screen further includes a fifth area in which a change in the vehicle information within the predetermined range is displayed.
The teacher data creation assistance apparatus of any one of Claims 1-6.

Images