TW202320551A - Music recommendation system and method based on image recognition and situational state - Google Patents

Abstract

A music recommendation system based on image recognition and situational status including: a training music database module; a situational image database module; an audio-visual training module; a scoring record module; a cloud database module, and the cloud database module includes a user database and a music file data; an image recognition module; an integrated analysis module; and a music recommendation module. A music recommendation method based on image recognition and contextual state, including: a step of creating audio and video data, a step of training of audio and video; a step of scoring preferences; a step of image recognition and classification; a step of selecting music styles; and a feature value ratio and recommended steps for music. Through the music recommendation system and method based on image recognition and situational state of the present invention, users are allowed to watch and listen to each situational image and training music track through the audio-visual training module in advance, and then score and record them. Then, when the user is driving, according to the scoring result, the integrated analysis module and the music recommendation module are calculated and processed, and the music tracks for the user's current situational is selected from a music file database and played.

Description

Music recommendation system and method based on image recognition and situational state

The present invention relates to a music recommendation system and method, in particular to a method for judging the driver's current situational state through an image recognition method, and recommending from a music file database that is suitable for the driver's current situational state to listen to based on image recognition and situational state Music recommendation system and method.

In recent years, music streaming users have grown significantly. It can be seen that people's demand for listening to music is increasing day by day. The current music streaming media lacks a suitable recommendation method to suggest music that is suitable for users to listen to. In the past, algorithms were used to recommend Music can only infer the type of music the user might like.

Known technology Patent No. M523146 of the Republic of China is a smart music and digital content player, device and system that uses brainwave feedback to intelligently select the most suitable music or digital content in the music or digital database based on the current brainwave signal. and play the music content. In this way, music or digital content suitable for the user's current state of mind can be selected and played according to the user's brain wave signal, so as to achieve the purpose of relieving stress or promoting activities.

However, the above-mentioned conventional technology does not disclose the analysis of the driver's situational state. Through the prior training and scoring results, the accuracy of recommending suitable music tracks is improved for the driver to listen to at the moment, satisfying the driver's understanding of traffic conditions and weather. When causing emotional impact, it can provide soothing music tracks to maintain a better driving spirit and improve driving safety. Moreover, the conventional technology does not have a better music recommendation technology for the situational state encountered by the driver at the moment of driving to meet the driver's current emotional needs. In addition to playing fixed music tracks on the car or listening through the radio, and There is no practical way to improve the accuracy of music recommendation through prior simulation training, so it is necessary to improve the known technology.

In addition, although another patent No. I410811 is related to music recommendation, there is no corresponding identification and recommendation for driving vehicles. Although patents such as I699186 and I723853 are related to emotions and brain waves, there is no corresponding vehicle condition recognition and recommendation. There is also patent No. I71710, which is based on image recognition for the Internet of Vehicles, but does not involve music recommendation or human emotions.

In view of this, the present invention recommends music that is more suitable and in line with the current expectations of the vehicle driver by collecting the rating records of the user after watching the situational video and listening to the music, and combining the music feature value similarity comparison technology. A scoring mechanism for music recommendation is added to optimize the recommendation results.

An object of the present invention is to provide a music recommendation system and method based on image recognition and situational state, which can provide suitable information for the driver based on the results obtained by the driver through audio-visual training in advance, and after judging the current situational state while driving through image recognition. A function of the music track you are listening to while driving.

Another object of the present invention is to provide a music recommendation system and method based on image recognition and situation status, which can provide drivers with the ability to record favorite music tracks corresponding to the situation according to the current situation of driving, so that they can be directly used as music in the future Function to play tracks.

Another object of the present invention is to provide a music recommendation system and method based on image recognition and situational status, which has the function of providing the driver with a re-rating mechanism for recommended music to optimize the recommendation results to achieve more accurate recommended music tracks.

In order to achieve the above and other purposes, the music recommendation system based on image recognition and situational status of the present invention includes: a training music database module storing multiple training tracks classified according to music types; a situational image database module, Store multiple situational images classified according to situational types; connect the training music database module and the situational image database module, access the training tracks in the training music database module and the situational video database module Situational images in the group, wherein each situational image is matched with at least one training track to be played at the same time; a score recording module is connected to the audio-visual training module to provide the user with the training track played according to the music video training module and the scene image for preference scoring and recording; a cloud database module, connected to the scoring record module, the cloud database module includes a user database and a music file database, wherein the user database Store user data and the user's rating data, the music file database stores music tracks of multiple types of music genres; an image recognition module, which recognizes and classifies the situation and state according to the user's current climate and road condition characteristics while driving, the The image recognition module includes an image deep learning unit to improve the recognition accuracy; an integrated analysis module connects the cloud database module and the image recognition module, and classifies the results according to the situational status currently recognized by the image recognition module performing integrated analysis and comparison with the user database of the cloud database module, and generating a screened music track data table; and a music recommendation module, connecting the cloud database module and the integrated analysis module, and Search and compare the music characteristics of the training tracks of the screened music track data table with the music file database of the cloud database module, and generate a listening experience suitable for the user's current driving situation from the music file database A playlist of recommended music tracks for playback and listening.

In order to achieve the above and other objects, the music recommendation method based on image recognition and situational status of the present invention includes: an audio-visual data archiving step, storing multiple training tracks and multiple situational images in a training music database module and In a scene image database module, wherein, the plurality of training tracks are classified into a plurality of music genres; an audio-visual training step provides an audio-visual training module for the user to operate, and the audio-visual training module is respectively obtained from the training music In the database module and the situational image database module, the training tracks and the situational images are extracted, and the user is provided to click on the situational images and the training tracks to play at the same time, allowing the user to watch and listen to the execution training, wherein each When the scene image is played, at least one of the plurality of training tracks is matched at the same time; a preference scoring step, after the user completes each audio-visual training in the audio-visual training step, a scoring record module is provided for the user to evaluate the current Second, score the situational image screen and the preferences and feelings of the corresponding training tracks, and store the user information and its scoring records in a user database for future use, wherein the user can set the scoring standard threshold; 1. Image recognition and classification The step of using an image recognition module to identify and classify the climate and road condition characteristics of the user while driving, and classify the climate and road condition characteristics into one of the plurality of situation images; a music genre screening step, based on Find out the user information and scoring records from the user database for the situational image status judged in the image recognition and classification step, and find out the corresponding track from the plurality of training tracks through the set scoring standard threshold , generating a screened music track data table; and a feature value comparison and music recommendation step, sequentially analyzing the music features of the training tracks in the screened music track data table, and searching a music file database, from the music file database Find a music track similar to the music feature corresponding to the training track in the filtered music track data table as a recommended music track, and use the recommended music track as a playlist for the user to play and listen to.

In some embodiments of the present invention, the image recognition module further includes an image data enhancement unit, which has data enhancement technology including limited contrast adaptive histogram equalization technology or image enlargement technology.

In some embodiments of the present invention, the image recognition module further includes a vehicle speed detection unit to detect and record the vehicle speed state.

In some embodiments of the present invention, the music recommendation module further has a favorite music storage unit.

In some embodiments of the present invention, the music recommendation module further has a recommended track scoring unit.

In some embodiments of the present invention, the image recognition and classification step further includes an image data enhancement step to enhance the image data to increase the accuracy of situational state recognition.

In some embodiments of the present invention, the image recognition and classification step further includes a vehicle speed detection step, which detects the current vehicle speed state during driving, and assists the accuracy of situational state recognition and classification.

In some embodiments of the present invention, there is also a step of recording favorite music, allowing the user to record the favorite music that he has heard before, so that he can listen to it again in the same situation.

In some embodiments of the present invention, a recommended track scoring step is further included, which provides users with ratings for the recommended tracks to optimize the accuracy of future recommended tracks.

FIG. 1 is a structural diagram of an embodiment of a music recommendation system based on image recognition and situation status according to the present invention, please refer to FIG. 1 . The music recommendation system based on image recognition and situation status of the present invention includes: a training music database module (10), a situation image database module (20), an audio-visual training module (30), and a scoring record module (40), a cloud database module (50), an image recognition module (60), an integrated analysis module (70) and a music recommendation module (80). Wherein, the training music database module (10) stores a plurality of training tracks classified according to music types, such as blues, country, folk, jazz, Latin, new age, pop, rap, rock, classical, musical instruments Performance, vocal music and dance music are divided into 13 types, and 10 pieces of music are collected for each type as training repertoire.

The situational image database module (20) stores multiple situational images classified according to the type of situation, and the multiple situational images can be classified and recorded according to actual needs according to climate or environmental conditions, such as: day or night, sunny or rainy, road Situation states such as smooth or blocked roads, urban roads or country roads, and various types of situational images are recorded and stored in the situational image database module (20).

Preferably, in this embodiment, the situational image database module (20) stores eight kinds of situational image files in total, including: a daytime-sunny day-smooth situational image, a daytime-sunny day-Yongsai situational image, a Daytime-Rainy-Smooth Situation Image, One Day-Rainy-Yongsai Situational Image, One Night-Night Clear-Smooth Situational Image, One Night-Yeqing-Yongsai Situational Image, One Night-Rainy-Smooth Situational Image and One Night -Rainy day-Yongsai situational video mainly records the above eight kinds of situational videos that may be encountered while driving based on day and night, weather, rain, and smooth road congestion. Generally speaking, when driving during the day and night, as well as on sunny and rainy days, due to the relationship between spirit and sight, it is more likely to cause different ups and downs in driving emotions. Anxiety factor. Therefore, using the above eight kinds of situational images can be more realistically used as situational images for simulation training, so as to be more in line with the driver's current situational state.

The audio-visual training module (30) is connected to the training music database module (10) and the situational image database module (20), accessing the training tracks and playing in the training music database module (10) The situational images in the situational imagery database module (20), wherein each situational image is matched with at least one training track to be played simultaneously. The audio-visual training module (30) can be a VR (Virtual Reality) virtual reality audio-visual device or audio-visual playback equipment for users to perform simulation training of "seeing" and "listening". For example: if there are M sets of situational images and N training tracks are selected, each user will generate MxN sets of audio-visual training records. In addition, the length of the audio-visual file for each playback can be set by yourself. For example, 30 seconds of playback is used as a training time. During the playback process, the user can experience the current mood through the video scene and music melody to determine whether they like it or not. degree of judgment.

The score recording module (40) is connected to the audio-visual training module (30), and provides users with preference scoring according to the training track and the situational image played by the music video training module (30). Since the situational image screen is to simulate the situational state that the user may encounter when driving, in this embodiment, for example, playing the daytime-rainy day-Yongsai situational image is to simulate the situation when the driver drives in the daytime. When it is raining and the road condition is a traffic jam, of course, the matching music training repertoire can be automatically selected by the audio-visual training module (30) or selected arbitrarily by the user or according to the type of music.

When the user watches each scene video and listens to the matching training track music, there will be different emotions and a feeling of liking or dislike for the current music. After the video file is played, the scoring record module (40) Provide users with ratings for each piece of music, such as 1 to 5 points, the higher the score, the more they like it. After the user gives a rating, the rating recording module (40) will record the user's rating for the day-rainy day - Score the situational video of Yongsai and its matching training tracks, which will be used as a reference for subsequent music recommendations. Preferably, the scoring record module (40) provides users with the ability to set the scoring standard threshold. For example, if the recommended music scoring standard threshold is set to 4 points, then the training tracks with a score of 4 points or more will be included in the recommended music reference. Scores of 1 to 3 will only be included in the score record, and will not be included in the recommended music reference.

The cloud database module (50) is connected to the scoring record module (40), and the cloud database module (50) includes a user database (51) and a music file database (52), wherein the The user database (51) stores user data and rating data of the user, and the music file database (52) stores music tracks of multiple types of music genres. The user data includes user registration account, name, audio-visual training time and login time... etc. The user rating data includes the record when the user uses the rating record module (40), including each situational image and its The score record and the date and time of the score record of each corresponding training track can be updated by the score record time to the degree of preference for the training track. The music track stored in the music file database (52) is used as A music file library for recommended music, and music file data can be added, deleted or modified at any time.

The image recognition module (60) performs situational state recognition and classification according to the climate and road condition characteristics of the user's current driving, and the image module (60) includes an image deep learning unit (61) to improve the recognition accuracy. When the user is driving, the image recognition module (60) is used to analyze and judge the environmental road condition image captured ahead, including the brightness of the light, rain or shine, the distance to the vehicle ahead, and the number of vehicles ahead, etc. to judge the current state of the situation. The image recognition module (60) can capture image frames through the camera lens of the driving recorder on the car, and the image deep learning unit (61) is used as an image training neural network model for image recognition, and is used as an image recognition model. for deep learning training.

In order to avoid an excessively high error rate of image recognition results during actual execution, the image deep learning unit (61) of the image recognition module (60) uses the ResNet50 framework for training to improve recognition accuracy. ResNet50 is a deep learning architecture. In order to avoid the problems of overfitting and training degradation, through the residual module of ResNet50, as the number of training increases, a deeper neural network can be trained, and the recognition accuracy can be improved at the same time. More than 85%. In this embodiment, by collecting 6,000 daytime and night situation image pictures as training data, 20,000 sunny and rainy situation image pictures as training data, and 9,000 Yongsai and smooth situation image pictures as training data, including : Daytime-Sunny-Smooth Situation Image, Day-Sunny-Yongsai Situational Image, Day-Rainy-Smooth Situational Image, Day-Rainy-Yongsai Situational Image, Night-Night Clear-Smooth Situational Image, Night -Sunny night-Yongsai situational image picture, night-rainy day-smooth situational image picture and night-rainy day-Yongsai situational image picture, a total of eight categories of pictures of the situational state of climate and road conditions are used as data for deep learning.

FIG. 2 is a structural diagram of another embodiment of the music recommendation system based on image recognition and situation status of the present invention, please refer to FIG. 2 . In order to further improve the recognition accuracy of the image recognition module (60), the image recognition module (60) further includes an image data enhancement unit (62), which has data enhancement technology including limited contrast adaptive histogram equalization technology or Image magnification technology. The image data enhancement unit (62) has two kinds of data enhancement techniques, the first is Contrast Limited Adaptive Histogram Equalization (CLAHE for short), when the foreground or background of the video picture is too bright or too dark When using this method, the brightness can be better distributed on the image, and it can avoid the cloudy sky on a sunny day, which is easily misjudged as a rainy day. This method is suitable for the judgment and identification of sunny days and rainy days. In this embodiment, the originally collected image data is enhanced, so that 6,000 original images plus 6,000 enhanced images, a total of more than 12,000 image data are used for deep learning training, which can improve the accuracy and maintain it at more than 85%. .

Continuing from the above, the second data enhancement technology is the image enlargement method. In order to avoid the insufficiency of the deep learning model in some situations or the lack of data, this method is used to increase the learning data to increase the recognition The accuracy rate, this method is suitable for the judgment and identification of smooth road conditions and congestion. The enlargement method adopts the method of cropping the image, but the size specification of the original image will be preserved. For example, when cropping an image with a width of 1280 pixels and a height of 720 pixels, the cropped size will still maintain a width of 1280 pixels and a height of 720 pixels. The size will not be reduced, but the image will be enlarged. When the same image is scaled and then used as training data, although the pre-processed image and the processed image are essentially the same image, they are completely different data for the machine. Therefore, by using this method to increase training The amount of data can improve the recognition accuracy and maintain it above 85%.

Continuing from the above, in this embodiment, since this state is located at the bottom of the tree diagram of the learning model, it means that the more situational conditions need to be met, the more difficult it is to find the collected data, so the originally collected The video data are the situation of smooth and sunny day and congested sunny day, the situation of smooth and rainy day and congested rainy day, the situation of smooth and sunny night and congested night, and the situation of smooth and congested night rain and night rain. After data enhancement, the above four There will be 6,000 original images plus 6,000 enhanced images for each situational state. There will be 12,000 images for each situational state, and the recognition accuracy will remain above 85%.

Please continue to refer to Figure 2. Preferably, the image recognition module (60) further includes a vehicle speed detection unit (63) to detect and record the vehicle speed state. In order to increase the smoothness of road conditions or the recognition accuracy of Yongsai, the vehicle speed detection unit (63) can be used to detect and record the speed status to improve the recognition accuracy, especially when the front screen is difficult to identify smooth or Yongsai Under the situation, for example the line of sight picture is not clear or the congested situation that road construction causes, rather than the Yong congested situation when the general traffic volume is much congested, wherein, this vehicle speed detection unit (63) can be a GPS positioning system or It is electrically connected with a driving navigation device, and the driving speed is estimated based on the distance and time spent in moving the position. For example, if the vehicle speed remains below a certain speed within a short period of time, for example, the vehicle speed remains below 40 km/h within five minutes, or the vehicle speed changes from low speed to high speed and then from high speed to low speed for more than 5 times within ten minutes, And if the speed difference is 10-30 km/h, it means that the vehicle is in a stop-and-go situation, and all of the above are traffic jams. By collocating the vehicle speed detection unit (63), the judgment accuracy of road congestion or smooth situations can be improved.

Please continue to refer to Figure 1. The integrated analysis module (70) is connected to the cloud database module (50) and the image recognition module (60), according to the situation status classification result currently recognized by the image recognition module (60) and the cloud data The user database (51) of the library module (50) is analyzed and compared, and a screened music track data table is generated. After the image recognition module (60) has identified the driver's current situation, the recognition result will determine which situation image belongs to the situation image database module (20). In this embodiment, there are eight types Situational state, respectively one day-sunny-smooth situational state, one day-sunny-yongsai situational state, one day-rainy-smooth situational state, one day-rainy-yongsai situational state, one night-night sunny-smooth Situational state, one night-night clear-Yongsai situational state, one night-rainy day-smooth situational state and one night-rainy day-Yongsai situational state.

For example, if the recognition result of the image recognition module (60) is determined to belong to the daytime-rainy-smooth situational state, then the integrated analysis module (70) through a network connection such as a 4G or 5G communication system, Using the user database (51) for analysis and comparison, the integrated analysis module (70) is based on the identity of the driver (i.e. the user) and his previous use of the scoring record module (40) and the audio-visual training module Analyze the audio-visual training and scoring records made by group (30), find out the scores of each training track that was originally matched with the daytime-rainy-smooth scene image, and filter according to the scoring setting of the scoring record module (40) The screened music track data table is generated, wherein the screened music track data table is a training track that has previously reached the threshold value of the scoring standard.

The music recommendation module (80) is connected to the cloud database module (50) and the integrated analysis module (70), and the training track of the filtered music track data table is connected with the cloud database module (50) The music file database (52) performs music feature search and comparison, and generates a playlist of recommended music tracks suitable for the user to listen to in a situational state while driving from the music file database (52) for playing and listening. The music features include melody, rhythm and chord, etc. In this embodiment, the music feature extraction method uses Mel cepstral coefficients (MFCC). Mel cepstrum coefficients include two important sub-steps, namely Mel frequency analysis and cepstrum analysis. The frequency, amplitude and phase of the audio signal generated through the music clip determine the characteristics of the audio signal. According to the music track data table According to the music features of the training track, find out the music track that the music feature matches from the music file database (52) as the recommended music track, and organize it into a playlist for the music player to play. Of course, the number and selection method of the recommended music tracks can be set according to user settings, for example, 30 matching recommended music tracks are randomly selected each time as a playlist.

Please continue to refer to Figure 2. Preferably, the music recommendation module (80) also has a favorite music storage unit (81). Due to the large number of music tracks in the music file database (52), when the favorite music is recommended, the favorite track storage unit (81) can be used to record and store the favorite tracks, which can be used in the future when encountering the same situation directly into the playlist of recommended music tracks.

Preferably, the music recommendation module (80) further has a recommended track scoring unit (82). The songs recommended by the music recommendation module (80) may not be liked by the driver. In order to avoid or reduce the continuous occurrence of such situations when recommending music in the future, the recommended song scoring unit (82) provides the user with the recommended Procedures for regrading music tracks. For example, if a recommended song is rated as 2 points by the driver, and the recommended music reference threshold is set to 4 points, it means that although the music and its style meet the initial recommendation standard, the music preference is indeed subjective . Therefore, this recommended track or music tracks with similar music characteristics will no longer be included in the recommended music track playlist, which can increase the efficiency of the music recommendation module (80) when searching for recommended tracks.

FIG. 3 is a flow chart of an embodiment of the music recommendation method based on image recognition and situational status of the present invention, please refer to FIG. 3 . The music recommendation method based on image recognition and situation status of the present invention is applied to the music recommendation system based on image recognition and situation status, and the method includes: an audio-visual data archiving step (S0), an audio-visual training step (S1), A preference scoring step (S2), an image recognition and classification step (S3), a music genre screening step (S4), a feature value comparison and music recommendation step (S5). The audio-visual data archiving step (S0) stores the plural training repertoires and the plural situational images in a training music database module and a situational image database module respectively, wherein the plural training repertoires are classified according to music style attributes A plurality of music styles, each type of music style stores a plurality of tracks, and the plurality of situational images include images of different types of situational states.

In this embodiment, the plurality of situational images are classified into one daytime-sunny-smooth situational state, one daytime-sunny day-Yongsai situational state, one daytime-rainy day-smooth situational state, and one daytime-rainy day-smooth situational state according to weather and road conditions. Situational status of Sai, one night-night-clear-smooth situational state, one night-night-clear-Yongsai situational state, one night-rainy day-smooth situational state and one night-rainy day-Yongsai situational state, a total of eight situational images.

The audio-visual training step (S1) provides an audio-visual training module for the user to operate, and the audio-visual training module extracts training tracks and situational images from the training music database module and the situational image database module respectively, And by providing the user to click on the scene image and the training track to play at the same time, so that the user can watch and listen to the training, wherein, when each scene image is played, each track in the plurality of training tracks is matched in sequence at the same time. The audio-visual training step (S1) provides the user with a choice of matching training tracks for each situational image. Of course, the user can also directly select a favorite style of music for training at the beginning. For example: if the user likes country music, classical music and For jazz music, users can select some training tracks from these three genres for audio-visual training, directly avoiding disliked music genres, which can improve the efficiency of subsequent music recommendations.

The preference scoring step (S2) is to provide a scoring record module for the user to perform on the preferences of the current situational image screen and the corresponding training track after the user completes each audio-visual training in the audio-visual training step. Scoring, and storing the user information and scoring records in a user database for backup, where the user can set the scoring standard threshold. The user gives a score for each piece of music that is matched with each scene image, such as 1 to 5 points. The higher the score, the more he likes it, which is used as a reference for subsequent music recommendations. Situational images and a training track of a country music style with which they are matched give a 2-point evaluation, which means that when the user does not like the first training track under the actual conditions of daytime-rainy day-Yongsai, therefore, the same as the training track If the first training track has similar musical characteristics, the probability of being recommended will be very low.

Preferably, the preference scoring step (S2) provides users with the ability to set the scoring standard threshold, for example, if the recommended music scoring standard threshold is set to 4 points, then the training tracks with a score of 4 points or more are included in the recommended music reference, scoring If the score is 1~3, it will only be included in the score record, and will not be included in the recommended music reference.

The image recognition and classification step (S3) uses an image recognition module to recognize and classify the climate and road condition characteristics of the user while driving, and classify the climate and road condition characteristics into one of the plurality of situation images. When the user is driving, the image recognition and classification step (S3) captures the current image picture through the image recognition module, for example, through the camera lens of the driving recorder on the car, and according to the brightness of the light in the picture , Rain or shine conditions, the distance to the vehicle in front and the number of vehicles in front, etc., to judge the current situation status. The image recognition and classification step (S3) uses a neural network model as a method for image recognition. In order to avoid an excessively high error rate of image recognition results during actual execution, the image recognition and classification step (S3) uses a ResNet50 framework for training. Improve recognition accuracy.

ResNet50 is a deep learning architecture. In order to avoid the problems of overfitting and training degradation, through the residual module of ResNet50, as the number of training increases, a deeper neural network can be trained, and the recognition accuracy can also be improved by 85%. %above. In this embodiment, by collecting 6,000 daytime and night situation image pictures as training data, 20,000 sunny and rainy situation image pictures as training data, and 9,000 Yongsai and smooth situation image pictures as training data, including : Daytime-Sunny-Smooth Situation Image, Day-Sunny-Yongsai Situational Image, Day-Rainy-Smooth Situational Image, Day-Rainy-Yongsai Situational Image, Night-Night Clear-Smooth Situational Image, Night -Sunny night-Yongsai situational image picture, night-rainy day-smooth situational image picture and night-rainy day-Yongsai situational image picture, a total of eight categories of pictures of the situational state of climate and road conditions are used as data for deep learning.

Please continue to refer to FIG. 3 . Preferably, in some embodiments of the present invention, the image recognition and classification step (S3) further includes an image data enhancement step to enhance the image data to increase the accuracy of situational state recognition, and the image data enhancement step uses two kinds of data Enhancement technology, the first is Contrast Limited Adaptive Histogram Equalization (CLAHE for short), when the foreground or background of the image is too bright or too dark, this method can make the brightness better in the image The upper distribution can avoid the situation that when the sky is sunny and cloudy, it is easy to be misjudged as a rainy day. This method is suitable for the judgment and identification of sunny days and rainy days. In this embodiment, the originally collected image data is enhanced, so that 6,000 original images plus 6,000 enhanced images, a total of more than 12,000 image data are used for deep learning training, which can improve the accuracy and maintain it at more than 85%. .

Continuing from the above, the second data enhancement technology is the image enlargement method. In order to avoid the insufficiency of the deep learning model in some situations or the lack of data, this method is used to increase the learning data to increase the recognition This method is suitable for the identification and judgment of smooth road conditions and congestion. In this embodiment, since this state is located at the bottom of the tree diagram of the deep learning model, it means that the more situational conditions need to be met, the more difficult it is to find the collected data. Therefore, the originally collected image data are divided into After data augmentation for the situational states of sunny and congested sunny days, smooth rainy days and congested rainy days, smooth nights and congested rainy nights, and smooth rainy nights and congested night rains, the above four situational states changed from 6,000 original images plus 6,000 enhanced images, there will be 12,000 images for each situation, and the accuracy will be maintained at more than 85%

The music style screening step (S4) finds out the user information and its rating records from the user database according to the situational image status judged in the image recognition and classification step (S3), and uses the set rating The standard threshold finds the corresponding track from the plurality of training tracks to generate a filtered music track data table. For example, the image recognition and classification step (S3) recognizes that the current situation is a daytime-rainy day-smooth situation, and then find out from the user database the trainings that the user used to match the daytime-rainy-smooth situation image. The score of the track, and select the training track that meets the threshold of the scoring standard and put it into the screened music track data table for subsequent comparison of recommended music collection. For example, the training track with a score of 4 or above is screened out and included.

The feature value comparison and music recommendation step (S5) sequentially analyzes the music features of the training tracks in the screened music track data table, and searches a music file database, and finds out the music track corresponding to the screened music track from the music file database. The music tracks corresponding to the training tracks in the data table with similar music characteristics are used as recommended music tracks, and the recommended music tracks are used as a playlist for the user to play and listen to. In this embodiment, the music features include melody, rhythm, and chord, etc. In the present invention, the music feature extraction method uses Mel Cepstral Coefficients (MFCC). Mel cepstrum coefficients include two important sub-steps, namely Mel frequency analysis and cepstrum analysis. The frequency, amplitude and phase of the audio signal generated through the music clip determine the characteristics of the audio signal. According to the music track data table According to the music features of the training tracks, the music tracks matching the music features are found from the music file database as the recommended music tracks, and they are organized into a playlist for the user to play and listen to. Of course, the number and selection method of the recommended music tracks can be set according to user settings, for example, 30 matching recommended music tracks are randomly selected each time as a playlist.

Preferably, in some embodiments of the present invention, the image recognition and classification step further includes a vehicle speed detection step to detect the current vehicle speed state while driving, so as to assist the accuracy of situational state recognition and classification. In order to increase the smoothness of the road conditions or the recognition accuracy of Yongsai, the vehicle speed detection step can be used to detect and record the speed status of the vehicle to improve the recognition accuracy, especially when the front screen is smooth or the recognition of Yongsai is difficult, For example, the line of sight is unclear or the congestion caused by road construction is not the normal congestion situation when there is a lot of traffic. For example, if the vehicle speed remains below a certain speed within a short period of time, for example, the vehicle speed remains below 40 km/h within ten minutes, or the vehicle speed changes from low speed to high speed and then from high speed to low speed for more than 5 times within ten minutes, And if the speed difference is 10-30 km/h, it means that the vehicle is in a stop-and-go situation, and all of the above are traffic jams.

FIG. 4 is a flow chart of another embodiment of the music recommendation method based on image recognition and situational status of the present invention, please refer to FIG. 4 . Preferably, in some embodiments of the present invention, the music recommendation method based on image recognition and situational status of the present invention further includes a favorite music recording step (S6), which provides the user with a record of music that he has heard before and thinks he likes it, for Subsequent listening can be done again in the same situational state. Due to the large number of music tracks in the music file database, when the favorite music is recommended, the favorite track record can be stored through the favorite music recording step (S6), which can be directly listed when encountering the same situation in the future Enter the recommended music track playlist to avoid being cleared or re-recommended to reduce the efficiency of the music recommendation process.

FIG. 5 is a flow chart of another embodiment of the music recommendation method based on image recognition and situational status of the present invention, please refer to FIG. 5 . Preferably, in some embodiments of the present invention, the music recommendation method based on image recognition and situational status of the present invention further includes a recommended song scoring step (S7), providing users with ratings for recommended songs, and optimizing future recommended songs the accuracy. The songs recommended in the feature value comparison and music recommendation step (S5) are automatically selected by the software system through the training of the deep learning model, and may not be liked by the driver in the subjective situation of the driver. Therefore, In order to avoid or reduce the continuous occurrence of this kind of situation when recommending music in the future, the recommended track scoring step (S7) provides a procedure for the user to re-score the recommended music track heard.

For example, if a certain recommended music track is rated as 2 points by the driver, and the recommended music reference threshold is set to 4 points, it means that although the music and its genre meet the initial recommendation standard, the music preference is indeed Subjective. Therefore, this recommended track or music tracks with similar music characteristics will no longer be included in the recommended music track playlist, which can increase the efficiency of the music recommendation module (80) when searching for recommended tracks.

To sum up, with the music recommendation system and method based on image recognition and situational status of the present invention, the situational status can be judged by identifying the driver's current weather conditions and the smoothness of the road ahead of the vehicle, and by As a result of prior audio-visual training, the two are integrated and then analyzed through music features to automatically search for music tracks suitable for the driver's current listening situation, providing the driver with a pleasant driving mood or alleviating negative emotions while driving.

The above-mentioned embodiments are only to illustrate the technical ideas and characteristics of the present invention, and its purpose is to enable those who are familiar with this art to understand the content of the present invention and implement it accordingly, and should not limit the patent scope of the present invention. , all equivalent changes or modifications made in accordance with the spirit of the present invention and the content of the description shall be covered within the patent scope of the present invention.

10: Training music database module 20: Situation image database module 30: Video training module 40: Scoring record module 50:Cloud database module 51: User database 52: Music file database 60: Image recognition module 61: Image deep learning unit 62: Image data enhancement unit 63:Vehicle speed detection unit 70:Integrated analysis module 80:Music recommendation module 81: favorite music storage unit 82:Recommended track scoring unit S0: Audio-visual data archiving steps S1: Audio-visual training steps S2: Like scoring step S3: Image recognition and classification steps S4: music genre screening steps S5: Feature value comparison and music recommendation steps S6: Steps for recording favorite music S7: Recommended track scoring steps

Fig. 1 is the architecture diagram of an embodiment of the music recommendation system based on image recognition and situation status of the present invention; FIG. 2 is a structural diagram of another embodiment of the music recommendation system based on image recognition and situational status of the present invention; 3 is a flow chart of an embodiment of the music recommendation method based on image recognition and situational status of the present invention; 4 is a flow chart of another embodiment of the music recommendation method based on image recognition and situational status of the present invention; FIG. 5 is a flow chart of another embodiment of the music recommendation method based on image recognition and situation status of the present invention.

10: Training music database module

20: Situation image database module

30: Video training module

40: Scoring record module

50:Cloud database module

60: Image recognition module

70:Integrated analysis module

80:Music recommendation module

Claims (10)

A music recommendation system based on image recognition and situational status, including: A training music database module (10), storing plural training tracks classified according to music types; A situational image database module (20), storing multiple situational images classified according to situational types; An audio-visual training module (30), connected to the training music database module (10) and the situational image database module (20), accessing the training tracks played in the training music database module (10) And the situational images in the situational image database module (20), wherein each situational image corresponds to at least one training track to be played simultaneously; A score recording module (40), connected to the audio-visual training module (30), provides the user with scoring and recording preferences based on the training repertoire and the situational image played by the music video training module (30); A cloud database module (50), connected to the scoring record module (40), the cloud database module (50) includes a user database (51) and a music file database (52), wherein, The user database (51) stores user data and the user rating data, and the music file database (52) stores music tracks of multiple types of music styles; An image recognition module (60), which performs situational state recognition and classification according to the climate and road condition characteristics of the user while driving, and the image recognition module (60) includes an image deep learning unit (61) to improve the recognition accuracy; An integrated analysis module (70), connected to the cloud database module (50) and the image recognition module (60), based on the classification result of the situation status currently recognized by the image recognition module (60) and the cloud data The user database (51) of the library module (50) is analyzed and compared, and a screened music track data table is generated; and A music recommendation module (80), which connects the cloud database module (50) and the integrated analysis module (70), and filters the training tracks of the music track data table with the cloud database module (50) The music file database (52) performs music feature search and comparison, and generates a playlist of recommended music tracks suitable for the user to listen to in a situational state while driving from the music file database (52) for playing and listening. The music recommendation system based on image recognition and situational status as described in Claim 1, wherein the image recognition module (60) further includes an image data enhancement unit (62), which has data enhancement technology including limited contrast adaptive histogram Equalization technology or image enlargement technology. The music recommendation system based on image recognition and situation status as described in claim 1, wherein the image recognition module (60) further includes a vehicle speed detection unit (63) to detect and record the vehicle speed status. The music recommendation system based on image recognition and situation status as described in claim 1, wherein the music recommendation module (80) further has a favorite music storage unit (81). The music recommendation system based on image recognition and situation status as described in Claim 1, wherein the music recommendation module (80) further has a recommended track scoring unit (82). A music recommendation method based on image recognition and situational status, comprising: An audio-visual data archiving step (S0), storing the plurality of training tracks and the plurality of situational images in a training music database module and a situational image database module respectively, wherein the plurality of training tracks are classified into a plurality of kinds of music genre; An audio-visual training step (S1), providing an audio-visual training module for the user to operate, and the audio-visual training module extracts training tracks and situational images from the training music database module and the situational image database module respectively , and provide the user to click on the situational image and the training track to play at the same time, so that the user can watch and listen to the training, wherein, when each situational image is played, at least one of the plurality of training tracks is matched at the same time; A preference scoring step (S2), when the user completes each audio-visual training in the audio-visual training step, a scoring record module is provided for the user to rate the current situational video screen and the corresponding training repertoire , and store the user information and its scoring records in a user database for backup, where the user can set the scoring standard threshold; An image recognition and classification step (S3), using an image recognition module to recognize and classify the climate and road condition characteristics of the user while driving, and classify the climate and road condition characteristics into one of the plurality of situation images; A music genre screening step (S4), according to the situational image status judged in the image recognition and classification step, find out the user information and its scoring records from the user database, and pass the set scoring standard threshold Find the corresponding track from the plurality of training tracks to generate a filtered music track data table; and A feature value comparison and music recommendation step (S5), sequentially analyzing the music features of the training tracks in the screened music track data table, and searching for a music file database, and finding the music that matches the screened music from the music file database Music tracks with similar music features corresponding to the training tracks in the track data table are used as recommended music tracks, and the recommended music tracks are used as playlists for users to play and listen to. The music recommendation method based on image recognition and situational state as described in claim 6, wherein the image recognition and classification step (S3) further includes an image data enhancement step, which strengthens the image data to increase the accuracy of situational state recognition. The music recommendation method based on image recognition and situational state as described in claim 6, wherein the image recognition and classification step (S3) further includes a vehicle speed detection step, which detects the current speed state of the vehicle while driving, and assists the situational state recognition and classification the accuracy. The music recommendation method based on image recognition and situational state as described in claim 6, which further includes a favorite music recording step (S6), providing the user to record the favorite music that the user has heard before, for subsequent use in the same situational state Can listen again. The music recommendation method based on image recognition and situational status as described in claim 6, further comprising a recommended track scoring step (S7), which provides users with ratings for recommended tracks to optimize the accuracy of future recommended tracks.

Images