CN115329059A - Electronic specification retrieval method and device and vehicle machine - Google Patents

Abstract

The application relates to the technical field of intelligent vehicles, and provides an electronic specification retrieval method, an electronic specification retrieval device and a vehicle machine. According to voice awakening words received by the sound pick-up, an electronic specification use mode is started, at least one camera related to the position of the sound pick-up is started to obtain an intelligent cabin image, gesture recognition and tracking are carried out on the intelligent cabin image to determine attribute information of a target gesture, a 3D model of the intelligent cabin is combined to quickly and accurately determine a target query item in the intelligent cabin, and then a display screen and a sound box installed in an awakening area are used for playing the related content of the target query item.

Description

Electronic specification retrieval method and device and vehicle machine

Technical Field

The application relates to the technical field of intelligent vehicles, and provides an electronic specification retrieval method, an electronic specification retrieval device and a vehicle machine.

Background

At present, the automobile market presents a warming-up situation, and from the automobile holding capacity, the automobile quantity still has great growth space, and the whole automobile market keeps a better growth situation.

With the development of the automobile industry, compared with the traditional automobile, the automobile has the advantages that the modeling, intelligent interaction, operation complexity, option setting and the like are greatly improved, and various new products which are intelligent, electric, personalized and highly configured are provided by new automobile building efforts to meet the requirements of users. However, the functions and keys of these new products are more and more, and the operation, system and interface are more and more complex, which results in the higher learning cost for the user to use the vehicle.

Therefore, a good vehicle specification, especially an electronic vehicle specification easy to search, is very important for a user to quickly grasp the using method of the vehicle, and the user can skillfully use high-tech configuration to really experience the convenience of various configurations of the vehicle.

Disclosure of Invention

The application provides an electronic specification retrieval method, an electronic specification retrieval device and a vehicle machine, which are used for improving the accuracy of vehicle specification retrieval and facilitating the learning of a user on a specification.

In one aspect, the present application provides a method for retrieving an electronic specification, applied to a vehicle, including:

acquiring voice awakening words received by a sound pick-up to start an electronic specification use mode;

opening at least one camera related to the position according to the position information of the sound pickup in the intelligent cabin, and acquiring an image of the intelligent cabin acquired by the opened camera;

gesture recognition and tracking are carried out on the intelligent cockpit image, and attribute information of a target gesture is determined;

determining a target query item in the intelligent cabin according to the constructed 3D model of the intelligent cabin and the attribute information; and retrieving the associated content of the target query item from the electronic specification and playing the associated content.

In another aspect, the present application provides an electronic specification search apparatus, including:

the sound pickup device comprises a plurality of sound pickups, a voice module and a control module, wherein each sound pickup is preset at different positions of an intelligent cabin and is configured to receive voice wake-up words in the intelligent cabin;

the intelligent cabin comprises a plurality of cameras, wherein a preset position of each camera comprises a sound pickup which is configured to collect images in the intelligent cabin;

a central processor configured to retrieve a voice wake-up word received by the microphone to enable an electronic specification use mode; opening at least one camera related to the position according to the position information of the sound pickup in the intelligent cabin, and acquiring an image of the intelligent cabin acquired by the opened camera; performing gesture recognition and tracking on the intelligent cockpit image, and determining attribute information of a target gesture; determining a target query item in the intelligent cabin according to the constructed 3D model of the intelligent cabin and the attribute information; and retrieving the associated content of the target query item from the electronic specification and playing the associated content.

On the other hand, the application provides a car machine, which comprises a central processor, a memory, a plurality of sound collectors and a plurality of cameras, wherein the sound collectors and the cameras are arranged at different preset positions in an intelligent cabin of the car machine, the intelligent cabin is divided into a plurality of areas, and each area comprises at least one sound collector and camera, a sound box and a display screen;

the sound pickup is configured to receive voice wake-up words in the intelligent cabin;

the camera is configured to acquire images in the intelligent cabin;

the sound box and the display screen are configured to play the content retrieved from the electronic specification;

the memory includes a data storage unit configured to store electronic instructions and a program storage unit configured to store a computer program;

the central processor executes the steps of the electronic specification retrieval method according to the computer program.

In another aspect, the present application provides a computer-readable storage medium storing computer-executable instructions for causing a computer device to execute an electronic specification retrieval method provided by an embodiment of the present application.

In the electronic specification retrieval method, the electronic specification retrieval device and the vehicle machine, a voice awakening electronic specification use mode is supported by means of a sound pickup in an intelligent cabin, at least one camera related to the position can be started to obtain an intelligent cabin image according to the position information of the sound pickup receiving a voice awakening word, the attribute information of a target gesture is determined by performing gesture recognition and tracking on the intelligent cabin image, and a target query item in the intelligent cabin is rapidly and accurately determined by combining a 3D (three-dimensional) model of the intelligent cabin.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the description below are some embodiments of the present application, and those skilled in the art can obtain other drawings based on the drawings without inventive labor.

FIG. 1 is a schematic diagram of an electronic specification retrieved based on keywords according to an embodiment of the present application;

FIG. 2 is a schematic diagram of retrieving an electronic specification based on key activation according to an embodiment of the present application;

fig. 3 is a schematic view of a sound pickup and a camera arranged in an intelligent cabin according to an embodiment of the present application;

fig. 4A is a schematic configuration diagram of a sound box in a main driving area according to an embodiment of the present application;

fig. 4B is a schematic configuration diagram of a sound box in a copilot area according to an embodiment of the present application;

fig. 4C is a schematic configuration diagram of a sound box in a back row area according to an embodiment of the present application;

FIG. 5A is a schematic configuration diagram of a display screen in a main driving area according to an embodiment of the present application;

fig. 5B is a schematic configuration diagram of a display screen in a passenger compartment area according to an embodiment of the present application;

fig. 5C is a schematic configuration diagram of a display screen in the back row area according to an embodiment of the present application;

fig. 6 is a flowchart of an electronic specification retrieval method provided in an embodiment of the present application;

fig. 7 is a flowchart of a method for turning on a camera in an intelligent cabin according to an embodiment of the present disclosure;

FIG. 8 is a flowchart of a method for determining a target query term according to an embodiment of the present application;

FIG. 9 is an interaction flow diagram of a method for retrieving an electronic specification provided by an embodiment of the present application;

fig. 10 is a schematic view of an application scenario of a method for retrieving an electronic specification according to an embodiment of the present application;

fig. 11 is a block diagram of an electronic specification retrieval apparatus according to an embodiment of the present application;

fig. 12 is a structural diagram of a vehicle machine provided in an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments, but not all embodiments, of the technical solutions of the present application. All other embodiments obtained by a person skilled in the art without any inventive step based on the embodiments described in the present application are within the scope of the protection of the present application.

With the development of the automobile industry, vehicle specifications are thicker and thicker due to more and more functions and complex operation of automobiles, for users, the traditional paper specifications are difficult to help the users to learn systematically, the learning time and the learning difficulty are increased, particularly for new users, the difficulty in learning new knowledge is higher, each brand of vehicle has own characteristics, and a learning process is also needed after the users change the automobile.

Currently, there are three main ways of learning vehicle specifications:

and (I) learning new functions and operation modes of the automobile by reading paper specifications. However, moving a few hundred sheets of instruction books results in very poor browsing efficiency and experience, moreover, the instruction books may involve multiple languages, and even if the instruction books are turned over, the requirement on the query skills of the user is high, and the same instruction book often contains multiple vehicle models of the same type of vehicle, which may result in the interpretation inquired by the user not being functions which the vehicle model does not have. Moreover, the operation steps described in the paper specification are not easy to understand, and the operation steps are required to be compared with the specification to understand.

(II) the key words search electronic specification learns new functions and operation modes of the automobile. The input of the keywords comprises two modes of manual input and voice input on a central control screen in the vehicle, corresponding query items are retrieved from an electronic specification through the keywords input by a user, and then broadcasting is carried out through voice or displaying is carried out through the central control screen. However, this method requires the user to organize the keywords of the query function or the key, generally, most of the terms used in the specification of the vehicle are professional terms, proper terms and translated terms, and each brand of vehicle has its own unique term, so when the user knows the function or the key, it is very difficult to organize the keywords describing the unknown professional device key or function, and the keywords expressing the terms or the specific function may not be understood, so even if the user uses voice recognition to interact, the user may need to perform many times of keyword retries, combinations and the like to accurately understand the semantics, resulting in low search efficiency.

Because the method needs the user to extract the key words from the contents of the buttons, the function keys, the prompt marks and the like to be queried through the language description, the user needs to know the meanings of the buttons, the function keys and the prompt marks to be queried, and for the user, the user needs to query the electronic specification for learning just by not knowing the contents of the buttons, the function keys, the prompt marks and the like to be queried, so that the user is difficult to organize the key words for describing strange functions, the buttons or the prompt marks, and therefore, when the user interacts through the voice recognition key words, certain errors can be brought to the difference of the language description, the dialect description and the unofficial description of the key words, certain difficulties and deviations can be brought to the accuracy of the voice recognition and analysis, and the contents of the items to be queried in the specification can not be accurately retrieved.

For example, as shown in fig. 1, three warning light signs in the cockpit of the vehicle are provided, an electronic specification of the vehicle is retrieved based on a keyword, and when the electronic specification of the vehicle is retrieved interactively with a voice system for the warning light sign shown in fig. 1 (a), a keyword extracted by a user organization is described as "a little carries a sword", and actually, the warning light sign means "no safety belt is tied"; for the warning light sign shown in fig. 1 (b), when the user interacts with the voice system to retrieve the specification, the keyword extracted by the user organization is described as "alatin god light", and actually the warning light sign means "insufficient engine oil"; for the warning light sign shown in fig. 1 (c), when the user interactively searches the instruction book with the voice system, the keyword extracted by the user organization is described as "two circles are drawn? Or the flight path of the flying insect ", and in practice the warning light sign means" diesel engine abnormal ".

As can be seen from fig. 1, the keywords extracted by the user organization are far from the actual meanings of the warning light signs in the description, and the voice system cannot accurately retrieve the related contents of the warning light signs in the electronic description.

And (III) triggering and retrieving the electronic specification to learn the new function and the operation mode of the automobile, retrieving the contents of the pressed key from the electronic specification by pressing the key, for example, as shown in FIG. 2, retrieving the explanation of the key from the specification by pressing the key 21 when the user needs to learn the function of the key 21 on the steering wheel.

Because a certain key, button, knob or shift lever and the like must be triggered in the mode, the triggering has certain danger under the condition that the function and the operation mode are not clear, and the operation is improper, the button to be inquired can be triggered only after the mode enters a specification mode or a parking mode, the specification retrieval function cannot be used in the driving process, the use scene is limited, the operation is inconvenient, and the learning limitation is increased for a user. Moreover, the electronic specification can not be inquired about the inquiry items without keys, buttons, knobs or poking rods (for example, inquiry items which can not be triggered such as various marks on a screen and warning lamps), so that the user can not know the vehicle use instruction.

Therefore, no matter the electronic specification of the vehicle is retrieved based on the keywords or the electronic specification of the vehicle is retrieved based on the trigger keys, for the user, under the condition that the keywords are not known and the specific functions of the keys are not clear, the retrieval efficiency and accuracy of the specification are low, so that the cost for the user to learn the vehicle knowledge is high, and the experience is poor.

Meanwhile, although the relevant vehicle supports the use of the electronic specification, the electronic specification is generally only stored locally in the vehicle and is only limited to the retrieval of fixed contents in the specification, and the specification cannot be updated in a networking manner; moreover, when the electronic specification is retrieved by supporting voice recognition, the voice recognition needs to be carried out only by networking because cloud computing is required, so that the voice recognition can be carried out only by one AI loudspeaker box on the vehicle without networking, the use scene of the electronic specification is limited, and the electronic specification can not be used under the condition of no network.

In view of this, embodiments of the present application provide an electronic specification retrieval method, an apparatus, and a vehicle machine, which enable a user to retrieve a what you see is what you get mode in an intelligent cabin, thereby improving the intellectualization and convenience of using an electronic specification on a vehicle, and when the user needs to know any query item such as a certain button, a pull rod, a button, a mark, etc. in the intelligent cabin, the user can start an electronic specification using mode by using a sound pickup in the intelligent cabin, and start at least one camera related to a position according to the position information of the sound pickup to obtain an intelligent cabin image, and search for related content of a target query item pointed by the user from the electronic specification by performing gesture recognition and tracking on the intelligent cabin image, without organizing and refining keywords of the target query item by the user, and without triggering the target query item by the user, thereby effectively improving the accuracy and efficiency of retrieval, and simultaneously facilitating the user to learn the function and operation mode of the query item anytime and anywhere, and improving the vehicle experience of the user.

In order to support the retrieval method of the electronic specification provided by the embodiment of the application, in the intelligent cabin of the vehicle, the sound pick-up is configured as a peripheral access device to support a voice awakening function, so that the intelligent cabin can receive voice awakening words used by a user, the electronic specification use mode of the vehicle can be awakened quickly and conveniently, and the situation that passengers can be seated on front-row seats and rear-row seats of the intelligent cabin and the vehicle can be used for learning at different positions is considered, so that the sound pick-up devices are respectively installed at different positions of the intelligent cabin, and the intelligent cabin is divided into a plurality of areas. Meanwhile, according to the embodiment of the application, a camera is configured for each sound pick-up so as to collect and awaken images in the intelligent cabin, so that after a user starts a voice awakening function, the electronic specification can be retrieved according to query items indicated by gestures of the user, and a retrieval mode of a WYSIWYG (what you see is what you get) mode is realized.

As an example, the embodiment of the present application provides a smart cabin with a sound pickup and a cameraSchematic view, as shown in FIG. 3, from (1) to

At different locations, a sound pick-up and a camera are provided, the 12 sound pick-up and camera being identified by the numbers 1-12, respectively. 12 to adapter and camera, divide into four regions with the intelligent passenger cabin, mark as region one, region two, region three and region four respectively.

It should be noted that, in some embodiments, the number and the installation position of the sound pickup and the camera may also be different, and it is sufficient that the camera can acquire images of various angles in the area.

Generally, keys, knobs and indication marks in different areas in the intelligent cockpit are different, so that a user can conveniently learn vehicle functions and operation modes at different positions, electronic specifications can be independently searched in each area, and meanwhile, search results can be independently displayed. Therefore, in the embodiment of the present application, speakers for performing audio playback on the search result are installed in each region. The installation position of each sound box can be set according to the actual hardware configuration of the corresponding area in the intelligent cabin.

For example, as shown in fig. 4A to 4C, schematic diagrams of configuring each speaker in the smart cabin according to the embodiments of the present application are provided, where as shown in fig. 4A, the speaker 22 is configured at the lower right of the steering wheel of the main driving in the first area; as shown in fig. 4B, in the second region, the sound box 23 is disposed above the front passenger compartment; as shown in fig. 4C, in the third and fourth areas, the sound boxes 24 are disposed behind the seat backs of the main driver and the passenger car, respectively.

In some implementations, in order to facilitate a user to better learn the functions and operation modes of buttons, function keys, prompt identifiers and the like in the intelligent cabin, a display screen is further configured for each area in the intelligent cabin for more intuitively demonstrating a retrieval result.

For example, as shown in fig. 5A to 5C, schematic diagrams of configuring each display screen in the intelligent cabin provided in the embodiment of the present application are provided, where as shown in fig. 5A, the display screen 25 is configured at the lower left of the steering wheel of the main driver in the first area; as shown in fig. 5B, the display screen 26 is disposed above the passenger compartment in the second region; as shown in fig. 5C, in the third and fourth areas, the display screen 27 is disposed behind the seat backs of the main and passenger cars, respectively.

Based on the configuration in the intelligent cabin, the following describes in detail the flow of the electronic specification retrieval method provided in the embodiment of the present application, and referring to fig. 6, the flow is executed by a central processor in the intelligent cabin, and the central processor serves as a control center of the intelligent cabin, and can retrieve the electronic specification of the host vehicle offline or online, and mainly includes the following steps:

s601: and acquiring voice wake-up words received by a sound pick-up to enable the electronic specification using mode.

In the embodiment of the application, after the vehicle is started, all sound pickup devices in the intelligent cabin enter the working mode to monitor the voice of the user. When the user does not know the functions and operation modes of a certain button, a function key, a prompt mark and the like in the intelligent cabin, a preset voice awakening word is spoken, the sound pickup receives the voice awakening word and then sends the voice awakening word to the central processor, and the central processor enables the electronic instruction book using mode to be used for the user to learn.

In S601, the voice wakeup word enabling the use mode of the electronic specification may be set by itself, which is not a limiting requirement in the embodiment of the present application. For example, "universal WeChat" is set as a voice wake-up word that enables the electronic specification use mode.

S602: and opening at least one camera related to the position according to the position information of the sound pickup in the intelligent cockpit, and acquiring the image of the intelligent cockpit acquired by the opened camera.

According to the embodiment, the plurality of sound pickups are installed in the intelligent cabin and located at different positions, so that voice in the intelligent cabin can be accurately acquired. Meanwhile, the intelligent cabin is divided into a plurality of areas by the sound pickup devices, the positions of the awakening users can be distinguished, and the use mode of the user in which the electronic specification is awakened is determined in which area of the front row, the rear row, the left side and the right side, so that the area where the user is awakened is accurately positioned.

In the embodiment of the application, a plurality of cameras are installed at different positions of the intelligent cabin of the vehicle, and images of all areas in the intelligent cabin can be collected from a plurality of angles, so that gestures and tracks of a user can be accurately identified and tracked, and a target query item which is pointed by the user and needs to be learned can be accurately positioned.

In order to save vehicle energy consumption, a plurality of cameras in the intelligent cabin enter a sleep mode after the vehicle is started, and are switched to a working mode after the sound pick-up related to the position of the cameras is awakened.

Considering that the sound pickup at different positions receives different signal strengths of the voice awakening words, the closer the user is to the sound pickup, the stronger the signal strength is. Therefore, in S602, the central processor in the smart cabin may perform accurate partitioning on each sound pickup according to the signal strength of the sound pickup receiving the voice wake-up word, and perform accurate partitioning on the camera related to the opened position of each sound pickup. Referring to fig. 7, a specific opening process of the camera mainly includes the following steps:

s6021: and determining the position information of at least one sound pickup with the signal strength larger than the strength threshold value according to the signal strength of the voice awakening words received by each sound pickup.

S6022: and accurately partitioning the sound pickup according to the position information of at least one sound pickup, determining the area where the user used by the learning vehicle is located, and taking the area as the awakening area of the intelligent cabin.

S6023: and opening a main camera surrounding the awakening area, and opening an adjacent area surrounding the awakening area and an auxiliary camera adjacent to the main camera.

For example, taking the configuration of each sound pickup and each camera in the smart cabin shown in fig. 3 as an example, after a user in the area three wakes up the usage mode of the electronic specification, the area three is determined as a wake-up area, and the area four is used as an adjacent area, at this time, the cameras marked as 6, 8, 9, and 11 in the surrounding area three are turned on, and meanwhile, the auxiliary cameras marked as 7 and 12 in the surrounding area four are turned on.

In S602, when the cameras (including the main camera and the auxiliary camera) in the intelligent cabin are turned on (i.e., switched from the sleep mode to the working mode), the images of the intelligent cabin are collected and transmitted to the central processor, so that the central processor identifies and tracks the gesture of the user in the wake-up area to determine the target query item pointed by the user.

It should be noted that, the gesture of the user during the search using the electronic specification may be defined by itself, and the embodiment of the present application is not limited to this. For example, a ten-finger gesture to spread the remaining four limbs to make a fist may be used for retrieval.

S603: and performing gesture recognition and tracking on the intelligent cockpit image, and determining attribute information of the target gesture.

In S603, the central processor performs gesture recognition and tracking after acquiring an image of the intelligent cockpit acquired by the turned-on camera, and determines attribute information of a target gesture of a target user in the wake-up area, where the attribute information includes an object coordinate pointed by the target gesture and an object identifier pointed by the target gesture.

The gesture recognition and gesture tracking technology is mature, and this part is not regarded as the focus of the present application, and is not described herein again.

S604: and determining a target query item in the intelligent cockpit according to the constructed 3D model and the attribute information of the intelligent cockpit.

In the embodiment of the application, a real 3D model in the intelligent cabin of the vehicle is constructed in advance and comprises information such as position coordinates and identifications of all query items in the intelligent cabin. Therefore, in S604, after determining the attribute information of the target gesture, the target query item pointed by the user gesture in the wake area may be determined in combination with the 3D model. The specific process is shown in fig. 8, which mainly comprises the following steps:

s6041: and comparing the object coordinate pointed by the target gesture with the position coordinate of each query item in the 3D model of the intelligent cabin, and comparing the object identifier pointed by the target gesture with the identifier of each query item in the 3D model of the intelligent cabin.

S6042: and determining a target query item in the intelligent cabin according to the comparison result of the coordinates and the identification.

The target query item is determined by combining the object coordinate pointed by the target gesture and the object identification, so that the accuracy of the target query item is improved.

By means of the sound pickup and the camera in the intelligent cockpit, when a user wants to know any object such as a certain button, a pull rod, a key and an indication mark in the intelligent cockpit, the user only needs to use a voice wake-up word to start the use mode of the electronic specification, then directly points to any object such as the certain button, the pull rod, the key and the indication mark with a hand, and identifies and tracks the gesture of the user through the image of the intelligent cockpit acquired by the camera, so that a target query item which the user wants to know is acquired, the learning mode which is obtained as seen is really achieved, the user does not need to organize and refine key words of the button, the pull rod, the key and the indication mark which need to be known, the difficulty of retrieving the electronic specification is reduced, the user does not need to trigger the button, the pull rod, the key and the indication mark which need to be known, and the defect that the user cannot learn without triggering of the button (such as some marks, decorations and the like) is solved, and meanwhile, the learning in a parking state and a driving state is supported, the use restriction of the electronic specification is reduced, and the vehicle can be conveniently learned anytime and anywhere.

S605: and retrieving the associated content of the target query item from the electronic specification and playing the associated content.

In S605, after the target query item that the user needs to know is determined, the search of the electronic specification of the vehicle is started. Specifically, the method includes the steps of firstly, retrieving the associated content of a target query item from a locally stored electronic specification, retrieving the associated content of the target query item from the electronic specification of the cloud platform when the associated content of the target query item is not retrieved from the locally stored electronic specification, uploading the target query item to the cloud platform when the associated content of the target query item is not retrieved from the electronic specification of the cloud platform, and reminding a vehicle manufacturer to make and submit the associated content of the target query item through the cloud platform so as to update the electronic specification.

In the embodiment of the application, the retrieval is firstly carried out from the locally stored electronic specification, the retrieval speed is improved, the use of the learning vehicle under the condition of no networking is supported, when the associated content of the target query item is not retrieved from the locally stored electronic specification, the central processor in the intelligent cabin is networked with the cloud platform of the vehicle factory, the electronic specification is downloaded from the cloud platform and retrieved, the reliability of obtaining the associated content of the target query item is improved, and when the cloud platform does not have the associated content of the target query item at present, the vehicle manufacturer can update the content of the electronic specification at the later stage according to the requirement of a user, and the reliability of specification retrieval is further improved.

Generally, the associated content of a query item, such as a certain button, a pull rod, a key, and an indication mark, which a user needs to know, may be a text description, and therefore, in order to support the associated content of a target query item in an electronic specification, in the embodiment of the present application, a sound box is respectively installed in each area of an intelligent cabin, as shown in fig. 4A to 4C, so that, in S605, when the associated content of the target query item is a text description, after the associated content of the target query item is retrieved, the text description of the target query item is played by waking up the sound box in the area in a voice manner.

In consideration of the fact that the user needs to know the associated content of a certain query item such as a button, a pull rod, a key, an indication mark and the like, the user can also be introduced by an image, a video and the like, compared with the text description, the image and the video are more convenient for the user to understand the function and the operation mode of the target query item, especially the electronic specification of the vehicle has some terms, and if the user does not need to display the video, a new user may not understand the specific meaning and the operation mode. Therefore, in the embodiment of the present application, a display screen is installed in each area of the smart car, as shown in fig. 5A to 5C, so that in S605, after the associated content of the target query item is retrieved, the video of the target query item is played on the screen by using the display screen in the wake-up area.

It should be noted that the sound box and the display screen in the same area can be used simultaneously.

Because this application embodiment all has installed audio amplifier and display screen for every region, so, no matter be the user of front row owner driver's seat, also the user of copilot seat, moreover be the user on the back row seat, all can use the electronic specification of vehicle.

For example, in the driving process, when a back row user wants to know the functions and operation modes of a certain mark, a key, an air-conditioning air outlet and the like in the back row of the intelligent cabin, language does not need to be organized for description, the electronic specification use mode can be directly started through a voice wake-up word, a target query item is queried by fingers, and the sound box and the display screen in a wake-up area can display the associated content of the target query item.

According to the electronic specification retrieval method, a voice awakening electronic specification use mode is supported by means of a sound pickup in an intelligent cabin, at least one camera related to the position can be started to obtain an intelligent cabin image according to the position information of the sound pickup receiving a voice awakening word, gesture recognition and tracking are conducted on the intelligent cabin image, attribute information of a target gesture is determined, a 3D model of the intelligent cabin is combined, a target query item in the intelligent cabin is rapidly and accurately determined, and the whole determination process of the target query item is automatically conducted.

In some embodiments, when a video case (e.g., video operation content, instructional operation, etc.) presented by a real person exists in the associated content of the target query item, the video case can also be played by using a display screen in the wake-up area so as to better help a user to know the function and the specific operation mode of the target query item.

In some embodiments, in the case of networking of the intelligent cockpit, when there is statistical data (such as playing times, praise times, video duration, etc.) in the associated content of the retrieved target query item, the suggested video of the target query item may be recommended and played according to the statistical data, so that the user further knows how to set and operate the target query item by the users of most vehicles, and the user can learn quickly.

The following describes a flowchart of the electronic specification retrieval method provided by the embodiment of the present application from the perspective of interaction between the central processor, the sound pickup, the camera, the sound box, and the display screen, and with reference to fig. 9, the method mainly includes the following steps:

s900: after the vehicle starts, the sound pick-up enters a working mode, and the camera enters a sleep mode.

S901: the sound pick-up receives the voice awakening words and transmits the voice awakening words to the central processor.

S902: the central processor determines the position information of at least one sound pickup with the signal strength larger than the strength threshold value according to the signal strength of the voice awakening words received by each sound pickup.

S903: and the central processor determines the awakening area of the intelligent cabin according to the position information of the at least one sound pickup.

S904: the central processor starts a main camera surrounding the awakening area and starts an auxiliary camera surrounding the adjacent area of the awakening area and adjacent to the main camera.

S905: the opened camera collects images of the intelligent cabin and transmits the images to the central processor.

S906: and the central processor performs gesture recognition and tracking on the intelligent cockpit image, and determines object coordinates and object identification pointed by the target gesture.

S907: the central processor compares the object coordinate pointed by the target gesture with the position coordinates of all query items in the 3D model of the intelligent cabin, and compares the object identification pointed by the target gesture with the identification of all query items in the 3D model of the intelligent cabin.

S908: and the central processor determines a target query item in the intelligent cabin according to the comparison result of the coordinates and the identification.

S909: the central processor retrieves the associated content of the target query term from the locally stored electronic specification.

S910: the central processor determines whether the content associated with the target query term is retrieved from the local electronic specification, and if not, proceeds to step S911, and if so, proceeds to step S915.

S911: the central processor accesses the cloud platform of the manufacturer of the vehicle through the network and retrieves the associated content of the target query item from the electronic specification of the cloud platform.

S912: the central processor determines whether the associated content of the target query item is acquired from the cloud platform, if so, performs S913, otherwise, performs S914.

S913: and the central processor downloads the associated content of the target query item from the electronic specification of the cloud platform, updates the local electronic specification and then executes S915.

S914: and the central processor reports the associated content of the target query item made by the cloud platform to update the electronic specification.

S915: the central processor determines whether the associated content is a text description or a video, and if the associated content is a text description, S916 is performed, and if the associated content is a video, S917 is performed.

S916: and the sound box plays the text description of the target query item in a voice mode.

S917: the display screen plays a video of the target query term.

S918: the central processor determines whether a video case exists, if so, performs S919, and if not, performs S920.

S919: the display screen plays the video case.

S920: and the central processor responds to the video recommendation operation and recommends the suggested video of the target query item according to the statistical data.

S921: the display screen plays the suggested video.

The following describes an electronic specification retrieval method provided by the embodiment of the application in a specific application scenario.

Assuming that a user sits on a seat on the left side of the rear row of the vehicle, an icon as shown in (a) in fig. 1 is highlighted behind an armrest box of the vehicle between a main driving position and a subsidiary driving position during the driving of the vehicle, the user does not know what warning prompt the icon is, and the icon cannot be touched, so that a key-type triggered retrieval method fails, the user cannot know the meaning of the icon and also does not know how to operate and solve the problem of icon warning, and in the case that the user does not know the meaning of the icon, the keywords extracted by organizing are not accurate enough, and the icon may not be identified by adopting a keyword retrieval mode. The efficiency and accuracy of retrieval are reduced.

As shown in fig. 10, when the user yells the voice wake-up word "omnipotent xian" when the method provided by the embodiment of the present application is used, the sound pickup receives the voice wake-up word and transmits the voice wake-up word to the central processor to respond, recognize that the user is currently located in the third area, turn on the cameras identified as 6, 8, 9, and 11 in the third area, and turn on the cameras identified as 7 and 12 in the fourth area for assistance. When the user receives the response of the intelligent cockpit to the 'WeChat on' response, the user points to the bright icon and inquires 'what' the user is, and meanwhile, the turned-on camera collects the image of the intelligent cockpit containing the gesture of the user and transmits the image to the central processor. The central processor identifies and tracks the gesture of the user, determines the position coordinates and the icon identification of the icon executed by the gesture of the user, and determines the icon pointed by the user through analysis. After determining the icon that the user wants to know, the central processor starts to search the electronic specification, and the searched video introducing the icon is played on the display screen of the third area. Through the played video, the user can learn that the icon is a safety belt warning sign, and after the vehicle is started, the user can be reminded to wear a safety belt to protect the safety of the user if the safety belt is not worn.

After the video introducing the icon is played, the video case of how to fasten the safety belt in the real person demonstration area III can be played so as to teach the user how to operate to eliminate the warning prompt of the icon. After the video case is played, the user fastens the safety belt according to the video case, the icon disappears, and the using mode of the electronic specification is finished. The embodiment of the application provides a more intelligent and more convenient retrieval method and a more convenient presentation method of the electronic specification, so that the description and the operation demonstration of the target query item can be simply, directly and conveniently known, the retrieval speed is high, and the accuracy is high.

It should be noted that the embodiment of the present application can be applied to an engine box and a trunk of a vehicle when the embodiment of the present application can be applied to an intelligent cabin of the vehicle, and a sound pickup, a camera, a sound box, a display screen, and the like are installed in the engine box and the trunk, so that a user can conveniently learn and use a certain button, a pull rod, a key, an indication mark, and the like in the engine box and the trunk.

Based on the same technical concept, the embodiments of the present application provide an electronic specification retrieval apparatus, which can implement the steps of the electronic specification retrieval method in the embodiments described above, and achieve the same technical effects, and are not described herein again.

Referring to fig. 11, the apparatus includes a plurality of microphones 1101, a plurality of cameras 1102, and a central processor 1103, wherein each microphone 1101 is preset at a different location of the intelligent cabin and configured to receive a voice wake-up word inside the intelligent cabin; a preset position of each camera 1102 comprises a sound pickup 1101 configured to collect images in the intelligent cabin; a central processor 1103 configured to retrieve a voice wake-up word received by the microphone 1101 to enable an electronic specification use mode; according to the position information of the sound pickup in the intelligent cockpit, at least one camera related to the position is started, and an intelligent cockpit image acquired by the started camera 1102 is acquired; gesture recognition and tracking are carried out on the intelligent cockpit image, and attribute information of a target gesture is determined; determining a target query item in the intelligent cockpit according to the constructed 3D model of the intelligent cockpit and the attribute information; and retrieving the associated content of the target query item from the electronic specification and playing the associated content.

Optionally, a sound pickup 1101 and a camera 1102 at multiple preset positions are respectively installed at multiple preset positions of the intelligent cabin, and the sound pickup 1101 and the camera 1102 at multiple preset positions divide the intelligent cabin into multiple areas;

the central processor 1103 is specifically configured to:

according to the signal intensity of the voice awakening words received by each sound pickup 1101 in the intelligent cabin, determining the position information of at least one sound pickup of which the signal intensity is greater than an intensity threshold value;

according to the position information of the at least one sound pickup 1101, determining the awakening area of the intelligent cabin;

and opening a main camera surrounding the awakening area, and opening an auxiliary camera surrounding the adjacent area of the awakening area and adjacent to the main camera.

Optionally, the attribute information includes coordinates of an object pointed by the target gesture and an identifier of the object pointed by the target gesture;

the central processor 1103 is specifically configured to:

comparing the object coordinate pointed by the target gesture with the position coordinate of each query item in the 3D model of the intelligent cabin, and comparing the object identifier pointed by the target gesture with the identifier of each query item in the 3D model of the intelligent cabin;

and determining a target query item in the intelligent cabin according to the comparison result of the coordinates and the identification.

Optionally, the central processor 1103 is specifically configured to:

retrieving content associated with the target query term from a locally stored electronic specification;

when the associated content of the target query item is not retrieved from the locally stored electronic specification, retrieving the associated content of the target query item from the electronic specification of the cloud platform;

when the associated content of the target query item is not retrieved from the electronic specification of the cloud platform, uploading the target query item to the cloud platform to update the electronic specification.

Optionally, a sound box 1104 is installed in each area of the intelligent cabin;

when the associated content is a textual description, the central processor 1103 is specifically configured to:

and playing the text description of the target query item in a voice mode by using a sound box 1104 in the awakening area.

Optionally, a display screen 1105 is installed in each area of the intelligent cockpit;

when the associated content is a video, the central processor 1103 is specifically configured to:

and using a display screen 1105 in the awakening area to screen and play the video of the target query item.

Optionally, when the associated content of the target query term exists in a video case of a live presentation, the central processor 1103 is further configured to:

the video case is played with the display screen 1105 in the wake-up area.

Optionally, when there is statistical data in the retrieved associated content of the target query term, the central processor 1003 is further configured to:

and recommending the suggested video of the target query item according to the statistical data and playing the recommended video.

The device can be applied to a vehicle, so as to obtain a vehicle machine provided by the embodiment of the present application, referring to fig. 12, the vehicle machine includes a central processor 1201, a memory 1202, multiple sound collectors 1203, multiple cameras 1204, multiple sound boxes 1205 and multiple display screens 1206, where the multiple sound collectors 1203 and the cameras 1204 are installed at different preset positions in an intelligent cabin of the vehicle machine, so as to divide the intelligent cabin into multiple areas, and each area includes at least one sound collector 1203 and camera 1204, one sound box 1205 and one display screen 1206; the memory 1202, the sound pickup 1203, the camera 1204, the sound box 1205 and the display screen 1206 are connected with the central processor through a bus 1207;

the microphone 1203 is configured to receive voice wake-up words in the intelligent cabin;

the camera 1204 is configured to acquire images in the intelligent cabin;

the sound box 1205 and the display 1206 configured to play content retrieved from an electronic specification;

the memory 1202 includes a data storage unit configured to store electronic instructions and a program storage unit configured to store a computer program;

the central processor 1201 executes the steps of the electronic specification retrieval method provided by the embodiment of the present application according to the computer program.

It should be noted that fig. 12 is only an example, and shows hardware necessary for implementing the steps of the electronic specification retrieval method provided by the embodiment of the present application. Not shown, the vehicle machine further includes conventional vehicle components such as wheels, a hood, a seat, an engine, a steering wheel, and the like.

The Processor referred to in fig. 12 in this embodiment may be a Central Processing Unit (CPU), a general purpose Processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application-specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a transistor logic device, a hardware component, or any combination thereof.

The embodiment of the present application further provides a computer-readable storage medium for storing instructions, which when executed, may perform an electronic instruction retrieving method in the foregoing embodiment.

The embodiment of the present application further provides a computer program product for storing a computer program, where the computer program is used to execute an electronic specification retrieval method in the foregoing embodiment.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims (10)

1. An electronic specification retrieval method, applied to a vehicle, includes:

acquiring voice awakening words received by a sound pick-up to start an electronic specification use mode;

opening at least one camera related to the position according to the position information of the sound pickup in the intelligent cabin, and acquiring an image of the intelligent cabin acquired by the opened camera;

gesture recognition and tracking are carried out on the intelligent cockpit image, and attribute information of a target gesture is determined;

determining a target query item in the intelligent cabin according to the constructed 3D model of the intelligent cabin and the attribute information;

and retrieving the associated content of the target query item from the electronic specification and playing the associated content.

2. The method of claim 1, wherein a sound pickup and a camera are respectively installed at a plurality of preset positions of the intelligent capsule, and the sound pickup and the camera at the plurality of preset positions divide the intelligent capsule into a plurality of areas;

the method for starting at least one camera related to the position according to the position information of the sound pickup in the intelligent cabin comprises the following steps:

according to the signal intensity of the voice awakening words received by each sound pickup in the intelligent cabin, determining the position information of at least one sound pickup of which the signal intensity is greater than an intensity threshold value;

determining a wake-up area of the intelligent cabin according to the position information of the at least one sound pickup;

and opening a main camera surrounding the awakening area, and opening an auxiliary camera surrounding the adjacent area of the awakening area and adjacent to the main camera.

3. The method of claim 1, wherein the attribute information includes coordinates of an object pointed to by the target gesture and an identification of the object pointed to by the target gesture;

the method for determining the target query item in the intelligent cockpit according to the constructed 3D model of the intelligent cockpit and the attribute information comprises the following steps:

comparing the object coordinate pointed by the target gesture with the position coordinate of each query item in the 3D model of the intelligent cabin, and comparing the object identifier pointed by the target gesture with the identifier of each query item in the 3D model of the intelligent cabin;

and determining a target query item in the intelligent cabin according to a comparison result of the coordinates and the identification.

4. The method of claim 1, wherein retrieving the associated content of the target query term from the electronic specification comprises:

retrieving content associated with the target query term from a locally stored electronic specification;

when the associated content of the target query item is not retrieved from the locally stored electronic specification, retrieving the associated content of the target query item from the electronic specification of the cloud platform;

uploading the target query item to a cloud platform to update an electronic specification when the associated content of the target query item is not retrieved from the electronic specification of the cloud platform.

5. The method according to any one of claims 1-4, wherein a sound box is installed in each area of the intelligent cabin;

when the associated content is a text description, retrieving and playing the associated content of the target query item from the electronic specification, including:

and playing the text description of the target query item in a voice mode by using a sound box in the awakening area.

6. The method according to any one of claims 1-4, wherein a display screen is installed in each zone of the intelligent cockpit;

when the associated content is a video, retrieving and playing the associated content of the target query item from the electronic specification, including:

and using a display screen in the awakening area to screen and play the video of the target query item.

7. The method in accordance with claim 6, wherein when the associated content of the target query term exists in a video case of a live presentation, the method further comprises:

and playing the video case by using a display screen in the awakening area.

8. The method of any one of claims 1-4, wherein when statistics exist for the retrieved associated content of the target query term, the method further comprises:

and recommending the suggested video of the target query item according to the statistical data and playing the recommended video.

9. An electronic specification retrieval apparatus, comprising:

the system comprises a plurality of sound pickups, a voice recognition module and a voice recognition module, wherein each sound pickup is preset at different positions of an intelligent cabin and is configured to receive voice wake-up words in the intelligent cabin;

the intelligent seat cabin comprises a plurality of cameras, wherein each camera comprises a sound pick-up at a preset position and is configured to collect images in the intelligent seat cabin;

a central processor configured to retrieve voice wake-up words received by the microphone to enable an electronic specification use mode; opening at least one camera related to the position according to the position information of the sound pickup in the intelligent cabin, and acquiring an image of the intelligent cabin acquired by the opened camera; gesture recognition and tracking are carried out on the intelligent cockpit image, and attribute information of a target gesture is determined; determining a target query item in the intelligent cabin according to the constructed 3D model of the intelligent cabin and the attribute information; and retrieving the associated content of the target query item from the electronic specification and playing the associated content.

10. A car machine is characterized by comprising a central processor, a memory, a plurality of sound pickup devices and a plurality of cameras, wherein the sound pickup devices and the cameras are arranged at different preset positions in an intelligent cabin of the car machine, the intelligent cabin is divided into a plurality of areas, each area comprises at least one sound pickup device and camera, a sound box and a display screen, and the memory, the sound pickup devices, the cameras, the sound box and the display screen are connected with the central processor through a bus;

the sound pickup is configured to receive voice wake-up words in the intelligent cabin;

the camera is configured to acquire images in the intelligent cabin;

the sound box and the display screen are configured to play the content retrieved from the electronic specification;

the memory includes a data storage unit configured to store electronic instructions and a program storage unit configured to store a computer program;

the central processor performs the method of any one of claims 1-8 in accordance with the computer program.

Images