Adaptive transitions for automation in cars, trucks, buses and motorcycles

: Automated vehicles are entering the roads and automation is applied to cars, trucks, buses, and even motorcycles today. High automation foresees transitions during driving in both directions. The driver and rider state become a critical parameter since reliable automation allows safe intervention and transit control to the automation when manual driving is not performed safely anymore. When the control transits from automation to manual an appropriate driver state needs to be identified before releasing the automated control. The detection of driver states during manual and automated driving and an appropriate design of the human–machine interaction (HMI) are crucial steps to support these transitions. State-of-the-art systems do not take the driver state, personal preferences, and predictions of road conditions into account. The ADAS&ME project, funded by the H2020 Programme of the European Commission, proposes an innovative and fully adaptive HMI framework, able to support driver/rider state monitoring-based transitions in automated driving. The HMI framework is applied in the target vehicles: passenger car, truck, bus, and motorcycle, and in seven different use cases.


Introduction
Automation levels of all kinds of vehicles are increasing. Tasksharing between vehicle automation and humans is a major challenge and research topic especially in Society of Automotive Engineers (SAE) levels 3 and 4. Automation will support and ease mobility but the human needs to remain in the loop at least temporarily in the next generations of vehicles. The humanmachine interaction (HMI) needs knowledge about the state and intention of both partners to enable safe cooperative control. The transition from manual driving to automated driving and vice versa has been recognised as a major challenge [1][2][3][4][5]. To decide on how and when to communicate the transition with the driver, the vehicle should consider different inputs such as environmental and traffic conditions, driver state, driver characteristics, and the available sensing platform of the vehicle. This is a complex task that requires decision-making algorithms that consider the current system status, but also make predictions about upcoming states (e.g. road work ahead, driver readiness time, after sleeping or attentive road scanning). In addition, both driver reaction and response need to be included.
A current state-of-the-art assessment gives an overview of the existing HMI information and warning technology, different driver/ rider state sensing equipment, as well as the application of driver/ rider systems [6]. The maturity level of the relevant driver/rider state sensing equipment and algorithms still differs. Driver/rider states such as sleepiness, fatigue, and stress are fairly well-defined in research and first in-vehicle systems for their detection exist in the market [6]. Resting and physiological states such as fainting, hyperthermia, or dehydration, relevant primarily to special user scenarios (e.g. professional long-term drivers or motorcycle riders), so far lack research attention. Such driver/rider states are less understood, and applications and products are rare. Automatic classification of emotions and their impact on HMI are a very innovative area of research with increased potential for automation since they are currently of major interest in marketing research. They could potentially be applied to determine the usefulness of various interactions and HMI strategies in vehicles [6]. Detection systems existing today are primarily used to make the driver aware of their sleepiness or lack of attention, or recommend the drivers to handle (e.g. decrease) their sleepiness by taking a break. The driver response is determined using simplified driver attention and driver action analysis. In case the driver does not respond, warnings and eventually automated functions such as automatic emergency braking are triggered. So far, the extent of the drivers' response often constitutes the main factor in adjusting vehicle automation in IET Intell. Transp. Syst., 2020, Vol. 14 Iss. 8, pp. 889-899 This is an open access article published by the IET under the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/) risky situations. However, adapting the HMI strategy based on the driver state and user profiles to support transitions and advanced driver assistance systems (ADAS) offers promising advantages. The first attempts in this direction include, e.g. Daimler's 'Fit & Healthy', which uses fragrances, cold air, lights, and seat position to counteract sleepiness [7].
In this study, a HMI framework that enables modelling of personalised adaptation considering driver/rider states and available HMI elements is presented. The framework is general and therefore flexible to adapt to all types of vehicles and driver/ rider monitoring sensors, as well as HMI elements. It is meant to be used by vehicle manufacturers, suppliers, and others involved in the development of either sensors, vehicle functions, or actuators (e.g. HMI elements) when defining use cases (UCs) for the uservehicle interaction. The term user could be a driver or a motorcycle rider. To set a focus of the framework, this study focuses on scenarios with bi-directional transitions between automation and manual driving. This study also proposes a general HMI architecture to depict how the concepts could be realised through components.
This framework is based on the European-funded project ADAS&ME, an acronym for 'adaptive advanced driver assistance systems to support incapacitated drivers and mitigate effectively risks through tailor made HMI under automation'. The project investigates adaptive ADAS and automated functions that incorporate driver/rider states, as well as situational and environmental context and adaptive interaction to automatically transition control between an automated vehicle and the driver/ rider. In the project, automation is considered to be an important part of the HMI, since automation is a major channel of interaction with the driver. It is used to support drivers in their current state, enhance intuitiveness, acceptance, and safety. The project works on a great variety of vehicle types such as conventional and electric cars, trucks, buses, and motorcycles. The challenge for ADAS&ME HMI is to find a common architecture and framework that suits all vehicle types and to be flexible enough to adapt to distinct requirements. All vehicles have in common that the transition of control needs to be handled under different levels of automation, and that driver and environmental states are factors that influence the decision system. Furthermore, it should enhance intuitiveness and acceptance, provide constant mode awareness, support safe and comfortable transitions, and guide drivers to different states. Experimental research is being carried out on algorithms for unobtrusive driver state monitoring (of driver/rider states such as rest, sleepiness, stress, inattention, and impairing emotions) as well as on personalised HMIs and automated driving transitions.
In total, ADAS&ME focuses on seven UCs that cover different types of transportation, different levels of automation, different combination of HMI elements, sensors, and driver states. The seven UCs in ADAS&ME are defined to present the functional requirements of a system, in a detailed, clear, and easy-to-learn way. The UCs are as follows (details on the UCs as well as their importance for the population can be found in [8]

HMI in automated vehicles
State-of-the-art in HMI for automated vehicles mainly facilitates transitions between manual and automated driving by indicating the availability of automated driving and warnings when automation will be deactivated by the system [9]. Some manufacturers provide information during automation to enhance the comprehension of the limits of the automated system and information during manual driving about oncoming periods of possible automation. Manufacturers apply some kind of driver state monitoring which is most of the time a variation of the dead-manswitch. In this approach, the driver is requested to signalise availability by touching the steering wheel when the system asks for it. The moment of asking the driver to confirm availability is based on the contextual information and sensor data quality. This approach assures that drivers will be able to take back control when the automated system plans to hand over control to the driver. However, this approach interrupts the non-driving-related-tasks at sequences that are not self-paced by the driver. It prevents a flow experience of non-driving-related-tasks and makes automated driving periods annoying. Additionally, misuse is possible and so far most systems do not even control if the eyes are on the road. ADAS&ME investigates how driver and rider states can be used during automation to improve the knowledge about driver availability and at the same time extend interruption-free driving time.
ADAS&ME also uses driver state monitoring under manual driving periods to offer supportive automation when the driver is sleepy, distracted, stressed, or for any other reason absent. The ADAS&ME decision support system decides when automation could be perceived to be more helpful or safer than manual driving and offers automated functions when available. Automation is used inside a general HMI strategy to raise awareness about the driver state, support drivers to recover, and protect the ego vehicles and others from collisions.

Advanced rider assistants systems (ARASs)
While ADAS is a quite established term already, the acronym ARAS, which is the ADAS counterpart for powered-two-wheelers, has only gained attention over the last few years. The acronym was used for the first time in the SAFERIDER EU-funded project alongside with the acronym on-bike information systems (OBISs) [10][11][12][13][14]. Especially automation functions represent a completely new feature for motorcycles, i.e. at the moment there are no powered-two-wheelers on the market with this functionality. The challenge lies in their very complex 3D dynamics: they are not statically stable and the rider directly influences their behaviour by moving his/her body. ARASs are inspired by the developments in the automotive field, e.g. the antilock-braking system (ABS), but their definition and implementation proceed on a completely different lane since they have to take into account the quite significant motorcycle constraints. This has also led to the development of specific motorcycle functions, such as cornering ABS (allowing the rider to brake safely while turning without falling down) or anti-wheelie, but also to discard others that have no use on a motorbike.

Adaptation to driver and rider states in automated vehicles
The modern HMI faces challenges of not overloading or distracting the driver. Several research projects passed the way to what we see in the market today and where we stand in research now, such as: 'adaptive integrated driver-vehicle interface' (AIDE) [15], 'generic intelligent driver support system' (reported in [16]), 'application of real-time intelligent aid for driving and navigation enhancement' [17], 'communication multimedia unit inside CAR' [18], and the 'advanced telematics for enhancing the safety and comfort of motorcycle riders' (SAFERIDER) [13], SAFESPOT [19], SAFEWAY2School [20], TEAM [21], and I-VITAL [22]. These and other research projects resulted in various sets of recommendations for adaptive HMI development. Among them are the eight golden rules regarding interaction design proposed by Schneiderman [23], the ten heuristic usability principles defined by Nielsen as an extension of the golden rules [24], and a set of recommendations or principles in the areas of design, installation, information presentation, and system behaviour of in-vehicle information and communication systems proposed by the European Statement of Principles on Human Machine Interface (ESoP) [25]. With respect to the ESoP an updated version considering vehicle automation is missing.
Currently, available HMI components include visual, auditory, haptic, and kinaesthetic modalities, including the interaction towards the driver and rider via automated interventions to the vehicle lateral and longitudinal control. Several car companies launched driver monitoring and assistance systems, monitoring driving behaviour (such as lane keeping and/or driving patterns) and/or the driver him/herself (via monitoring head and eyes and/or physiological parameters), detecting drowsy/sleepy or inattentive/ distracted drivers. Once anomalies are detected, systems advices drivers to rest via graphical symbols (such as a coffee cup) and/or texts (such as 'take a break') and audible signals (BMW [26] , and others). Warnings of some companies are done in stages (Ford, Toyota, SAAB, Jaguar Land Rover, and others) starting with soft notifications, moving to more intrusive ones, if no action was taken or no acknowledgment was received from the driver. Today the driver state warnings usually do not result in automated driving interventions, but this is to come soon with increasing reliability levels of automation. It is a principal concept in ADAS&ME. Today this is applied in transitions from automated to manual driving. While early level 2 systems simply deactivated all automated control when the driver did not react to take over requests, recent systems apply minimal risk manoeuvres such as a full stop (Ford) or take over and apply the brakes smoothly (Toyota). A system developed by Bosch not only warns the driver of his/her state but also reminds of the danger of nodding off at the wheel. Other companies used vibrations in the driver's seat, which then became common in trucks. For example, a system by Citroën [35] monitored lane-keeping and alerted the driver by vibrations on the right or left side of the seat depending on which lane is dangerously close or crossed. Existing concepts are able to connect the information with the maps and recommend or guide the drivers to the next possible rest area.
The development, integration, and research of HMI components are speeding up and new prototypes in that field advance increasingly. New emerging materials and technologies are becoming real in the near future that can be used as HMI elements. Furthermore, the potential of 3D virtual images or new technological e-textiles and plastics are developed for usage in these environments [36]. Speech recognition technology is improving in maturity to be used as a communication layer, instead of complex manual actions, such as menu selections and destination entry to the navigation system, reducing driver distraction [36]. Other HMI components may include configuring the climate control to interact with the driver with fresh air or fragrances in case of fatigue [36], blocking the possibility of use or even support the safe use of certain functions or devices to increase driver attention. The integration of automated functions in the HMI strategy by using the kinaesthetic effect on the driver when longitudinal and lateral control is activated is state-of-the-art in series development but little investigated in research yet.
Including driver states in the system adaptation allows the system to gain awareness of the driver or rider and its capabilities based on recognition of the current state. However, little research is done on the combination of multi-modal HMI elements and adaptation to the driver/rider state. Hence, ADAS&ME investigates the area of adapting HMI elements to account for driver states and environmental conditions.

Personalisation and personalised adaptation in autonomous vehicles
The introduction of multi-modal HMI elements in autonomous vehicles has introduced new layers of interaction complexity because of the increasing number of on-board functions (not only related to the driving task) and the massive introduction of ADAS systems. Drivers are not always capable of perceiving and understanding the plethora of messages produced by the vehicle/ system [37] due to their physiological state (sleepy, absent-minded etc.) and complex traffic environment. The user acceptance of intrusive systems also underlies a large variance of user's preferences. As a prominent example, personalised designs of HMI may embrace speech output or be reduced to minimalistic iconmessages. Also, the timing of warnings should be personalised, as some drivers prefer early warnings and others prefer warnings and interventions only in the latest reasonable moment. Towards this direction, the development of HMI technologies needs to be context-aware (i.e. driver, vehicle, and environmental state) as well as to be adapted to the user's characteristics, needs, and expectations.
According to the literature [38], research efforts have targeted the potential of developing a personalised, safe in-car HMI that automatically adapts to the targeted design and interaction concept, as well as to the personal needs of the driver. In the same context, various efforts have been made to increase the driver's performance and satisfaction by employing personalised HMI technologies. For instance, spoken dialogue systems can be used to operate devices in the automotive environment. Since drivers using these systems usually have different levels of experience, a method has been proposed on building a dialogue system in an automotive environment that automatically adapts to the user experience [39]. Another example of personalised interaction was developed in the AIDE project [40] that aimed at investigating the integration of different ADAS systems and in-vehicle information systems taking into account the driver and traffic conditions. In particular, the information presented to the driver could be adapted on the basis of environmental conditions (weather and traffic), as well as on the basis of assessed workload, distraction, and physical condition of the driver. The myCOMAND case study [41] provides insights into the applicability of personalisation and recommendation approaches for the visual ranking and grouping of items using interactive HMI layout components.
Additional research efforts have focused on the domain of navigation systems, as they demand many highly interactive activities from the driver [42]. Such systems are highly complex since they have countless functions and in some cases coexist with infotainment systems and other components (i.e. radio, phone etc.). Indisputably, during stressful situations, the HMI of the driver navigation system can be made adaptive to reduce the mental workload of the driver. Regarding the personalisation of in-carinfotainment systems, Garzon introduces two different approaches to simplify the task of executing a preferred infotainment feature by either personalising a list of context-dependent shortcuts or by automatically executing regularly used features [43].
In the context of automated driving technology, HMI with driver-system coordination is considered to be of the utmost importance for reducing or even eliminating possible cases when the driver gets confused about the responsibility for monitoring roadways and performing safe operations. From an HMI point of view, the personalisation and personalised adaptation of the available HMI elements paves the way for a harmonious relationship between the driver and the vehicle. According to Endo [44], if the harmonisation between the driver and the vehicle works well, the driver develops a deeper understanding of the process. At the same time, the vehicle continuously adapts to the driver characteristics. Thus, the vehicle becomes personalised and understands the driver's driving style and skills, reacting appropriately. Recently, the research area around nudging has been of increased interest [45]. The aim here is to push users in the wanted direction without patronising them [46]. This might be of importance for future designs of in-vehicle systems.

HMI framework
In ADAS&ME, a generic HMI framework is used to define the HMI strategies for all target vehicles. The framework enables us to define vehicle-system-user interactions, considering the driver/ rider state, personalisation, the environmental context, and all HMI elements. Fig. 1 presents the general structure of the framework.

Framework overview
The framework is inspired by unified modelling language diagrams and adjusted for describing HMIs in detail. These adjusted diagrams are called sequence-diagrams and focus on the interaction between human and machine. Sequence-diagrams were introduced and used before by Kelsch and Flemisch [47][48][49]. The HMI framework used in this study extends the sequence-diagrams with components of the system architecture and identifies complex dependencies regarding system functionality. Owing to this, it is possible to describe the HMI functionality of the automated system. The result is an interdisciplinary way to document interaction strategies. The sequence-diagrams present the key feature of the reported HMI-framework and are described below.

Sketch of the addressed scenario:
One of the most important parts is to get a common understanding of the addressed scenario for a certain HMI. A high-level sketch of the scenario can help to avoid misunderstandings and gives a first impression of the driving situation and the necessity for a well-designed HMI. The scenario sketch should only give a first insight into the situations where a HMI is needed.

Selection of actors:
The second step is to define different actors active in the described scenario. Dependent on the focus of the UC and system architecture the sequence-diagram could include different actors. The focus on the adaptation of HMI elements, considering the driver state, personalisation, and environment, is reflected in the sequence diagram in Fig. 1. Besides the more general actors such as the environment/vehicleto-everything (V2X), the level of automation, the vehicle automation, and the driver himself, we mainly include components to plan and execute the adaptation of the HMI as actors. The adaptation and planning of a tailored HMI strategy are conducted by the driver monitoring, the personalisation, and the decision system. Every system is represented as an actor in the sequence diagrams. All actors have a lifeline, representing the actor in the sequence diagram. Furthermore, all available HMI elements which could be used to interact with the driver are listed as actors. The selection of the actors could vary between different UCs and demonstrators in the project. Since the demonstrators are equipped with different hardware (e.g. HMI elements, driver monitoring systems) different HMI strategies are possible.

Sequence and timing of interaction:
In a sequence diagram, the time is represented from left to right. The sequence diagram enables the designer to plan and describe the sequence of the interactions in a certain scenario. Since all important actors are represented in the sequence diagram it is possible to see the complete composition of used devices and modalities for interaction at a certain point in the scenario. Furthermore, the sequence diagrams allow displaying different actors, interacting in parallel. However, the sequence of actions represents a scenario over time but the distance between elements is not equal to a time that has past. It is common to mark important timeframes by dashed lines (e.g. t1 in Fig. 1).

Flow of information:
The main feature of the sequence diagrams is the description and flow of information between the different actors. Information is represented by arrows which spread from the lifeline of an actor to the lifeline of another actor. Information is flowing from a source (something happens in the environment) to a receiver (sensors detects the change in the environment). Most important, at least from an HMI perspective, is the information that addresses the driver. This information could be highlighted in a different colour. Using the sequence diagram, it is easy to check if and how essential information is communicated to the driver.

Driver states:
In some scenarios, it is crucial to display the current state of an actor (e.g. driver state, automation state, automation level). States can be represented by placing a rectangle with a meaningful headline on the lifeline of an actor to indicate state changes over time or in dependence of an action.

Transition automation to driver, driver to automation
The HMI framework allows to model transitions from vehicle automation to manual driving and vice versa. This can be represented in the framework by starting with the vehicle automation lifeline and triggering a process from this actor, till finally the driver is informed through the appropriate HMI elements and takes over. The same accounts for when starting from the actor driver (when the driver is in charge), detecting that the driver is, e.g. sleepy and might need the vehicle automation function. Again, the HMI elements can be personalised to inform the driver about the availability of the vehicle automation function, until the driver finally accepts it and the vehicle takes over.

Environmental monitoring
During autonomous driving, the system needs to have full control of the vehicle including knowing exactly the environmental state at each point in time. A vehicle that is equipped with autonomous driving functionality should also be aware of the environment even when it is only assisting the driver in the driving task to be prepared to take over the driving. The environmental monitoring includes information from sensors that the vehicle is equipped with. Hence, in the environmental/V2X module all possible sensors that the vehicle is equipped with can be modelled to understand which information can be collected and what can be concluded and detected through these sensors.

Driver state monitoring
The driver state monitoring system is responsible for constantly monitoring the driver and providing information about his/her current state (e.g. sleepiness, distraction, stress, rest, physiological fatigue, and emotions). To this end, it will collect data from a variety of systems and sensors, e.g. through a camera system, microphones, electrocardiogram (ECG) seat, and other physiological sensors and wearable's. From these multimodal signals, the state of the driver is automatically being assessed through machine-learned classification systems in the scope of affective computing [50]. Existing algorithms from other groups within the ADAS consortium can be reused. The framework allows modeling the specific sensors available for use or the specific algorithms used. The state detection is personalised to the individual driving style and to physiological data retrieved from the personalisation system.

Personalisation system
The personalisation system is the enabler for personalisation and consists of two different tasks. One that offers driver identification and another, a profiling mechanism that gathers, stores, and provides relevant information about the identified driver. Information in the later can be static (i.e. does not change over time) and dynamic (i.e. changes over time). Examples of the information include but are not limited to demographics (e.g. age and gender of the driver), driving history, history of the detected driver states, HMI element customisations, past system decisions, respective driver actions etc. Main users of the personalisation system are the decision system, personalised HMI, and driver state monitoring.

Decision system
The decision support system (also called decision support module, abbreviated as decision system in the rest of the paper) is the central component of the framework deciding about the appropriate interaction or transition strategy based on the driver state and environmental information. It is an expert system that uses decision-making activities based on fuzzy rules to inform, notify, or alert the driver as well as request handovers or trigger automation functions when necessary. Decisions are adapted based on the criticality and confidence levels provided by the driver state monitoring, the dangerousness of the environment and the level of required attention provided by the environmental monitoring, and personalised based on the driver profile, behaviour, and past activity retrieved from the personalisation system. The decision system is responsible to realise the interaction strategies utilising HMI modules and automated functions supporting the cases of transitions from automation to driver and vice versa as well as emergency activation of automated functions.

Personalised HMI
The personalised HMI applies rule-based reasoning to produce and realise the personalisation and adaptation decisions towards driver interaction. It receives the decision strategy from the DECISION SYSTEM and harmonises the way those strategies are propagated and transformed into personalised HMI elements. HMI personalisation takes into account the driver profile, state, and environmental situation that are retrieved by communicating with the personalisation system. For example, the system could use louder sound notifications for sleepy drivers or use visual and haptic feedback in loud environments (e.g. open windows or loud voices in the vehicle). The personalisation shall also adopt the intrusiveness of the interventions to the drivers' needs. Thus the intention to deactivate such a system is reduced to a minimum.

System architecture
An information system architecture was designed to depict the framework capabilities to efficiently handle the interface of the automated vehicle with the driver in both situations of transitions from automation to driver and vice versa. The system architecture was developed to properly cover all significant components of the system as well as their interrelationships. Fig. 2 depicts the six components comprising the system and provides the conceptual model of information flow between them. The central component is the decision system, i.e. as previously discussed, employs expert rules from the knowledge base to derive facts and evidence from the driver monitoring, environmental monitoring, and personalisation system and decides on a proper selection of interaction/ transition strategy. With the decision system as the major instance, the system architecture differentiates between the decision system input and output.

Decision system input
Three components, namely the driver monitoring, environmental monitoring, and the personalisation system produce the input information required for the decision system to proceed with the interaction/transition strategies. These are further clustered in two parts: the one that is relevant to the environment and the vehicle including the various equipped sensors. The other one comprised systems related to the driver, which include the functionalities of monitoring and profiling the driver. Information about the driver can be stored in an ontology model [51].

Decision system output
Two components comprise the decision system outputs including the selected personalised HMIs and the automated functions of the vehicle. The HMI interaction strategy as depicted in Fig. 2 integrates the decision system with its subsequent components of the personalised HMI controller and vehicle automation. Parameters of both means of interactions are modified by the personalisation system which bypasses its profiling to configure their functionalities. Most of the components are only sharing information in one direction. However, the decision system requires bi-directional information flow with the personalisation system and vehicle automation. This bidirectional information flow secures the updating of vehicle automation status required and other additional information to serve the profiling mechanism of the personalisation system. In particular, the vehicle automation can provide the decision system with information regarding the available automated functions and the timing constraints involved in their activation, while the decision system can provide the personalisation system with information about system decisions and respective driver actions, serving as historical data input that can be used in the future. Additional information on Vehicle-to-Vehicle/Infrastructure and Controller Area Network (V2X/CAN) is provided to the central component of the system by the vehicle through the appropriate interface.

Application of the HMI framework
The applicability and limitations of the framework are discussed in the following section.

Application of the framework
The framework can be applied in a broader context, but here the focus is on the application of the framework for different types of transportation (e.g. car, motorcycle, truck, and bus), different levels of automation (e.g. SAE levels 3 and 4), different driver states (e.g. the most critical ones such as sleepy, inattentive, and stressed), different HMI elements, and different environmental conditions. In the following, the application of the framework for transition scenarios is depicted based on examples from the ADAS&ME project. While the UCs focus on transition scenarios, the framework is not necessarily limited to scenarios including a transition, and might therefore also be applicable in general for other kinds of scenarios. A more abstract description of the applicability will be provided based on one of the UCs, UC B for the usage in an electrical vehicle.
In the end of the ADAS&ME project, implementation in road vehicles is planned.

UC A: professional long haul trucking:
A primary goal of UC A is to encourage drivers to make greater use of the automated driving functionality. Automated driving functionality is one tool that Scania will be providing drivers to increase their safety, efficiency, and pleasure. The choice to use this tool or not is made by the driver. When the driver's state has an increased possibility to reduce safety or efficiency, the ADAS&ME system is intended to encourage even more handovers to automated driving than would be the case without the system. It is expected that the HMI (by affording understanding, trust, and acceptance) will play a critical role in convincing drivers to exploit the automated driving capability.
Environmental monitoring: The environmental monitoring module provides input to determine whether or not automated driving is possible and for how long.
Driver state monitoring: This module determines the driver state. For UC A, four drivers states that might reduce the capability of safe driving are of interest: strong emotions, distraction, sleepiness, and stress. Additionally, the detection of a state called rest is of high interest during automated driving to allow extended resting times during vehicle operation.
Personalisation system: This module will update the driver display preferences (the look and feel) of the two primary visual displays (instrument cluster, secondary display). For this module, drivers will connect to a specific vehicle (e.g. via Wi-Fi) and the driver display preferences (defined during user profile creation) will be uploaded.
Decision system: When automated driving becomes available (i.e. when afforded by the infrastructure), the driver is notified and encouraged to hand over control to the vehicle. During hand over, a clear simple hand over method will be used. If continuing with manual driving and distraction, sleepiness or stress is detected, then the driver receives information encouraging them to hand over control due to potential compromises to safety or efficiency. Different warnings will be given for different driver states. A key warning feature should mitigate or reduces the potential negative effect of the driver state. For example, the sleepiness warning should rouse the driver. If during automated driving the driver enters a state of rest, feedback will be given to the driver, and a theoretical argument made to change the driver's tachograph (their legal record of work activities) from driving to resting mode. Finally, when the system detects that an upcoming takeover by the driver is required, then, depending on the different driver states, different pre-takeover warnings will be given. The takeover process will be the same for all driver states to avoid confusion. If the driver fails to take over control, then a safe vehicle stop will be executed.
Personalised HMI: This module will adopt the HMI to driver traits registered when they create their user profile (and select their display preferences).

UC B: range anxiety in electrical vehicles:
The goal of UC B is to increase the driver's confidence towards the energy management system of the electric vehicle when needed as well as act as a support to the driver when range related issues occur. For this reason, the role of the HMI is crucial as it is the tool that will enable the UC scenario to happen. One of the scenarios in this UC is 'range anxiety mitigation' (illustrated in Fig. 3). According to

Fig. 3 Application of the framework for UC B -'range anxiety mitigation' scenario
this scenario, the driving session starts with driver identification and system personalisation. Once the driver enters the destination, the decision system initialises the corresponding mission. After updating the remaining range information the default 'range HMI' is presented to the driver. Whenever the driver monitoring module detects anxiety (presented as t1 in the time lap of Fig. 3), the decision system switches to the 'adapted range HMI'. This 'adapted range HMI' remains the active one until the next detected 'neutral' state of the driver.
Environmental monitoring: One of the basic strategies to prevent the range anxiety is to provide accurate and transparent information on the remaining range information [52]. In this scope, the environmental monitoring module will provide extra contextual information to make more trustable range prediction of the vehicle. Concretely the remaining range information will vary according to the weather, the traffic density and the provided route information such as the upcoming uphill and downhill drives. In addition, it will be used to communicate with the infrastructure in the case of a safe stop.
Driver state monitoring: This module will provide information on whether the driver feels too anxious to drive manually when range related issues occur. The driver state that will be taken into account is emotions using visual, audio, and physiological cues.
Personalisation system: Initially, this module will identify the driver using facial recognition. Furthermore, this module will prepare information about this specific driver (e.g. information collected earlier or default values). During the entire driving session, this module will gather, store, and provide relevant information about the driver to other modules, e.g. to adapt the display and its content.
Decision system: If anxiety is detected by the driver state monitoring, the decision system will use context information to define if this state can be classified as 'range anxiety'. In addition, depending on the information received by the environmental monitoring system (mainly related to vehicle's estimated remaining range), it will trigger the appropriate HMI interaction/state. Finally, the decision system is also responsible for transmitting the information that the driver provides through the graphical user interface (GUI), to vehicle automation.
Personalised HMI: This module is responsible for the adaptation of the HMI to driver's specific needs. These adaptations can be on a display level (GUI colours, GUI sound), ambient lighting etc.) as well as on a functional level (content to be displayed, the format of the information communicated). The HMI elements used for this UC are (i) central stack screen, (ii) audio system, and (iii) A-pillar light-emitting device (LED) strips.
Adaptations of the HMI can be triggered by the personalisation system or the decision system. In the first case, the goal of these adaptations will be to offer a tailor-made HMI experience corresponding to the driver's personal preferences and/or needs. In the second case, the objective will be to adapt the content displayed to increase driver's comfort and safety. In the 'range anxiety mitigation' scenario, the display of the estimated remaining range information will be adapted in a way that increases driver's confidence to the system's prediction. In the 'range incident mitigation' and '5 km protection' scenarios, the content will be adapted to provide the driver with alternatives to minimise the impact of an incident on the driving experience.

UC C/D: driver state-based smooth and safe transitions of control/non-reacting driver emergency manoeuvre:
The goal of UC C and UC D is to develop interaction strategies for smooth and safe transitions between manual and automated driving (Fig. 4).
However, drivers may need different information and timing of interaction when they are in different driver states. To ensure maximal comfort and safety, these transitions will be adapted to the present driver states and the needs of the driver.
Environmental monitoring: The environmental monitoring module will provide contextual information (road type, traffic, layout of the road) to plan and predict the possible need for a transition. In addition, it will be used to communicate with the infrastructure in the case of a safe stop.
Driver state monitoring: This module will provide information on whether the driver is attentive and alert enough to drive manually either for taking over after the automated driving or while driving manually already. Driver states that will be taken into account are sleepiness, distraction, stress, and emotions.
Personalisation system: See the general description given in UC B.
Decision system: If a specific driver state is detected by the driver state monitoring, the decision system will use context information to define if this state is considered as an inadequate driver state. In addition, depending on the information received by the environmental monitoring system, the decision system will trigger the driver sate specific interaction strategy. 12 s before the takeover, the warning cascade starts. The timing of the warning is subject to investigation, especially its dependency on driver states and the potential for personalisation.
Personalised HMI: This module is responsible for adapting the HMI to driver's specific needs. These adaptations can be on a display level (e.g. colours, sounds, and ambient lighting) as well as on a functional level (content to be displayed, the format of the information communicated). The HMI elements used for this UC are (i) central stack screen, (ii) audio system, (iii) ambient light display, (iv) nomadic device, and (v) LEDs on the steering wheel.

UC E/F -long range attentive touring/rider faint:
UC E&F will include two active strategies based on two different ARASs. The target of these two UCs is to monitor and help motorcyclists who ride for many consecutive hours: the exposure to the external environment combined with the task of riding, which is physically more demanding compared to other means of transport, together with a constrained posture and vehicle vibrations can induce risky states whose effects need to be prevented or, if this is not possible, mitigated. An example of UC E is illustrated in Fig. 5. The states that are taken into account in UC E are physical fatigue, visual distraction, and secondarily stress, while UC F deals with an extreme sub-state of physical fatigue, i.e. thermal faint. The HMI will be used to effectively communicate with the rider.
Looking at the strategy timing, once one of the aforementioned states is detected, the rider is informed in a given timeframe (e.g. looking at Fig. 5 we plan 7 s for level 1 of physical fatigue and 12 s for level 2 of physical fatigue). On the other hand, in case an active strategy is triggered the timeframe is not fixed: considering the motorcycle complex dynamics and to avoid safety-critical situations the system has to perform a pre-check first (e.g. an activation while the motorcycle is bending must not be possible and is postponed till the vehicle goes straight again). The identification of the rider will be based on the wearable units and not on facial recognition as in the car and bus cases. This will allow HMI adaptation but represents, along with the development, a very challenging and limited case: Constraints such as limited space capacity on the motorcycle and the direct exposure to the external environment reduce feasible solutions (e.g. only high contrast colours combinations can be used because of the direct sunlight).
The presence of such external disturbances suggests the implementation of a multi-modal approach with different sensorial feedbacks simultaneously or in sequence [53]. Therefore, the HMI elements will be able to deliver • Visual feedback: Dashboard, navigation unit, info-helmet (addon module to put on the helmet with a LED array that can be controlled and customised). • Haptic feedback: Vibro-motors can be integrated into the gloves or into the helmet. • Acoustic feedback: Using helmet headphones.
The absence of V2V communication modules in UC E&F will be balanced through visual HMI: the vehicle hazard lights and a specific LED strip mounted on the rear of the helmet can be used to warn the surrounding traffic.

UCG: passenger pick-up/drop-off automation for buses:
The goal of UC G is to support the bus driver when docking the bus stop to pick-up or drop-off passengers. The support is needed to make city bus driving less stressful, to release drivers from driving in the most critical part of their driving, and when their attention is likely distracted by passengers in the bus and outside the bus. It shall also guarantee a smooth, environmentally friendly, and safe deceleration and acceleration, which is very important for elderly passengers and all passengers in general.
Environmental monitoring: The environmental monitoring module will be used to communicate with the infrastructure, so it is clear when the bus is approaching the bus stop area -called safe zone -and to keep track on of vulnerable road users or other vehicles or obstacles nearby, that need to be taken into consideration during the smooth and safe stop.
Driver state monitoring: This module will provide information on whether the driver is attentive and alert enough to take over after exiting the bus stop station at the end of the safe zone area in automated mode. Blink duration based on eye-tracking and a wristworn sensor to measure heart rate variability will be used. The driver states that will be taken into account are sleepiness, distraction, and emotions.
Personalisation system: See general description in UC B.
Decision system: If sleepiness or inattention is detected, the system will not ask the driver to confirm that he/she is able to take back the control. Instead, the automated driving will continue. Personalised HMI: This module is responsible for the adaptation of the HMI to the driver's specific needs. Adaptations can be done for visual feedback using the red-green-blue sensor in the steering wheel, and the LED strips in the window or around the front doors. It could also be a combination of vibrations in seat and warning sounds. The selection of modalities will also dependent on passengers on-board. The bus will always ask the bus driver if she or he is ready to take over or give over control. The confirmation will be based on tapping the steering wheel or by pressing a button. The final solution might be decided by the bus driver himself or herself.

Supporting transitions through adaptation
Driver states and their assessment: In ADAS & ME, the framework is applied to six different driver states. Each of the driver states is covered by at least one UC (see Table 1).
In ADAS&ME, distraction is limited to the scope of visual distraction, which is operationalised as looking away from traffic too often or for too long [54]. The driver state emotion focuses on four emotions: neutral, positive, frustrated, and anxious (in particular range anxiety [55]). The state of physical fatigue is dedicated to the motorcycle UCs E and F. It covers impairment due to long-term exposure to extreme temperatures and/or impeded thermal regulation, and muscular fatigue resulting from repeated and/or prolonged activity. Rest is a state characterised by relaxation, and in most cases, mental and physical inactivity. The purpose of rest is to mitigate fatigue and stress. Sleepiness is the physiological drive to fall asleep [48]. Stress is the 'anticipation of a situation that cannot be mastered using available resources' which is characterised by a feeling of strain and pressure [56].
For each driver state, a set of indicators which defines the sensors that should be used to detect such driver state needs to be defined. The driver states are detected and rated by the fusion of different indicators which are monitored during the autonomous or manual drive. These indicators can be divided into the groups (indicator sets): eye and head information (e.g. blinking frequency, pupil size, and head movement), motion information (e.g. hand position on steering wheel and further video data), cardiorespiratory information (e.g. heart rate, heart rate variability, and respiratory rate), speech features, other physiological information (e.g. skin temperature and electro dermal activity), personal information (e.g. age and gender of driver), and environmental and vehicle data (e.g. time of the day and traffic, velocity).
The intention is to obtain signals through sensors integrated into the car rather than sensors applied on the body for UCs A, B, C, D, and G. In contrast, sensors for the rider monitoring (UCs E and F) will be integrated into the rider's clothing, in which case no eyetracking but only head movement will be used, recorded by accelerometers within the helmet. Non-contact sensor options for obtaining cardiorespiratory information will be radar, magnetic induction [57], and capacitive ECG [58] techniques. Speech features will be recorded by microphones and eye-tracking will be provided by a remote head and eye-tracking system (Smart Eye Pro, SmartEye AB, Gothenburg, Sweden).
Driver states output: It was decided that as an output each driver state will have a value appropriately describing this state. Driver states distraction and rest will have one of the two values indicating whether these states are present or not. Value describing sleepiness, stress, and physiological impairment will represent the severity of the state between uncritical, risky, and critical. Emotions will have one of the four values each representing a different emotion: neutral, positive, frustration, and anxiety.
Confidence level of the driver state outputs: Since driver states are determined through algorithms that partly use sensor fusion or learning-based techniques, an additional indicator is introduced that helps to judge how reliable the algorithm output is. Confidence level is also known as 'confidence definition' and 'quality parameter' is suggested to be measured in per cent and have a value between 1 and 100.
Usage of known HMI concepts: With adaptation concepts in the vehicle, the driver still needs to be aware of the system knowledge of the current driver state. Even after the system has adapted the HMI to a certain driver state, the awareness needs to be communicated to the driver. Hence, the driver should still be aware of the system's assessment of the current driver state. Here, the risk is to have an ever-increasing amount of multi-modal notifications in the driver cabin or rider wearable platform. To keep the additional learning curve imposed on drivers as low as possible, it is proposed to re-use generally known everyday concepts in the design of the notification scheme. To this end, the driver staterelated information given by the system is visually displayed to the driver in the form of specifically designed Emoticons (see examples in Fig. 6). These are already commonly used to convey feelings and emotions in the context of digital social interactions. While not all investigated driver states and their display modalities in the project are readily available for re-use, the styling guidelines as introduced by popular messaging applications together with minor adaptions provide a good base to induce intuitive understanding of the kind of physiological and psychological state that the system has identified the driver to be in.
For acoustic messages, the project applies the concept of Hybricons [59]. Hybricons are hybrid sounds fusing abstract earcons with a controlled urgency and representational auditory icons that tell something about the reason for the warning. The hybricons may as an example be composed of a low urgency beep and a snoring sound to inform a sleepy driver. The auditory icon component might be replaced by a vocal message, e.g. 'you may become sleepy, would you like to buy a coffee at the next shop?'

Next steps and evaluation
The HMI framework is deployed across all presented UCs, with systems being implemented and then evaluated in the context of the ADAS&ME project. In particular, for each UC, specific scenarios have been identified and prioritised and there is already active development towards their realisation in terms of driver state monitoring algorithms, HMI elements, and automated functions. Once developed, the system components will be integrated into vehicles and simulators and evaluated. Road demonstrator vehicles are planned for a final demonstration in Barcelona in 2020. In particular, UC A will be realised on SCANIA's virtual reality simulator, full cabin simulator, and automated truck, UC B will be realised on VEDECOM's 'Wizard of Oz', Renault ZOE, UC C&D will be realised on DLR's automated car and FRAUNHOFER IAO's car simulator, UC E&F will be realised on DUCATI's Multistrada motorcycle and CERTH's motorcycle simulator, and UC G will be realised on VTI's bus simulator. Depending on the UC, the evaluations will take place in a combination of simulators, IDIADA's proving grounds and public roads in Barcelona. IDIADA offers a safe testing environment featuring a high-speed track to be used for testing highway-based UCs, and a dynamic platform of completely flat asphalt surface to be used for urban scenarios. Public roads will also be used for the most mature (and hence safer) technologies, following the path started by the Barcelona Board for Cooperative and Automated Driving [60].

Discussion
The proposed HMI framework and its application in seven very distinct UCs present the ADAS&ME work on a high-overview level. The work in the individual UCs is on-going, still, the HMI framework is fixed and the UCs are described with all HMI elements and transition strategies inside this framework. Based on this status, implementation into simulators and vehicles might require small adaptations, and the final designs of the transitions are subject to iterative and user-centred optimisation. The study focussed on the HMI and transitions, however, the innovative approach is to aim for driver state adaptive HMI and transitions and also considering environmental information and aiming for a high degree of personalisation to reduce the risk of annoyance and the possible wish of deactivating such a system. A robust driver state monitoring and an HMI that does not patronise the drivers wishes shall, together with personalisation, ensure that the drivers accept the system and set the ground for a high take rate.

Conclusion
The HMI as part of the user experience in vehicles supporting different SAE levels becomes increasingly important. Considering driver states in the system adaptation allows the system to gain awareness of the driver and its capabilities based on recognition of the current driver state. However, little research is existing on the combination of multi-modal HMI elements and adaptation to the driver/rider state. This study has presented an HMI framework to be used as a method to define the adaptation of HMI elements to account for driver states, personalisation, and environmental conditions. Furthermore, a system architecture has been presented which gives an overview of how the different parts of the framework are connected and can be implemented. The HMI framework has been applied in seven very different UCs from the EU-project ADAS&ME. That illustrates how this framework can be applied, considering different road vehicles, driver states, combinations of HMI elements, and different levels of automation. The framework and system architecture provide a promising approach to enhance driver and rider assistance systems.

Acknowledgments
This study has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement no. 68890.