Research and development in the field of autonomous vehicles has increased along with related work on automated driving (AD) software. Thorough testing of AD software using simulations must be conducted in advance of testing AD cars on the road. Parameters of the many objects around an AD car, such as other cars, traffic lanes and pedestrians are required as inputs of the simulation. Therefore, the number of parameter combinations becomes extremely large. A combination of parameters is called a test case; hence, the challenge is to search collision test cases from the extremely large number of combinations. A rule-based method is the main focus because an explicit method of searching test cases is required in certain industries in the real world. In this study, a method of rule-based searching for collision test cases of autonomous vehicles simulations is proposed. Simulation models that have rules between an AD car and other cars are defined. Algorithms were also developed to search collision test cases that generate test cases incrementally. Experiments on AD simulations involving the simulation models of a three-lane highway and a signalised intersection were conducted. The results indicate the efficiency of the method.

1 Introduction

Automated driving (AD) is an exciting field in the automotive industry. Research and development into AD conducted by information technology companies has increased along with related research in AD software [1, 2]. To roll out AD cars onto the market, it is estimated that at least 8.8 billion miles (14.2 billion kilometres) of testing, including simulation, are required [3]. Thorough testing of AD software using simulation must be conducted in advance of testing AD cars on the road.

There are two types of simulation [4]: long-distance simulation, such as determining the route from departure point to destination or traffic simulations; and short-distance simulation, such as path planning for changing lanes on a highway or turning left or right at an intersection. A short-distance simulation uses an environment model that has many objects, such as traffic information, other cars and pedestrians. Such a simulation is the main focus because it has more test cases than a long-distance simulation. Fig. 1 shows test cases generated from a combination of environment objects and an AD simulator tests the AD software and provides results. Test cases include parameters such as position, speed and acceleration. Example parameter values include 30 miles/h for speed, 60 ft for the front distance from the AD car and 3 ft/s² for acceleration. Other test cases can have lower values, such as 25 miles/h, 30 ft and 2 ft/s², respectively. Test cases are generated from a combination of the parameters.

Details are in the caption following the image — **Fig. 1**
Open in figure viewer PowerPoint

Testing AD software by AD simulator

There are issues with testing AD software in a simulation. First, an extremely large number of test cases are generated by combining many objects around the AD car, such as other cars, traffic lanes and pedestrians. The time and cost needed to conduct such a simulation are huge. Testing AD software is required, but it is not realistic to execute an extremely large number of test cases. Second, the test cases should be defined with some rules. There are related works for these issues, which are categorised as follows.

One category of related work is combinatorial testing. Combinatorial testing [5, 6] is an approach to reduce the number of combinations among parameters. Kuhn et al. [7] showed the cumulative percentage of faults triggered by the number of parameter conditions: 97–99% of faults were triggered from only two or three parameters in a software program. Combinatorial testing is based on such statistical data and can reduce the number of test cases. However, the rationale of reduction is based on the statistical faults of software, not driving faults. Driving faults involve collision, speeding and driving outside of a traffic lane. These driving faults have not been investigated statistically for combinatorial testing. If testing the quality of AD software is required, combinatorial testing may be sufficient for this rationale of reducing the number of test cases. If testing AD software is required for driving models, then the combinatorial testing method is not sufficient.

Another category is the probabilistic method. Abdessalem et al. [8] tested advanced driver-assistance systems (ADASs) using multi-objective search and neural networks. They characterised the condition under which a multi-objective search algorithm behaves the same with or without simulation modelling; thus, demonstrating the accuracy of their method. Althoff et al. [9] proposed a model-based probabilistic collision-detection method for AD that computes stochastically reachable sets using Markov chains [10], in which they succeeded in detecting collisions. These probabilistic methods use randomness or probability and can be applied to reduce the number of test cases. However, some companies, especially automotive companies, require an explicit method of reducing the number of test cases because they want to select the parameters on their own. Thus, rule-based reduction of the test cases is the main focus.

The final category is on testing AD software through simulation [11, 12]. Our study was based on the following studies on such testing. Stellet et al. [12] proposed a survey and taxonomy on current AD approaches to partition the problem of testing ADASs [13]. Their taxonomy framework permits the consideration of open research questions that have to be answered to pave the way for future advanced AD. They identified two complementary axes of defining and deriving test scenarios, i.e. test-scenario inferences and test-scenario scopes. Test-scenario inferences are divided into data-driven and knowledge-driven approaches. Test-scenario scopes are divided into case-by-case and traffic-based. The method involves the data-driven and case-by-case axes to derive test cases. Creating test scenarios for automated vehicle simulation [14] is based on real-world traffic data and a crash database for their simulation model. The proposed method in this paper is not dependent on the effort to create real-world traffic data, but based on the logical rules of simulation models. It enables test cases to be created more easily than by that of this related work. A graph model for cooperative collision warning system [15] applies graph theory to describe cooperative car models and warning collisions. The graph model describes types of relations between cars, but it cannot describe combinations of relation status.

Other related work involves AD simulator architecture. Pereira and Rossetti [4] presented an example of an automated vehicle simulation. The simulator's functions include automated vehicle agents, a controller for all the agents and traffic data. Input data is created to be used by the simulator. The data includes a traffic route for an automated vehicle, other vehicles, pedestrians and so on. Simulations are executed with the data and results are obtained. Zofka et al. [16] presented an example model of an automated vehicle simulation, which includes environment models, a human driver, sensors and the vehicle's dynamics. Broggi et al. [17] proposed an automated-vehicle system involving the vehicle's visual field-of-view and sensor coverage, which has a sensing suite that uses 10 cameras, 4 laser scanners, 1 radar and 16 fixed laser beams. As an example, if input data from each sensor of the automated vehicle was received as Boolean values, the total number of possible combinations would be 2¹⁰ × 2⁴ × 2¹ × 2¹⁶ = 2³¹ ≒ 2.1 × 10¹⁰. In actual test cases, the number would be higher because the values from each sensor would include various types of data. This is an example of an extremely large number of test cases.

Although the above-related works presented potential solutions, some companies require an explicit method of searching for collision test cases because they want to select the parameters based on their test objectives such as collision test cases and non-collision test cases [18]. Hence, a rule-based method for defining test cases is required.

In this paper, a method of rule-based searching for collision test cases of autonomous vehicle simulations is proposed. The method uses models that simulate an AD car and other cars. Each model is defined by a set of rules, specifically, the AD car attempts to reach the goal of a set path while the other cars attempt to prevent this action. The test cases of these models involve collisions with the AD car or preventing it from arriving at the end of its path. Algorithms were developed to search for test cases that are generated incrementally. The algorithms start with test cases that are generated from one car and generate further test cases step by step. Experiments were conducted using an AD-simulation involving the simulation of a three-lane highway and signalised intersection.

Compared with related works, the proposed method has advantages. First, it can search for all collision test cases of the simulation models. Combinatorial testing creates test cases from an enormous number of parameter combinations. However, it does not imply a required number of parameters for vehicle simulation, and the test cases do not include all of those related to collisions. Second, the method has explicit rules for searching test cases. A stochastic approach for creating test cases does not have accountability for how they are created.

The paper is organised as follows. Section 2 gives an overview of the method, simulation models and searching algorithms. Section 3 explains the AD simulation experiments involving a three-lane highway and signalised intersection. Section 4 discusses the evaluation results and future work. Section 5 concludes the paper.

2 Rule-based searching for collision test cases of autonomous vehicles simulation

2.1 Overview

Fig. 2 shows an overview of the method. The method starts by generating parameter combinations of one other car. Simulations with the parameters are executed and the types of results are determined to be ‘Next’, ‘Collision’ and ‘Never-collision’ by the rules defined in Section 2.3. The collision parameters are added to collisions test cases. These steps are one cycle for one other car. The method continues to generate combinations of ‘Next’ parameters, execute simulations with the combinations and determine types of simulation results. The proposed method does not cover all the cases that the AD car might encounter, however, it searches all collision cases under these simulation models faster and reduces the number of cumulative test cases from all collision ones. The method terminates when there are no Next simulation results. If there is a sufficient number of other cars on the road, the AD car must collide with other cars and the method will terminate.

The method consists of two simulation models [19] and one main searching algorithm and its three sub-algorithms.

The models define a path of an AD car and other cars as a model that is defined by rules. The model involves an AD car attempting to arrive at the end of path, i.e. a ‘goal’, and the other cars attempting to prevent the AD car from doing so. The paths of the other cars are defined by initial parameters. An AD car drives in accordance with the instructions of the AD software. Any collisions from the other cars will not enable the AD to arrive at the goal.

Algorithms were developed to prune other cars. When simulation models are defined, search techniques can be used. For instance, alpha-beta pruning [20, 21] is a search technique that searches for a better move by evaluating and searching subsequent moves [22]. The technique starts with a model involving an AD car and another car. It continues to the next simulation by increasing the number of other cars one by one by evaluating the results from the previous simulation.

2.2 Simulation models

The simulation models for an AD car's path under an AD simulation are defined in this subsection. Fig. 3 shows a simulation model of a three-lane highway. We define this model and the model rules as follows. Conditions of these rules are: (i) simulation requirements for the AD car: road type, path and time duration; (ii) physical road conditions: number of lanes, width and traffic rules of the road; and (iii) simulation functions that depend on the simulator: other cars move by using initial parameters and parameters of speed, acceleration, position and size. Once the requirements and simulator are defined, these conditions are determined. Therefore, these conditions are extensive for the simulation models. The three-lane highway simulation models are as follows:

The moving direction of cars is from left to right, as shown in Fig. 3.
The path of the AD car is from the right lane to centre lane.
The AD car attempts to arrive at the goal within a limited amount of time.
Other cars attempt to prevent the AD car from arriving at the goal. The AD car senses the other cars within a defined distance. The AD car detects them, determines the next action and carries out that action.
When the AD car could not arrive at the goal within the limited amount of time, it means the other cars collided with it.
Every move of the AD car and other cars is measured within a discrete amount of time, for instance, every 0.1 s for 10 s.
The AD car moves in accordance with the instructions of the AD software.
The other cars move in accordance with their initial parameters. All other cars’ paths are determined with these parameters and cannot be modified during execution.
The maximum number of other cars is five.

Fig. 4 shows the simulation model of a signalised intersection on a two-way traffic system. The model rule is as follows:

The path of the AD car is from south to west at the intersection.
To simplify the model, the environment is as follows:
- ○ The signal is green for the south–north direction
- ○ There are no pedestrians due to restrictions of the simulator. However, all parameters on the simulator are included.
- ○ All cars are moving on the road. No cars go off the road.
The AD car attempts to arrive at the goal to the west in a limited amount of time.
To prevent the AD car from arriving at the goal, other cars move from south to north, north to south, and west, east, or straight at the intersection.
When the AD car could not arrive at the goal within the limited amount of time, it means the other cars collided with it.
Every move of the AD car and other cars is measured within a discrete amount of time, for instance, every 0.1 s for 10 s.
The AD car moves in accordance with the instructions of the AD software.
The other cars move in accordance with their initial parameters. All other cars’ paths are determined with these parameters and cannot be modified during execution.
The maximum number of other cars is five.

2.3 Algorithms

The main searching algorithm and its three sub-algorithms are illustrated and defined in this section.

First, the terms for the simulation results are defined as follows:

Next: Another car is expected to collide with or prevent the AD car from arriving at the goal when another car is added.
Collision: Another car collided or prevented the AD car from arriving at the goal.
Never-collision: Another car never collided or prevented the AD car from reaching the goal because the other car never entered a path area of the AD car.

The concept of the main algorithm is illustrated in Fig. 5. Figs. 5a –c show examples when other cars are added to the simulation model incrementally. Fig. 5a shows that when another car (identified as 3) and an AD car were simulated in the model, the AD car could arrive at the goal with no collisions. Fig. 5b shows that when another car, 5, was added to, the AD car could still arrive at the goal with no collisions. However, Fig. 5c shows that when another car, 11, was added, the AD car collided with another car and could not arrive at the goal. Fig. 5d shows the ‘Never-collision’ result in which another car never prevented the AD car from arriving at the goal because this other car never entered the path area of the AD car. The path area is the rhombus-shaped area in which the AD car can move. The path area can be calculated from the simulation model. This other car is not expected to collide with the AD car when another car is added. Thus, this Never-collision car should be pruned if other cars that could collide with the AD car are searched. Fig. 5d shows the ‘Next’ result in which another car entered the path area of the AD car but did not collide with it. This other car is expected to collide with the AD car when another car is added. To collide with the AD car, the added car must at least enter the path area of the AD car. Thus, if other cars that could collide with the AD car are searched, this Next car is kept for the next simulation and is combined with other Next cars. Fig. 5e shows the ‘Collision’ result in which another car collided with the AD car. This other car will no longer be used because it has already collided with the AD car. Thus, this Collision car should be pruned if other cars that could collide with or prevent an AD car from arriving at the goal are searched. In this way, when the number of other cars is increased in a model, an AD car is expected to collide with one of them since the road is occupied by other cars and it is getting more difficult for the AD car to arrive at the goal. The number of other cars that can collide with the AD car is limited because if the number of them is large enough for them to occupy the road completely, the AD car will definitely collide with one of them. To collide with an AD car or prevent it from reaching the goal, Collision and Never-collision cars should be pruned and only Next cars should be kept. The main algorithm searches for cars and combinations of them that can collide with the AD car or prevent it from arriving at the goal.

Fig. 6 illustrates an overview of the main searching algorithm. The overview of algorithm is also described as follows:

(i) Inputs are parameters of other cars and model rules.
(ii) All combinations are generated from parameters.
(iii) An AD simulation is conducted with the combinations and model rules, providing the results.
(iv) The simulation results are evaluated and distributed in accordance with the evaluation.
- The results that another car won, ‘Collision’, are pruned.
- The results that another car lost, ‘Never-collision’, are pruned.
- The results that another car may collide with the AD car when another car is added, ‘Next’, are kept for the next simulation.
A loop starts, where n is the number of other cars.
(v) All combinations are generated from the Next of n− 1 other cars and the Next of one other car (not the Next of n− 1 other cars; the Next of one other car involves a possibility of colliding with the AD car). The combinations are evaluated if collisions with any other car occur.
- The combinations in which collisions of any other car occur are pruned.
- The combinations in which collisions of any other car do not occur are kept for the next simulation.
(vi) An AD simulation is conducted with the n other cars. The simulation results are evaluated and distributed in accordance with the evaluation.
- The results that another car won, ‘Collision’, are pruned.
- The results that another car lost, ‘Never-collision’, are pruned.
- The results that another car may collide with the AD car when another car is added, ‘Next’, are kept for the next simulation.
The loop exits if the number of Next is empty or the loop count reaches n times.
(vii) Outputs are all ‘Collision’ results of other cars.

The definitions of terms and variables for the algorithms are as follows:

Terms:

AD Car : a car controlled with AD software on the AD simulator
Other cars : cars other than the AD car on AD simulator
Collision : other cars that collide with the AD car or prevent it from arriving at the goal
Never-collision : other cars that never collide with the AD car or prevent it from arriving at the goal
Next : other cars that have the possibility of collision in the next simulation by adding another car Variables:
ob : out-of-bounds status. Other cars move off the roads (yes = 0 or no = 1).
goal : goal status. The AD car arrives at the goal (0 or 1).
sensing : sensing status. Other cars move into the AD car's sensing area (0 or 1).
collision : collision status. Other cars collide with the AD car (0 or 1).
in-ad-path : moving status. Other cars move into the AD car's path area (0 or 1).
collision-each : collision status. Other cars collide with one of the other cars (0 or 1).
n ：number of other cars (1, 2 or 3).
V : set of parameters values in another car, e.g. (right, car, 10,…), (right, bus, 10,…), (right, motorbike, 10,…).
id : the number assigned for V (1, 2 or 3).
C(n) : set of id s of n other cars, e.g. (9), (1, 5) or (3, 15, 256).
N(n) : set of id s of n other cars that are Next.
W(n) : set of id s of n other cars that are Collision.
L(n) : set of id s of n other cars that are Never-collision.
R(n) : set of AD simulation results for each C(n) and AD car.
Coli2: set of collision-each in C(2), e.g. (1, 15), (23, 56) or (34, 234).

Algorithm 1 (Fig. 7) shows the main searching algorithm. The algorithm loops on the basis of the number of other cars starting from one to the maximum number. Algorithm 2 (Fig. 8) shows the evaluation sub-algorithm for the results of another car. AD simulations are conducted with all combinations of parameters for another car and the results are obtained and evaluated. Algorithm 3 (Fig. 9) shows the evaluation sub-algorithm for the results of multiple cars. Table 1 shows an evaluation of the status results. Such evaluations are determined on the basis of the rationale to justify if it is worth testing. Algorithm 3 is different from Algorithm 2 in terms of evaluating the status or results. Algorithm 4 (Fig. 10) prunes the combinations that will include collision-each during the next simulation.

Table 1. Evaluation of status results

Goal	Collision	Sensing	In-ad-path	Evaluation	Rationale
0	0	0	0	Never-collision	Other car never collides with AD car if it is combined with others. Neither in-ad-path nor sensing occurs.
0	0	0	1	Next	Other car may collide with AD car if it is combined with others.
0	0	1	0	Never-collision	Other car never collides with AD car if it is combined with others. In-ad-path never occurs.
0	0	1	1	Next	Other car may collide with AD car if it combines with others.
0	1	0	0	Never-collision	This never occurs. Sensing must occur in advance if collision occurs.
0	1	0	1	Never-collision	This never occurs. Sensing must occur in advance if collision occurs.
0	1	1	0	Never-collision	This never occurs. In-ad-path must occur in advance if collision occurs.
0	1	1	1	Collision	Other car collides with AD car when collision occurs after sensing.
1	0	0	0	Never-collision	Other car never collides with AD car if it is combined with others. Neither in-ad-path nor sensing occurs.
1	0	0	1	Never-collision	Other car never collides with AD car if it is combined with others. Sensing never occurs.
1	0	1	0	Never-collision	Other car never collides with AD car if it is combined with others. In-ad-path never occurs.
1	0	1	1	Next	Other car may collide with AD car if it is combined with others.
1	1	0	0	Never-collision	This never occurs. Goal is never reached if collision occurs.
1	1	0	1	Never-collision	This never occurs. Goal is never reached if collision occurs.
1	1	1	0	Never-collision	This never occurs. Goal is never reached if collision occurs.
1	1	1	1	Never-collision	This never occurs. Goal is never reached if collision occurs.

3 Experiments

Two experiments were conducted. One involved an AD car changing lanes from the right lane to the centre one on a three-lane highway. This experiment is called Highway. The other involved an AD car turning from south to west at an intersection. This experiment is called Intersection. Figs. 3 and 4 illustrate the environment and car behaviour in Highway and Intersection, respectively.

3.1 Implementation

The algorithms were implemented into script language programs. These programs were executed on a laptop with a 2.7 GHz dual-core, 8 GB memory and 125 GB SSD. The method was compared with a baseline method. The baseline method procedure is as follows: first, an AD simulation is conducted with sampling combinations of a fixed number of cars, for instance, five other cars. Second, the results of the AD simulation are analysed in terms of collision and input-data patterns. Third, important parameters are determined. Finally, test cases are generated for each number of other cars. The baseline method generates test cases in one step. In contrast, the method generates test cases incrementally.

Highway experiment:

Lane: three values, e.g. right, centre and left.
Size: three values, e.g. bus, car and motorbike.
Distance: seven values, e.g. 100, 80, 60, 50, 40, 30 and 20 (ft).
Speed: three values, e.g. 80, 60 and 40 (miles/h).
Acceleration: three values, e.g. 10, 6 and 4 (ft/s).
LaneChange: three values, e.g. stay, right and left.
AcutalLaneChange: two values, e.g. go and no go.

The number of parameter combinations in Highway was 3 × 3 × 7 × 3 × 3 × 3 × 2 = 3402. When the number of other cars is five, the total number of parameter combinations is equal to 3402⁵ ≒ 4.6 × 10¹⁷.

Intersection experiment:

Direction: two values, e.g. north and south.
Size: three values, e.g. bus, car and motorbike.
Distance: seven values, e.g. 100, 80, 60, 50, 40, 30 and 20 (ft).
Speed: three values, e.g. 20, 10 and 8 (miles/h).
Acceleration: three values, e.g. 4, 2 and 1 (ft/s).
ChangeDirection: three values, e.g. straight, west and east.
ActualChangeDirection: two values, e.g. go and no go.

The number of parameter combinations in Intersection was 2 × 3 × 7 × 3 × 3 × 3 × 2 = 2268. When the number of other cars is five, the total number of parameter combinations is equal to 2268⁵ ≒ 6.0 × 10¹⁶.

3.2 Results

Figs. 11a and b show the results of Highway and Intersection, respectively. Table 2 shows the duration of each experiment in minutes. The duration of the baseline method included the total time of manual analysis for all numbers of cars because it was conducted with a maximum of five other cars. The findings are described below.

Table 2. Total execution duration (minutes)

Number of other cars	Highway		Intersection
Number of other cars	Baseline	Proposed	Baseline	Proposed
1		0.1		0.3
2		0.8		1.0
3	n/a	20.0	n/a	20.6
4		84.0		96.8
total	∼200	104.9	∼220	118.7

First, the proposed method finished the algorithms at four other cars before reaching the maximum number of other cars. The baseline method continued to calculate to the assigned maximum number of cars, e.g. five cars. Both experiments were compared with the four other cars. Second, the cumulative number of test cases increased exponentially for both experiments and methods. This exponential increase was due to the increase in the number of cars. Each polygonal line has a different slope. At the beginning of Highway and Intersection when the number of other cars was 1 and 2, respectively, there was no outstanding difference between the baseline and the proposed method. However, when the number of cars increased to 3 and 4, respectively, the cumulative number of test cases with the proposed method increased less than that with the baseline method. Third, the cumulative number of test cases in Intersection was greater than that in Highway. This difference in the cumulative number is thought to be due to the complexity of the road environment. Finally, the total execution duration with the proposed method was less than that of the baseline method for each experiment. There was not much difference in execution duration with the proposed method between Highway and Intersection. The duration of the proposed method depends on the number of test cases; it takes about 70 s per ten thousand test cases.

4 Discussion and future work

This section discusses the efficiency and effectiveness of the proposed method through the empirical experiments.

As shown in the results, the proposed method completed the simulations after four other cars in both experiments. The method searched all Collision cases by increasing the number of other cars incrementally and searching Collision and Never-collision cars. Furthermore, the numbers of test cases, which is equal to that of AD simulation executions, was less than that with the baseline method. On the other hand, the cumulative number of test cases in Intersection was greater than that in Highway. The simulation model of Intersection was simplified in which some objects were fixed, such as no pedestrians, green signal and no ob. These simplifications were restrictions for other cars to prevent the AD car from reaching the goal. More test cases were necessary for other cars to collide with the AD car. Thus, the cumulative number of test cases in Intersection was greater than that in Highway. In Intersection, the method searched all Collisions cases. Therefore, it is confirmed that the method searched all Collision test cases in Highway and Intersection.

For the clarity of the rules defined in the method, as described in Section 2.3, the method includes rule-based algorithms, which can be verified since they do not use random or probability functions. The algorithms were implemented into script language programs. The results from the two experiments indicate that the algorithms are superior to those of the baseline method. It is confirmed that the rule definitions were clear and confirmed the method's efficiency through the experiments.

The efficiency of the method is based on the coverage of test cases and execution costs and duration. Regarding the coverage of test cases, the challenge is to search all test cases, such as colliding with the AD car or preventing it from arriving at the goal. The algorithms were developed with rules to cover all test cases generated with the method that collide with the AD car. The method searches and comprehensively all test cases for colliding with the AD car and preventing it from reaching a goal, and these are the minimum number of test cases. Regarding the method's execution, as shown in Table 2, its duration was less than that with the baseline method, indicating its superiority in terms of duration and costs metrics and confirming its efficiency.

A short-distance simulation was the main focus of this paper, and simulation models of changing lanes on a highway and turning left at an intersection were developed. The simulation models were simplified from real environments, for instance, no pedestrians appeared in the Intersection experiment. In fact, there are many other environments for AD, such as roundabouts, T-intersections and five-way intersections [23]. If a more realistic AD simulation is required, more objects are required to be defined in the models. If the models involve more objects of real environments, the number of object combinations is expected to be larger. The method should be applied for each type of intersection or highway. Thus, another challenge is using a huge number of combinations of intersections, highways, junctions and so on, to test if an AD car can reach its destination [24]. When the environment changes such as pedestrians, weather and coefficient of friction for the road, the method can be applied to the environment because the rules used in the method have versatility. However, when the environment becomes complicated, the cumulative number of test cases is expected to increase, requiring the rules to be tuned. Future work will include using testing techniques other than the all-combinations-testing one used for this study such as equivalence partitioning, boundary-value analysis and a classification tree [25]. Equivalence partitioning and boundary-value analysis can be applied to the issue of the number of values to analyse those values. Equivalence partitioning can reduce the number of values by using one value to represent equivalent values. Boundary-value analysis tends to increase the number of values because values are required to be lower and greater than the boundary. A classification tree can be applied to address the issue of classifying the number of items into groups. The number of items can be reduced by such grouping. These issues are for future work.

5 Conclusions

Research and development in the field of autonomous vehicles has increased along with related work on AD software. Simulations of AD software must be executed in advance of testing AD cars on the road. A short-distance simulation was the main focus because it has more combinatorial test cases than in a long-distance simulation. There are issues with testing AD software with a short-distance simulator. One is the extremely large number of test cases are generated due to the combination of many objects around the AD car. The other is the time and costs of simulation execution are huge. The challenge is to reduce the number of test cases with rule-based algorithms and comprehensive coverage.

A method of searching test cases by applying simulation models of AD software using rule-based algorithms has been proposed. The test cases involve critical moving behaviours of other cars, such as collision or other risks to an AD car. The simulation models define a path of an AD car and other cars attempting to prevent the AD car from reaching the goal. The rule-based searching algorithms were developed to prune the combinations of other cars. Two AD-simulation experiments were conducted. The experiments involved the simulation models of a three-lane highway and signalised intersection.

It is confirmed that the method searches all test cases to be tested and is efficient in practice. Future works include expanding the method to cover long-distance simulations.

6 References

1‘ How Google's Self-Driving Car Works’, http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/how-google-self-driving-car-works, accessed 12 April 2017
2‘ Apple Aims to Test Self-Driving Cars at Military Base’, http://spectrum.ieee.org/cars-that-think/transportation/self-driving/apple-aims-to-test-selfdriving-car-at-military-base, accessed 12 April 2017
3‘ Toyota's Robot-Car Line In The Sand: 8.8 Billion Test Miles To Ensure Safety’, https://www.forbes.com/sites/alanohnsman/2016/10/03/toyotas-robot-car-line-in-the-sand-8-8-billion-test-miles-to-ensure-safety/#21d0f63f16f0, accessed 12 April 2017
4Pereira J. Rossetti R.: ‘ An integrated architecture for autonomous vehicles simulation’. Proc. of the 27th Annual ACM Symp. on Applied Computing, SAC ‘12, New York, NY, USA, 2012, pp. 286–292
5Ballance W. Vilkomir S. Jenkins W.: ‘ Effectiveness of pair-wise testing for software with boolean inputs’. 2012 IEEE Fifth Int. Conf. on Software Testing, Verification and Validation, Montreal, Canada, April 2012, pp. 580–586
6Mirzaei N. Garcia J. Bagheri H. et al.: ‘ Reducing combinatorics in gui testing of android applications’. 2016 IEEE/ACM 38th Int. Conf. on Software Engineering (ICSE), Austin, TX, USA, May 2016, pp. 559–570
7Kuhn D. Wallace D. Gallo A.: ‘Software fault interactions and implications for software testing’, IEEE Trans. Softw. Eng., 2004, 30, (6), pp. 418–421
8Abdessalem R. Nejati S. Briand L. et al.: ‘ Testing advanced driver assistance systems using multi-objective search and neural networks’. 2016 31st IEEE/ACM Int. Conf. on Automated Software Engineering (ASE), Singapore, September 2016, pp. 63–74
9Althoff M. Stursberg O. Buss M.: ‘Model-based probabilistic collision detection in autonomous driving’, IEEE Trans. Intell. Transp. Syst., 2009, 10, (2), pp. 299–310
10‘ Deep Learning Makes Driverless Cars Better at Spotting Pedestrians’, http://spectrum.ieee.org/cars-that-think/transportation/advanced-cars/deep-learning-makes-driverless-cars-better-at-spotting-pedestrians, accessed 12 April 2017
11Berger C. Chaudron M. Heldal R. et al.: ‘ Model-based, composable simulation for the development of autonomous miniature vehicles’. Proc. of the Symp. on Theory of Modeling & Simulation - DEVS Integrative M&S Symp., DEVS 13, San Diego, CA, USA, 2013, pp. 17:1–17:8
12Stellet J. Zofka M. Schumacher J. et al.: ‘ Testing of advanced driver assistance towards automated driving: a survey and taxonomy on existing approaches and open questions’. 2015 IEEE 18th Int. Conf. on Intelligent Transportation Systems, Canary Islands, Spain, September 2015, pp. 1455–1462
13Bengler K. Dietmayer K. Farber B. et al.: ‘Three decades of driver assistance systems: review and future perspectives’, IEEE Intell. Transp. Syst. Magaz., winter, 2014, 6, (4), pp. 6–22
14Zhao D. Lam H. Peng H. et al.: ‘Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques’, IEEE Trans. Intell. Transp. Syst., 2017, 18, (3), pp. 595–607
15Sebastian A. Tang M. Feng Y. et al.: ‘ Multi-vehicles interaction graph model for cooperative collision warning system’. Intelligent Vehicles symp. 2009 IEEE, Xi'an, China, June 2009, pp. 929–934
16Zofka M. Klemm S. Kuhnt F. et al.: ‘ Testing and validating high level components for automated driving: simulation framework for traffic scenarios’. In 2016 IEEE Intelligent Vehicles Symp. (IV), Gothenburg, Sweden, June 2016, pp. 144–150
17Broggi A. Buzzoni M. Debattisti S. et al.: ‘Extensive tests of autonomous driving technologies’, IEEE Trans. Intell. Transp. Syst., 2013, 14, (3), pp. 1403–1415
18‘ After Mastering Singapore ‘s Streets, NuTonomy ‘s Robo-t-axis Are Poised to Take on New Cities’, http://spectrum.ieee.org/transportation/self-driving/after-mastering-singapores-streets-nutonomys-robotaxis-are-poised-to-take-on-new-cities, accessed 12 April 2017
19Liu B. Sun N. Zhang Q. et al.: ‘ A surrogate model assisted evolutionary algorithm for computationally expensive design optimization problems with discrete variables’. 2016 IEEE Congress on Evolutionary Computation(CEC), Vancouver, Canada, July 2016, pp. 1650–1657
20Fuller S. Gaschnig J. Gillogly J.J. et al.: ‘ Analysis of the alpha-beta pruning algorithm’. Research Showcase @ CMU, Pittsburgh, PA, USA, 1973.
21Baudet G.: ‘ An analysis of the full alpha-beta pruning algorithm’. In Proc. of the Tenth Annual ACM Symp. on Theory of Computing, STOC'78, New York, NY, USA, 1978, pp. 296–313
22Sato N. Ikeda K.: ‘ Three types of forward pruning techniques to apply the alpha beta algorithm to turn-based strategy games’. 2016 IEEE Conf. on Computational Intelligence and Games (CIG), Santorini, Greece, September 2016, pp. 1–8
23Bento L. Parafita R. Nunes U.: ‘ Intelligent traffic management at intersections supported by v2v and v2i communications’. 2012 15th Int. IEEE Conf. on Intelligent Transportation Systems, Anchorage, AK, USA, September 2012, pp. 1495–1502
24Marczuk K. Hong H. Azevedo C. et al.: ‘ Autonomous mobility on demand in simmobility: case study of the central business district in Singapore’. 2015 IEEE 7th Int. Conf. on Cybernetics and Intelligent Systems (CIS) and IEEE Conf. on Robotics, Automation and Mechatronics (RAM), Angkor Wat, Cambodia, July 2015, pp. 167–172
25 ISO/IEC/IEEE 29119: ‘Software and systems engineering – software testing – part 4:test techniques’, 2015.

Citing Literature

Volume12, Issue9

November 2018

Pages 1088-1095

Rule-based searching for collision test cases of autonomous vehicles simulation

Abstract

1 Introduction