An empirical study of local ‐ decision ‐ making ‐ based software customization in distributed development

Making a decision for the requirements of multi ‐ stakeholders is a key process, especially in distributed software development projects. Local decision ‐ making for requirements in distributed software development is really difficult to accomplish as well as communicating these requirements over organizational boundaries and conveying them to the offshore developers is a big task. This study presents an empirical evaluation for the effectiveness of local decision ‐ making on the customization process of the software in the distributed development against productivity and cost reduction. This empirical evaluation utilizes the Communicating Customization Requirements of Multi ‐ Clients in a Distributed Domain (CCRD) model. The empirical study estimates the productivity of CCRD in terms of the number of requirements for which decisions are made. In addition, the study estimates the reduction in the total cost of the customization process in terms of the salaries of the required local decision ‐ makers. Besides, this study finds the critical point at which the CCRD is still valid (i.e. the minimum number of requirements that violate the significance and worthy of CCRD). The study uses a real data set of 18 clients distributed through 16 cities and involved in one customization project requested about 3000 requirements collected in 1290 working hours. The results of this study showed that the local decision ‐ making improved the productivity of the customization process from 503 requirements in 200 min of simulation to 1,499 requirements. In addition, it reduced 41.5% of the cost. Besides, the results showed that


| INTRODUCTION
Distributed software development (DSD) and global software development (GSD) are software development models in which developers spread across different geographical sites locally in one country or globally through the worldwide collaboration on developing the same software [1][2][3][4]. Since the late 1980s, many software vendors have moved to DSD and GSD instead of collocated development [5]. Although DSD and GSD have many advantages such as the availability of human resources in decentralized regions at low cost [6], they suffer many challenges in collaboration [7], coordination [8], communication [9], management [10][11][12], customization process [13,14], task allocation [15], and the quality of the product [16] due to the distance between the members of the project team.
Recently, customized software has become an essential sector of the software industry, mainly in consequence of the boom in outsourcing and offshore software development process [14]. Consequently, most of the released software is subject to the customization process to meet the changes demanded by the customers. Local decision-making for changes in released software (customization process) in DSD is extremely difficult to accomplish as well as communicating these requirements over organizational and cultural boundaries, conveying them to multiple distributed developers, and negotiating with multiple customers. To overcome the customization process challenges associated with DSD and GSD, and new paradigms in software engineering practices are required. Some work has been done to discuss the challenges of DSD and GSD and the suggested solution [5,6,[17][18][19].
Lately, GSD has been getting significant attention in the software development field by way of overcoming some challenges in recruiting highly skilled practitioners. Although GSD aids in reducing the cost and improving the productivity of software development projects [19], new difficulties had arisen such as inadequate coordination and communication among the distributed members of the project team [6].
Some works have discussed these challenges and their different aspects and proposed solutions for them in the distributed domain. Hayat et al. [20] suggested a model for requirement changes management through the software development process. Khan et al. [21] reformed this model into a framework called requirement change management (RCM) for global development projects. RCM includes a central repository to ease sharing the knowledge among the members of the distributed team and decreasing the required communication through the distributed boundaries. Damian [22] explored the key challenges of global development through cultural and organizational boundaries.
On the other hand, several scholars have concentrated on requirement management in the distributed domain. Damian and Zowghi [23] explored the influence of multi-distributed clients on management and communication of requirements. They presented a model to overcome some issues which disturb requirement management in distributed-based development projects. This model explored the negative influences of distant communication and knowledge management, the variation of cultural and differences in time on requirements eliciting, analysing, negotiating, and specifying. The results showed that features such as misunderstanding of requirements, lack of awareness of the working local context, level of trust, and the ability for sharing artefacts restrict the collaboration of offshore stakeholders in requirements negotiation for the satisfaction of geographically distributed customers. In addition, researchers have concentrated on the delay in satisfying customers' demands as these demands consume more time in communications among distributed teams [24]. Gopal et al. [24] studied the effects of coordination and communication barriers on software quality and development speed.
The main objective of this work is to study the effectiveness of local decision-making on the customization process in the distributed development against productivity and cost reduction. Therefore, the study applies the Communicating Customization Requirements of Multi-Clients in a Distributed Domain (CCRD) model to estimate the productivity of the customization process in terms of the number of requirements for which decisions are made. Furthermore, the study estimates the reduction in the total cost of the customization process in terms of the salaries of the required local decision-makers. Besides, this study finds the critical point at which the CCRD is still valid (i.e. the minimum number of requirements that violate the significance and worthy of CCRD). To achieve these objectives, this work presents three experimental studies: the first study (ES#1) aims at estimating the productivity of the customization process resulting from applying local decision-making in terms of the number of handled requirements; the second study (ES#2) aims at estimating the cost reduction of the customization process resulting from applying local decisionmaking; the third study (ES#3) targets to assess the critical number of requirements at which the CCRD model is valid and gives a significant cost and productivity. The conducted experiments used a real data set contains 18 clients distributed through 16 cities and involved in one project who requested 3000 requirements in 1290 working hours.
The rest of this study is organized as follows. Section 2 presents some basic definitions and a brief description of the CCRD model. The details of the empirical studies are discussed in Section 3. The conclusion and future work are given in Section 4.

| BACKGROUND
This section gives some basic definitions and details of the CCRD model.

| Customization process
DSD and GSD are software development methodologies in which software developers are located in geographical sites of the world locally in one country or globally through many countries collaborate on developing the same software [1][2][3][4]. Actually, many of the customers of the same software demand their own configurations for that software. Consequently, most of the released software is subject to the customization process to meet the changes demanded by the customers [14]. Figure 1 presents the outlines of the customization process in the distributed domain. This process comprises three main stages. In stage #1, the software vendor releases the software and evolutionarily changes it by performing the required modifications. In stage #2, customers announce their demanded modifications that have to convey to the software vendor. In stage #3, which is considered the core of the customization process, the software vendor communicates with the customers, negotiates them, and makes a decision for the required modifications. Then, the software vendor assigns customization teams to install the software and address customization needs and modifications. Most of the customization challenges are concealed in stage #3 such as decision-making, communicating, and negotiating with the customers, which have to be F I G U R E 1 Distributed software customization process GHIDUK AND QAHTANI -175 treated to decrease the influences of these difficulties on the customization process speed.
The flow of the actions in the customization process is commenced by gathering clients' requirements. Clients' requirements can be classified into two categories. The first category comprises bugs, which are considered technical problems of the working software. This category of requirements requires a comprehensive investigation of the software to locate these bugs and debug them. The second category includes novel features, which are new functional or non-functional requirements to be added up to the working software. To achieve these requirements, many software development activities are required. Indeed, each requirement in these two categories needs somewhat of communication between clients and software vendors with the aim of understanding the requirement. In totally offshore projects, a team as a proxy for the principal vendor is available on the client's site to take care of the client's requirements [25].
After gathering clients' requirements, a decision has to be taken on these requirements. The decision is one of the following three decisions: Once a decision is made on the request, the offshore customization team at the vendor's site examines the understandability of the request and its scheduling in the team tasks. If the requested modification is an improper bug or unclear new feature, the request goes to suspension state and is returned back to the client for further investigation and decision.
The final step in the customization procedure is the development process that contains implementing the request, testing and verifying it, and delivery of the final version of the customized software to the client's location.
The precious observation is that the offshore development team at the vendor's site collaborates with multi-distributed clients simultaneously, which causes a lot of communication and coordination challenges. A great amount of communication takes place between the development team at the vendor's site and clients at their sites through the decision-making and negotiation step.

| THE CCRD MODEL
From the above discussion, multi-clients-based customization process suffers many difficulties in the communication among the clients and offshore developers to make a decision on the requested requirement [13]. In contrast to this, the locality has accomplished significant success in many software development and management processes in different development styles and contexts.
The main idea of the CCRD model given in Figure 2 is based on the local decision-making conception in multi-clients customization process to simplify the communication and negotiation process between the development team and clients and overcome challenges of conveying the requirements of different clients through multiple locations. The CCRD model follows the processes of software development life cycle from collecting the requested requirements to analysis, design, implementation, testing and delivering the product [26]. In addition, CCRD benefits from the advantages of the collocated team and the features of agile software development through applying local decision-making and negotiation at the location of clients [27]. Consequently, processes of the CCRD model are starting up with the requirements analysis process while customization requirements collection takes place at the location of the client and the development process including design, implementation, testing, and evolution takes place at the location of the offshore vendor. In CCRD, decision-maker team is relocated to the client's site to diminish the communication challenges, which is the main contribution of CCRD.
The CCRD model introduces a technique to overcome the difficulties of multi-clients-based software customization process across distributed boundaries and emphasizes the worth of the communication and coordination among the principal parties in the distributed domain, who are the clients, the onshore customization team (at the location of the client), and the offshore customization team (at the location of the vendor), to achieve the customization requirements of the clients [28].

| THE EMPIRICAL STUDY
This section discusses in detail the empirical studies performed to assess the effectiveness of local decision-making on the customization process of the software in the distributed development against the productivity and cost reduction and calculate the minimum number of requirements at which the CCRD model is valid. The design of these empirical studies follows the guidelines given by Perry et al. [29].

| Problem definition
According to the discussion given above, the multi-clientbased customization process suffers many difficulties in the communication among the clients and offshore developers to make a decision on the requested requirement [18,19].

| Problem review
A recent CCRD model is proposed to address the difficulties in communication through the customization process [13]. CCRD has been evaluated only in terms of speeding up the customization process without considering the loss in productivity or estimating the added cost [13]. The objective of this work is gauging the quality of the CCRD in terms of its productivity and the variation in cost and identifying the critical point at which this model is valid.

| The hypotheses and research questions
Employing the locality concepts on decision-making throughout the software evolution process speeds up the customization process. The CCRD model has the ability to accelerate the customization process and reduce the needed time. The hypotheses of this study are: H1: Employing the locality concepts on decision-making throughout the software evolution process increases the productivity of the customization process.

H2:
Employing the locality concepts on decision-making throughout the software evolution process decreases the cost of the customization process.
The objectives of this study are to measure the influence of local decision-making on the customization process by estimating the ability of CCRD to increase the productivity of the customization process, assessing the variation in cost and identifying the critical point of the validity of this model. To accomplish these aims, the succeeding research questions are formulated:

RQ1:
What is the influence of onshore decision-making on the productivity of the customization process in the distributed domain?

RQ2:
What is the influence of onshore decision-making on the cost of the customization process in the distributed domain?
RQ3: What is the critical point of the validity of the CCRD model?

| Measurements
To answer the research questions and inspect the hypotheses of the study, we will measure the productivity and cost of the CCRD technique. To accomplish this, three experiments were conducted: one for computing the productivity in terms of the number of achieved requirements and another for estimating the cost in terms of the salaries of the local decision-makers team. In addition, the critical point of the validity of the CCRD model is calculated.

| Implementation tool
The implementation tool or simulation environment is desirable to achieve the experiments. Where modelling and simulation have become a common approach in software engineering in research and industry [30] and it is used increasingly for various purposes such as process improvement, control and operational management of software engineering [31], therefore, in this research, simulation has been used for the evaluation process of the CCRD model. According to Setamanit et al. [32], testing hypotheses and controlling software engineering experiments in real-world cases have some difficulties. Thus, using simulation to applying research experiments is an alternative approach to save cost and time.
This study uses a simulation package to build and run simulation models. The package selected is called SIMUL8 [33]. This package was developed by SIMUL8 Corporation. It provides an integrated environment including a simulation modeler and a result analyzer. GHIDUK AND QAHTANI -177

| Objects of analysis
A case study in research adds valuable and understanding to the subject [34]. It is suitable for software engineering research as it studies and discusses different phenomena in their natural context [35]. Ambreen et al. [36] indicate the importance and increase of conducting case studies in empirical research followed by experiments. Also, they state that in the future the trend and needs would be conducting studies in real contexts and scenarios to get the experience of practitioners.
In this research, a case study has been conducted to reflect the real scenario of the customization process and collect historical data to drive the simulation model for those scenarios. The conducted case study is an actual project applied by a company that produces software products and then customizes that software based on clients' requirements. It has about 18 clients distributed through 16 cities involved in this project. Figure 3 shows the conceptual model of the customization scenario which is used by the case study. In that scenario, the decision-making allocates at the vendor's site. Therefore, it faces different challenges such as communication.

| Independent variables
The experiment manipulated the following independent variables: (i) Object of analysis: A case study of 18 clients distributed through 16 cities and involved in one project who requested 3000 requirements in 1290 working hours. Data used in the study were real and collected through a real case study in an existing company (anonymous). However, we are not permitted to mention the company name, publish or use its data without their permission. Appendix A shows a part of the data set. (ii) Local decision-making-based customization process.

| Dependent variables
Two dependent variables are computed: (i) The productivity: Average of the achieved requirements.
(ii) Cost reduction ratio: the cost of the salaries of the customization team in case of local decision-making and offshore decision-making.

| Scenario and results
This section discusses the setting and procedures, which are used in the experiment. The experiment stages are presented in Figure 4 starting from converting the real-world scenario of the real case study into a simulation model called baseline model using historical data collected from that case study and the contextual review for the process and documents of the customization project in the case study to define the setting of the simulation model. Data collected from the case study were about 3000 requirements collected in 1290 working hours. Those requirements have been collected from 18 clients distributed through 16 cities and involved in one customization project of a company that produces software products to those clients and then customize that software based on their requirements. In this stage, all settings and distribution of activities in the baseline simulation model reflect the opposite activities in the case study.
The second stage in this experiment built the simulation of the CCRD model shown in Figure 2 using the same setting of the baseline model with some changes related to the concept of the CCRD model, which is allocating decision-making at the client's site. Then, third stage inserts random data as arrival clients' requirements into the baseline model and the CCRD model to evaluate them. For both models, arrival data were generated using a Poisson distribution with λ = 0.365 as that fits with historical arrival data in the case study. The outputs from both models were compared using a statistical t test to find out how much the CCRD model has a significant statistical difference in the productivity of decision-making. The last stage in the experiment is to use experts' views on the results achieved and benefits of allocating decision-making at clients' sites. One of the main concerns in software engineering is process productivity. Thus, the main goal of this experiment is to examine the productivity of both the simulated baseline and CCRD models to evaluate the productivity of the CCRD model when decision-making is allocated at client's sites. Figure 5 illustrates that the number of requirements has completed decision-making in simulation time, which was 200 min. Both models have the same arrival requirements, which were randomly generated from 20 samples and then run 100 times for each model. The histogram in Figure 5 shows the high, low and average number of requirements in each run for both models. (a) Baseline model, (b) the CCRD model In terms of statistical analysis, the paired sample t test was conducted and the histogram of the results is shown in Table 1. In Table 1, the mean of requirements complete decision process in 200 min of simulation time was 503 requirements under the baseline model, while 1,499 requirements in the same setting and conditions under the CCRD model. The P value of that test was less than 0.05, which means that there is a significant difference between the productivity of the decision-making process in both models. Overall, the results of the paired sample t test in this experiment indicates that the outputs of the baseline simulation model and the CCRD simulation model show a significant difference in the productivity close to three times better in the CCRD model when P < 0.05 at 95% level of confidence. That means the null hypothesis is rejected and the alternative hypothesis is accepted, and that there is a significant difference in the number of decisions completed between the baseline simulation model and the CCRD simulation model. Table 2 shows the number of decision-makers and their cost in the baseline model and CCRD model. This result has been collected from the case study, which was conducted on 18 distributed clients to communicate with the offshore development team to customize a software product. In the first column, the baseline model has 18 representatives for the company at the client's site. Those representatives collect requirements from clients and send them to offshore decisionmakers to confirm that requirement, reject it, or change it. In general, the representative at clients' sites assumed to be beginner software engineers. These engineers do collecting analysis requirements and deploy new changes comes from the offshore development team. On the other hand, decisionmakers at making decisions should be senior software engineers to assess the changes and aware of clients' contracts. Therefore, the baseline model has 18 representatives and 17 senior decision-makers at the offshore development team (based on the historical data collected from the case study). The number of the role of representatives has been eliminated and senior decision-makers moved into clients' sites in the CCRD model to move decision-making to client's sites. Thus, the number of human resources in the CCRD model has reduced from 35 to only 18. As a result of this change, the cost has been reduced from £1514490 to £886032 according to the salaries of software engineers and senior software engineers in the UK (i.e. local decision-making reduced 41.5% of the total cost). These salaries were taken from Glassdoor [37] website  (last visit in November 2018), which is a website that collects and analyses real salaries from employees of different large companies wide world.

| Results of ES #3
Robinson [38] listed three main areas for sensitivity analysis, one of which was assessing the robustness of the solution model. A sensitivity analysis has been conducted on the CCRD model to find out the critical point, which aims to define the change point in the model. Figure 6 shows the relation between the number of arrival requirements from 5 clients and decisions made in only 200 simulation-time units (minutes).
Point C on the chart shows the critical point of the CCRD model. At this point, the productivity of the model has been changed from negative to positive, which means that the CCRD model would be valid for clients and creates more than about 112 requirements for each client (560 for 5 clients) in 200 simulation-time units.
The main approach to perform sensitivity analysis and find out the critical point is to vary different the model inputs, run the simulation, and record the change in the responses many times until getting the goals [38]. In this case, the analysis starting from changing the arrival requirements number by changing the distribution parameters in a sequence of runs starting from Poisson 5 till Poisson 0.365 (the actual distribution) is shown in Table 3.

| CONCLUSION AND FUTURE WORK
This study aimed to study the influences of local decisionmaking in distributed development. The study presented an empirical evaluation for the effectiveness of local decisionmaking on the customization process of the software in the distributed development against productivity and cost reduction. The empirical study estimated the productivity of using local decision-making in the customization process which improved the process from 503 requirements in 200 min of simulation to 1499 requirements. In addition, the local decision-making reduced 41.5% of the total cost of the customization process in terms of the salaries of the required decision-makers. Besides, this study showed that the critical point at which the local decision-making is still valid is 112 requirements in 200 min. The future work will concentrate on exploring the influences of the local developer on the distributed development process. In addition, it will study the testing activity in distributed development.

A P P E N D I X A
Data used in the study were real and collected through a real-case study in an existing company (anonymous). However, we are not permitted to mention the company name, publish or use its data without their permission. This real data set consists of 18 clients distributed through 16 cities and involved in one customization project requested about 3000 requirements collected in 1290 working hours. The following is a part of the data set.