A modular and scalable control and data acquisition system for power hardware in the loop (PHIL) amplifiers

This paper presents a newly developed modular system for control of power electronic amplifiers in high-power, high-frequency power hardware in the loop (PHIL) experiments. The proposed design comprises computational nodes connected via multi-gigabit fibre optic links. Nodes are modular and reconfigurable, allowing interfacing with a range of sensors and actuators; typically voltage and current sensors, and switching converters. The system has been designed for scalability to permit arbitrarily high power ratings and fidelity to be achieved through paralleling and interleaving of multiple power converter modules, targeting an ultimate sample and control rate of 1 MHz. Experimental validation is presented using a two-node configuration to facilitate a power electronic experiment operating at 120 V rms and 10 A rms , with a fundamental frequency of 50 Hz and frequency content up to the 11 th harmonic.


Introduction
Power hardware in the loop (PHIL) is an attractive methodology for the characterisation or validation of the performance of power hardware, as it allows real, physical hardware to be tested against a simulated system (e.g. rural power network, ship propulsion load, etc.), for which it might otherwise be impractical or impossible to construct a physical analogue [1]. The design of interfacing hardware to support PHIL experiments becomes challenging as the power requirements of the experiment (e.g. testing of multimegawatt drives), or the required bandwidth of the system (e.g. short-circuit testing, non-linear loads) increase. To address these competing requirements, modular amplifiers composed of multiple, parallel, sub-rated converters can be constructed to achieve multiplication of both bandwidth and power [2]. In theory, arbitrarily high performance can be achieved via this methodology.
As the number of power modules is increased, however, the problem of how to sample the state and perform control of multiple nodes must be addressed. At some point, the performance of the control and data acquisition system becomes the limiting factor in the PHIL context [3]. This research aims to address this problem through the design of a modular, scalable, high-performance control and data acquisition system to support PHIL amplifiers composed of an arbitrarily high number of power electronic converter modules.

Proposed control and data acquisition system
A control and data acquisition system has been designed specifically to support the requirements of high-performance PHIL amplifiers. The system comprises multiple computational nodes connected via multi-gigabit fibre optic links (see Fig. 1). Each node can be configured to perform data acquisition and/or control functions, supporting a wide variety of possible control structures (centralised, distributed or hybrid). Each node supports up to four full-duplex, multigigabit transceivers, permitting both ring and star communication topologies. It is intended that a real-time simulator (RTS) is interfaced to the system via a fibre optic link; sampled values can be provided to the RTS via this link, and control references communicated to the nodes. The computational nodes provide an on-board digital interface to up to sixteen peripheral modules. Peripheral modules can be independently designed to support a variety of applicationspecific inputs (typically analogue voltages and currents) and outputs (typically converter switching and status signals). Figure 1 shows an example configuration for the system. A number of power electronic converters are paralleled and electrically coupled to achieve a higher effective power rating and bandwidth. Two computational nodes are associated with each converter module: one to sample the output state of the converter, and a second to provide control and modulation of the converter. Local measurements are provided via one fibre optic link to the controller node which may perform local, high-speed control (e.g. current control). A subset of these measurements may be provided to neighbouring controllers to support additional control functions (e.g. balancing), or communicated to the RTS via the ring network for the purposes of system-level control (i.e. the PHIL experiment).
In this example a separate ring network has been constructed between the measurement nodes for the purpose of logging all sampled values for offline analysis of the PHIL experiment. This structure reduces the bandwidth and line rate requirements of network segment responsible for the timecritical control, while still providing visibility of the entire system state for offline analysis.

Base board and FPGA module
Each computational node comprises an FPGA system-on-chip (SoC) module and base board (see Fig. 2). The FPGA module features a Xilinx Zynq-7000 series SoC (including programmable logic and dual-core ARM processor), four SFP+ ports supporting transceiver rates up to 6.6 Gbps, a HPC FMC connector for interfacing to the base board, and an array of ancillary peripherals (memory, USB, micro SD card, gigabit Ethernet etc.).
The base board provides an interface for up to sixteen peripheral modules, each with four independent digital I/O. Two shared digital I/O (for daisy-chaining or synchronisation) are also available to each peripheral module, along with a common I 2 C bus interface. Each module connector features a 7-bit interface for peripheral detection and identification, to support automatic reconfiguration. Peripheral modules can be designed to occupy multiple peripheral slots, providing access to additional I/O. An additional interface is also available supporting 44 digital I/O to enable high-bandwidth parallel interfacing (e.g. to a microcontroller). This interface is via a HSEC-180 connector, compatible with the Texas Instruments ControlCARD development boards. The base board is in a 6U Eurocard form factor (233 x 266 mm) to permit racking multiple boards into a subrack enclosure.

Analog to digital conversion (ADC) modules
A single-channel analogue to digital conversion (ADC) module has been designed to permit sampling of analogue signals by the computational nodes (see Fig. 3). The ADC module consumes a single peripheral socket on the base board. The module features both signal and power isolation up to 2.5 kV, permitting sampling of voltages and currents floating with respect to the base board. The isolation barrier additionally provides protection for the base board in case of a fault. The ADC module utilises a fully differential, 12-bit ADC (AD7450A) and analogue conditioning front end (based on an LTC6362 amplifier), capable of sampling rates up to 1 MSPS, and analogue bandwidth up to 1 MHz. Provision has been made in the analogue front end for the implementation of a second order multiple feedback lowpass filter, which may be used for anti-aliasing if required. Conversions are initiated on the falling edge of one of the module control signals, permitting complete flexibility as to the sampling instant of each individual ADC module (i.e. software defined).
Three assembly variants have been designed to cater for different applications: high voltage inputs, low voltage inputs and current inputs. The high voltage input variant is designed to accept direct voltage inputs up to ±1000 V via a Eurostyle PCB terminal block header, for measurement of power electronic system voltages. The low voltage variant is designed to accept input voltages up to ±10 V via a MMCX coaxial connector for interfacing to sensors or systems with a low-voltage analogue interface.
The current variant is designed to interface to current transducers with a current output (e.g. current transformers or closed-loop hall-effect sensors). The input to this variant is via a 50 Ω terminated MMCX coaxial connector and is designed for ±400 mA inputs.

Gate drive interface (GDI) module
The gate drive interface (GDI) module is designed to interface to a Semikron SEMIKUBE GB 11 half-bridge driver board (see Fig. 4). Each GDI module provides two 24 V PWM signals, an interface to a 24 V open drain fault bus (one input for reading the bus status and one output for driving the bus), and analogue to digital conversion of three ±10 V signals (typically bus voltage, temperature and phase current). While the GDI module has been designed for a half bridge driver board, through the use of an appropriate cable loom, three GDI modules can be trivially combined to interface to a three phase driver board, such as the Semikron GD 11 driver board. In this configuration a contiguous subset of pins on each GDI output connector is used to interface to each additional output phase (PWM signals and half-bridge output current analogue feedback signal).
The GDI module features a two-channel, 1 MSPS ADC (AD7265). The current and voltage feedback signals can be simultaneously sampled and converted, however, the temperature feedback signal is multiplexed with the voltage signal. Consequently a 500 kSPS conversion rate is achievable if all channels are sampled at a uniform rate. This sample rate is well in excess of the typical bandwidths of the feedback signals. Higher sample rates are possible for the voltage and current channels (up to the 1 MSPS limit), if the temperature channel is sampled less frequently.
The GDI module is fully isolated from the base board via digital isolators. This isolation is predominantly for protection of the base board, as the driver boards are typically also isolated. Due to the power requirements of the gate drive boards, power must be supplied externally to the isolated side of the GDI module via a 24 V header.

Hardware prototype
To demonstrate and experimentally validate the operation of the proposed control and data acquisition system, a two-node prototype system has been constructed, the schematic diagram for which is provided in Figure 5. This prototype system is comparable in function to one of the control-measure functional modules depicted in the example system in Figure 1.
The experimental prototype implements a single phase inverter composed of three, interleaved half-bridge modules paralleled via independent inductors L1-L3. The inverter drives an adjustable resistive load (R1) via an LCL filter composed of the paralleling inductors (L1-L3), a split filter capacitor (C1 || C2) and output inductor (L4). The values of the filter components are L1 = L2 = L3 = 630 µH, C1 = C2 = 25 µF and L4 = 1.2 mH, which results in an effective cut-off frequency for the filter of 900 Hz. The load current is returned to the centre tap of the DC bus. C3 and C4 are each 7050 µF and are composed of multiple electrolytic capacitors internal to the inverter module. The inverter used in the prototype is a Semikron SEMIKUBE three phase inverter module with GD 11 drive board.
The prototype is fully instrumented and controlled using the developed control and data acquisition system (see Fig. 6). One base board is used as a measurement node to sample the system currents and voltages. The measurement node is configured with eight ADC modules: four of which are the high-voltage variant and four of which are the current sensing variant. Currents are sensed using LEM LF210-S closed-loop hall-effect sensors, each with four turns through the transducer. The current and voltage measurement points are shown in Figure 5.  A second base board is utilised for the control node, equipped with three GDI modules. A ribbon cable loom is used to connect the three GDI modules to the SEMIKUBE GD 11 driver board, with the first GDI module fully functional, and the two additional GDI modules contributing the PWM signals for the second and third inverter legs, and used to sample the corresponding current feedback signals.
The measurement and control nodes are connected using Avago AFBR-57J9AMZ fibre optic transceivers via 1 m of OM3 multimode fibre optic cable. An RTDS is used as the real time simulator and connected to the control node using Finisar FTLF8519P3BNL fibre optic transceivers via 5 m of OM3 multimode fibre optic cable. A photo of the assembled system is shown in Figure 6.

Communications and control
The prototype system is designed to operate with the RTDS as a central controller; all measurement samples are passed from the measurement node, via the control node, to the RTDS over fibre optic links (depicted as dashed red lines in Figure 5, and shown physically as purple and cyan cables in Figure 6). The RTDS additionally synthesises PWM duty cycle reference signals, which are communicated to the control node. The system is configured to use the Xilinx Aurora 8b10b serial protocol [4], operating at 6.25 Gbps line rate for serial communications between the control and measurement nodes, and the same protocol operating at 2.0 Gbps to interface to the RTDS. The RTDS is configured in large time-step mode, with a time-step interval of 6 µs.
All ADC modules on the measurement node are simultaneously sampled at 1 MHz, following receipt of a packet from the control node, which triggers each sample. An eight-sample moving average filter is also applied to each measurement. Following conversion and filtering, digital samples of both unfiltered and filtered measurements are communicated to the control node over the fibre optic link. All samples are converted to signed integers and scaled to 16-bit full-scale prior to transmission. A 16-bit status word and FPGA die temperature measurement are also communicated to the control node from the measurement node. In total eighteen 16-bit words are transmitted from the measurement node at 1 µs intervals.
All samples received from the measurement node are retransmitted to the RTDS via the 2 Gbps Aurora Link fibre optic interface. As the RTDS operates on 32-bit signed integers, sign extension is performed on the 16-bit samples received from the measurement node prior to retransmission. As for the measurement node, a 16-bit status word and FPGA die temperature measurement from the control node are also communicated to the RTDS.
The control node also implements three, phase accumulator carrier pulse width modulators (PACPWM) with arbitrary resample rate zero order hold (ZOH) [5]. The modulators are configured for 10 kHz asymmetric PWM with 16-bit amplitude precision and 32-bit phase accumulator precision. The carriers of each modulator are equally phase-offset to produce interleaved switching signals for the three half-bridge inverter legs. PWM duty cycles are transmitted from the RTDS as 32-bit signed integers (with ranges constrained from 0-65535). The PWM compare value is updated immediately upon receipt of a new sample from the RTDS, with no requirement for synchronisation of the PWM and simulation time intervals. An edge lockout circuit is implemented to ensure a maximum of one switching edge per PWM halfperiod. A control word is also transmitted from the RTDS which enables/disables PWM output. In total, twenty-nine 32bit words are transmitted from the control node to the RTDS, and four 32-bit words are transmitted from the RTDS to the control node over the Aurora Link.
The control and communication functionality of the system is implemented entirely in the programmable logic portion of the FPGA SoC, with the ARM cores unused in this implementation.
The RTDS is configured to synthesise reference signals for the PACPWM modulators composed of a 50 Hz fundamental (of variable modulation depth) plus 10% 11 th harmonic content (550 Hz). As the prototype is running in open loop for this demonstration, the DC offset of the individual phase references can also be manually adjusted to minimise circulating currents. All sampled values transmitted to the RTDS are captured, scaled and plotted in RSCAD.

Results and discussion
To evaluate the performance of the control and measurement system, the latency of propagation of sampled values throughout the system has been measured. These data are presented in Table 1.
The delay between the sampling instant and availability of the sampled value in the measurement node was measured at 1.02 µs, which is expected, and largely irreducible due to the conversion time of the ADCs utilised in the system. The propagation delay between measurement and control nodes, however, is much higher than the theoretical latency of the 6.25 Gbps Aurora link, which can be attributed to nonoptimal timing for the triggering of transmission of samples. It is expected that this delay could be reduced in future implementations by at least 1 µs (i.e. one sampling interval).  Table 1: Measured latency of signals propagating though prototype control and data acquisition system.
As it is not possible to directly observe the timing of signals internal to the RTDS, the latency of signals propagating from the measurement node to the control node via the RTDS was instead measured. The large variability (~6 µs) in the measured latency can be attributed to the time-step of the RTDS, as the control and data acquisition system is not synchronised to the RTDS time-steps. This variability could be eliminated through synchronisation of the two systems. Discounting the variability, one time-step delay for propagation of the signal through the RTDS model, and the propagation delay from measurement node to control node results is a net round-trip latency for the 2 Gbps Aurora link of approximately 2.26 µs. This is again much higher than the theoretical latency achievable, and so may indicate there are further opportunities to optimise timing for reduced latency.
At a system level the prototype performed to expectations, with the single phase inverter successfully operated at 120 Vrms and 10 Arms (1.2 kW). Output waveforms for the prototype operating at these conditions, as captured in RSCAD via the RTDS, are presented in Figure 7. The 10% of 11 th harmonic content which was superimposed on the 50 Hz fundamentally is faithfully reproduced at the load. Some high-frequency excursions are observable in the voltage traces, which may indicate additional analogue low-pass filtering on the ADC modules is warranted (no analogue filtering is implemented in this work).
Good current balance was achieved across the three inverter leg outputs without any DC offset required in the control signals. Figure 8 demonstrates that the performance of the system is acceptable for observation and faithful reproduction of switching frequency content in the inverter output currents. Figure 7: Experimental waveforms captured in RSCAD via the prototype control and data acquisition system, with the inverter operating at 120 Vrms and 10 Arms with 10% 11 th harmonic injection. Shown are the inverter leg output currents (top), filter capacitor and load voltages (middle) and load current (bottom).