Volume 17, Issue 4 p. 175-183
ORIGINAL RESEARCH
Open Access

Electro-optical spiking neural networks using an enhanced optical axon with pulse amplitude modulation and automatic gain controller

George-Iulian Uleru

George-Iulian Uleru

Faculty of Automatic Control and Computer Engineering, Gheorghe Asachi Technical University of Iasi, Iasi, Romania

Contribution: Conceptualization, Data curation, Formal analysis, ​Investigation, Resources, Software, Visualization, Writing - review & editing

Search for more papers by this author
Mircea Hulea

Corresponding Author

Mircea Hulea

Faculty of Automatic Control and Computer Engineering, Gheorghe Asachi Technical University of Iasi, Iasi, Romania

Correspondence

Mircea Hulea.

Email: [email protected]

Contribution: Conceptualization, ​Investigation, Methodology, Validation, Writing - original draft

Search for more papers by this author
Othman Isam Younus

Othman Isam Younus

Optical Communications Research Group, Department of Physics, Department, Maths and Electrical Engineering, Faculty of Engineering and Environment, Northumbria University, Newcastle upon Tyne, UK

Contribution: Visualization, Writing - review & editing

Search for more papers by this author
Zabih Ghassemlooy

Zabih Ghassemlooy

Optical Communications Research Group, Department of Physics, Department, Maths and Electrical Engineering, Faculty of Engineering and Environment, Northumbria University, Newcastle upon Tyne, UK

Contribution: ​Investigation, Supervision, Validation, Writing - review & editing

Search for more papers by this author
Sujan Rajbhandari

Sujan Rajbhandari

DSP Centre of Excellence, Bangor University, Bangor, UK

Contribution: Formal analysis, ​Investigation, Methodology, Supervision, Validation, Writing - review & editing

Search for more papers by this author
First published: 18 August 2023

Abstract

Visible light communication can be leveraged to establish a wireless link between neurons in spiking networks even when neural areas are in relative motions. In electro-optical spiking neural networks (SNN), parallel transmission is often achieved through wavelength division multiplexing (WDM). However, WDM can be prohibitive in certain applications due to the need for multiple narrow-band transmitters and receivers with optical bandpass filters. Instead of WDM, an alternative approach of using non-orthogonal multiple access is explored (NOMA) with a pulse amplitude modulation (PAM) scheme in optical axons to enable parallel neural paths in an SNN. To evaluate NOMA with PAM, the authors implement an electro-optical SNN that controls the force of two anthropomorphic fingers actuated by the shape memory alloy-based actuators. An optical reference channel is used to dynamically adjust the optical receiver's gain to improve the receiver's decoding performance. Experimental results demonstrate that the electro-optical SNN can maintain control over the fingers and hold an object under varying channel conditions. Hence, the proposed system offers robustness against dynamic optical channels induced by the relative motion of neurons.

1 INTRODUCTION

As in human neural cells, the electronic spiking neurons (SNs) have the advantage of parallel information processing and transmission. This enables SNs for the hardware implementation of artificial neural areas for real-time operations [1, 2]. Typically, humanoid robots' hands are in motion relative to the robotic body or head, which includes the main control unit (MCU). When MCU is based on spiking neural networks (SNNs) and the robotic hands that include neuromorphic sensors, the neurons in the sensors are at varying distances from the neural MCU. Visible light communication (VLC) can be utilised to enable communication in distributed SNNs with the moving SNs. Visible light communication leverages visible wavelengths to wirelessly transmit information between neurons. However, hand movements introduce challenges such as channel alignment and variations in signal strength, that is, varying channel length and angle of arrival induces intensity fluctuations in the received optical signal [3]. Recently, optical axons (OAs) were proposed to increase the tolerance of electro-optical neural networks (EONN) to dynamic optical signal variation at the receiver (Rx) [4]. Optical axons were used to connect the neuromorphic force sensors of two opposing fingers to the main SNN that regulates the fingers' force [3]. In addition to the optical connections of the neuromorphic sensors, VLC was successfully integrated into the neural drivers of the actuators. The system used a bidirectional VLC that connected effectors and receptors to the main SNN [5]. The usage of light-emitting-diode (LED)-based VLC in humanoid robots is justifiable due to the short link distance, typically less than 2 m, which aligns with the average height of humans. The neuromorphic sensors that implement a wide range of senses convert the physically measured values into spiking frequencies. For example, a robotic hand controlled by SNN incorporates three proprioceptive sensors per finger reaching 16 sensors for a realistic motor-driven hand [6, 7]. This implies that a fully connected hand employing VLC would require dozens of parallel channels to establish connections between the sensors and the main SNN.

Visible light communication/OWC offers several solutions to increase the number of parallel connections based on the spatial distribution of the optical signals. For example, (i) SLM and Fresnel lenses [8] or photoelectric multiplication in homodyne detectors [9] can be used to obtain the weighted sum of the laser signals generated by a significant number of neurons and (ii) using diffractive optical elements for spatial multiplexing of several channels [10]. Thus, multiplexing a large number of connections in all optical neural networks (ONN) is primarily based on the spatial distribution of optical channels, which is easier to implement in directed point-to-point links. However, all these techniques are proposed and evaluated for fixed point-to-point links, that is, the studies do not consider the relative motion between the transmitter (Tx) and the Rx, which affects the synaptic transmission due to signal fading.

The ONN benefits from high processing speed [11, 12], which is superior to the EONN [13], by using costly devices such as custom lasers [14] and silicon weight banks [15] which encodes weights by the light intensity. Although EONN is complex to implement due to the interconnection of electronic and optical components, the solution proposed in this work offers improved tolerance to signal fluctuation. This feature allows implementation of direct communication between neural areas that are in relative motion. Note that the static NNs are faster and costly while the NNs that tolerates relative motion are slower and cheaper, implying that speed represents the trade-off for obtaining motion tolerance at a lower price due to the use of electronics to transmit weights.

Wavelength division multiplexing (WDM) in VLC can be utilised to achieve a parallel data transmission in ONN [13] that tolerates the relative motion of neural areas [3]. However, the requirement of multiple narrow-band Txs and Rxs with narrow-band optical bandpass filters leads to higher costs and complexity, hindering widespread adoption. To overcome this challenge, low-cost solutions such as filterless WDM were explored for WDM-VLC systems [16-18]. In Ref. [16], multiple sets of independent weighted non-return-to-zero on-off keying data streams are transmitted at different wavelengths. At the Rx, the aggregated signal is decoded by leveraging the varying responsivity of the photodiode to different wavelengths. The combination of responsivity and LED characteristics is utilised to realise a WDM-MIMO (multiple-input-multiple-output) system for achieving parallel data transmission [17].

The commercially available LEDs have discrete wavelengths within a limited spectrum range that can be used in WDM-MIMO. Recent research shows that further to WDM and MIMO, more wireless channels can be multiplexed using the amplitude (power) domain multiplexing known as non-orthogonal multiple access (NOMA), which can improve the spectral efficiency and is also tolerant to the users' mobility [19, 20]. Additionally, the NOMA-based system is compatible with dimming control, allowing for energy efficiency improvements through transmit power adjustments [21].

In our previous work [5], we proposed NOMA with pulse amplitude modulation (PAM) for SNNs with multiple inputs of OAs. Each input has a distinct amplitude for the transmitted pulses, and a microcontroller (μC) at the Rx distinguishes the activated channel based on preset voltage thresholds (VTs). We demonstrated that the system was tolerant to minor fluctuations of the optical signal intensity caused by variations in the intervals set by the VTs. In this work, we have further improved the SNN's VLC connectivity by including an automatic gain controller (AGC) in the Rx, which employs a reference optical signal of a different wavelength to adjust the gain dynamically. At the Rx side, optical bandpass filters separate the communication and reference signals. The improved OAs are utilised to connect neuromorphic force sensors to the main SNN, which also controls the contraction of shape memory alloy (SMA)-based actuators. Similar to our previous work [5], the robotic hand is capable of grasping and holding an object when sensors and SMA actuators are connected to the main SNN using OAs. Additionally, we demonstrate that the improved bi-colour OAs with AGC exhibit increased tolerance to the relative motion between the Tx and the Rx.

The remainder of the paper is organised as follows: the system description and the experimental setup are presented in Sections 2 and 5, while Section 6 describes test scenarios and associated results. The conclusions of this work are given in the last section.

2 SYSTEM DESCRIPTIONS

As highlighted in the introduction, optical channels are multiplexed using NOMA with PAM scheme. The regenerated electrical signal at the Rx is given by the following [16]:
y ( t ) = x ( t ) h c ( t ) R ( λ ) + n ( t ) , $y(t)=\left(x(t)\otimes {h}_{c}(t)\right)\mathcal{R}(\lambda )+n(t),$ (1)
where x(t) is the signal from the LED, ⊗ is the convolution in the time-domain, R ( λ ) $\mathcal{R}(\lambda )$ is the photodiode (PD) responsivity at wavelength λ, hc(t) is the channel impulse response, and n(t) is the additive white Gaussian noise. In VLC systems, the dominant noise source is the background light-induced shot noise, which is given by the following [22]:
σ B G 2 = 2 q B e f R I b g , ${\sigma }_{\mathrm{B}\mathrm{G}}^{2}=2q{B}_{\mathrm{e}\mathrm{f}}\mathcal{R}{I}_{\mathrm{b}\mathrm{g}},$ (2)
where q is the electron charge, Bef is the electric filter bandwidth at the Rx, and Ibg is the photocurrent induced by the background radiation.

Since we operate the PD and the LED in the linear regions, the number of distinguishable channels achievable using NOMA-PAM depends on the ratio between the maximum voltage at the Rx and the noise level. When using PAM-VLC between the electro-optical SNs, the probability of concurrent axon activation may also be considered. However, the likelihood of simultaneous activation is reduced by the fact that the OAs transmit trigger pulses that are limited only by the hardware.

To evaluate the feasibility of the VLC link between SNs, we have developed a system consisting of an electronic SNN that controls the force exerted by a pair of opposing fingers in a humanoid robotic hand. Using VLC, SNN manages SMA drivers for finger movement and receives feedback signals from neuromorphic sensors that measure the force applied to the fingertips. To multiplex information, the VLC link incorporates OAs that use NOMA.

2.1 Optical axon with pulse amplitude modulation

Figure 1 depicts the general structure of OAs, where information is multiplexed by the SNN from several neuromorphic sensors and sent to more SMA drivers.

Details are in the caption following the image

The general structure of the optical axon with the reference and the main signals and automatic gain control.

This OA consists of a single LED for all inputs and a single optical receiver (ORx), which forms part of the OA. The OR includes a PD with a transimpedance amplifier and a dynamic gain amplifier connected to a μC's ADC. Each SOMA drives the LED with a predefined current, which determines the amplitude of the received pulses. Based on these signal levels, the μC activates the corresponding synapses (SYNs), which can be excitatory or inhibitory.

In the setup presented in Figure 1, the OAs receive input from the SOMAs included in the neuromorphic sensors and activate the synapses, which are inhibitory for sensors. The detailed schematics of the SOMA and the SYNs are provided in our previous work [4]. The current research is based on voltage-to-spike conversion (VSC), which is the main feature of the SOMA with the schematic presented in Figure 2a. The conversion is achieved using the capacitor CIN that integrates the incoming stimuli and determines the repetitive activation of the SOMA. When the stimuli are generated by excitatory and inhibitory neurons, the spiking rate of these neurons determines VC and consequently, the firing rate of the SOMA. In addition, the continuous voltage generated by the sensors is converted into a spiking rate by the SOMA in the absence of CIN.

Details are in the caption following the image

The schematics of: (a) SOMA, and (b) optical Rx with automatic gain.

The schematic of the amplification stage of the ORx is presented in Figure 2b. An amplifier with automatic gain control (AGC) is used to adjust the output of the transimpedance amplifier to a predefined value. To maintain the AGC output at the predefined level, the OA utilises an optical signal as a reference at a different wavelength that is generated at constant power. Via SPI, the gain microcontroller µCG programs the digital potentiometer (MCP4151) with an 8-bit Dn in order to set the gain of the AGC, which is given as the following:
V A M P = V A D C 1 + R A R B a n d R B = R P O T D n / 256 , ${V}_{\mathrm{A}\mathrm{M}\mathrm{P}}={V}_{\mathrm{A}\mathrm{D}\mathrm{C}}\left(1+\frac{{R}_{A}}{{R}_{B}}\right)\,\mathrm{a}\mathrm{n}\mathrm{d}\,{R}_{B}={R}_{\mathrm{P}\mathrm{O}\mathrm{T}}Dn/256,$ (3)
where VADC is the mean value of the reference signal over several readings, while RA and RB are the two components of the potentiometer with RPOT = 50 kΩ. Thus, Dn is given by:
D n = 256 V A D C × n A D C 256 . $Dn=256-{V}_{\mathrm{A}\mathrm{D}\mathrm{C}}\times \frac{{n}_{\mathrm{A}\mathrm{D}\mathrm{C}}}{256}.$ (4)

Note we have used nADC of 4095, which is the value returned by the ADC for the maximum VAMP = 3.3 V. In order to reduce complexity and improve the system's stability, the calculated gain is applied only to the AGC out, while the amplification of the reference remains constant.

When the OR moves across the reference beam, the illuminance level on the PD changes according to the inverse square law for path length. In addition, when the link alignment changes, it is important to consider the beam profile. Based on the experiments presented in this work, we considered the red and blue LED beam profiles to be similar.

2.2 Anthropomorphic hand

Figure 3 shows the robotic hand that includes two fingers with active and blocked junctions. A pair of compression load cells (CLCs) placed on the finger's apex can stop the finger's flexion by detecting external forces. Compression load cell and the SOMA form a neuromorphic sensor, which transmits optical pulses at different wavelengths through to the main SNNOAs. Also, SMA drivers receive optical pulses from SOMAs of the motor neurons, which control the force of the SMA actuators.

Details are in the caption following the image

The structure of the robotic hand with two opposing fingers.

Typically, in robotics, sensors and effectors are connected to the control unit via connections in opposite directions. This implies that, in practical applications, when VLC is used as a control unit interface, at least one pair of channels (main and reference) must be implemented for each direction. To simplify the validation of the proposed concept, optical connections are implemented on the same optical channel for both sensors and SMA drivers.

Figure 4 shows the general structure of the system based on electro-optical SNN for the control of the robotic hand. In the electro-optical SNN, OA utilising the blue spectrum (i.e. a blue LED) is employed to activate SMA drivers and to receive feedback signals from neuromorphic sensors. The SOMAs drive the blue LED with varying currents to implement NOMA with PAM. Based on several VT levels (i.e. 1.3, 1.8, 2.3 and 2.8 V), the μC (Infineon XMC4700) reads the pulses generated by the communication LED and activates the corresponding synapses. The gain of DGA is continuously adjusted based on the reference signal to implement OA's tolerance to optical signal fluctuations [4]. In this setup, the pulse amplitudes for each PAM level were set empirically (i.e. 1.4, 2.1, 2.5 and 3.1 V) between threshold levels [4]. Through the OA and μC, the SOMAEI, ET activate the corresponding synapses EST,I when the push button is pressed. The SMA drivers, driven by motor neurons MNT,I, determine the contraction of SMA actuators.

Details are in the caption following the image

Electro-optical spiking neural networks for the bidirectional control of two fingers; optical axons uses optical channels of different amplitudes.

SOMAST,SI is included in the neuromorphic sensors, which is activated via OA and μC the inhibitory synapses IST,I. Note that we associated PAM levels to each of the four SOMAs. The upper levels are used for actuators, while the lower levels are used by the sensors.

3 EXPERIMENTAL SETUP

Figure 5 illustrates the experimental setup for evaluating the ability of a robotic arm to hold an object through the use of OAs to connect the SNN with sensors and actuator drivers. The SNN receives input from sensors prior to transmitting its information to the robotic arm elements. In a typical application, an OA with a single optical channel and μC are used to implement a communication link. The OA is adopted here to connect sensors and drivers to the SNN. The main and reference LEDs are mounted on one finger which is static relative to the ORs while holding an object. The key parameters of the system, including the digital and analogue components, are given in Table 1.

Details are in the caption following the image

A photograph of the spiking neural networks with the main components of the optical axons with pulse amplitude modulation.

TABLE 1. System parameters.
Component Attributes
LED CREE XLamp Colour Wavelength, λ LED current, Iv
Red 640 nm 28 mA
Blue 465 nm Variable
PD Type λ for peak sensitivity Bandwidth, λ0.5
BPW20R 920 nm 550–1025 nm
BPW21R 565 nm 420–675 nm
Digital components Type Model Use
Micro-controller XMC4700, cortex M0
  • Pulse amplitude decoding

  • Synapse activation

  • Ref. voltage for ADC = 3.3V

Micro-controller XMC1100, ARM A0 Automatic gain control
Digital POT MCP4151 Dynamic gain
Optical axon The current for the blue LED per channel
5 mA 8 mA 10 mA 13 mA
Optical filter Colour Material
Red, blue Gel filter film
Environment Temp. Ambiental illuminance
25° C max 3 lux
SMA actuators Flexinol Diameter Cool time Max current Length
0.006” 2 s 410 mA 130 cm
  • Note: Bold values represent the attributes or system parameters of the components presented in the first column.

To analyse the system's operation, we observed the behaviour of the motor neurons and the output of CLCs both with and without OA, as well as in the scenario where SOMA and SYNs were directly connected through hardwiring. Based on mean values and the oscillation amplitude of the sensors' output, we determined the influence of OA on the regulatory performance of the SNN. Evaluation of the SNN characteristics were experimentally conducted when the hand was holding an object between the two fingers with or without optical connections. Thus, we conducted the experiment in two scenarios: (a) the OAs with VLC were interconnected between SOMAs and the corresponding SYNs, and (b) hardwired connections were used instead of OAs between the SOMAs and the corresponding SYNs. For the first phase, we performed several measurements for the line of sight (LOS) link when the optical Rx was moved along the x-axis between 10 and 14 cm (see Figure 3). Finally, the distance dx is set to 10 cm and we conducted multiple measurements while moving OR in the range of 0–5 cm along the Oy axis, which is perpendicular to the LOS propagation path.

For dx <10 cm, the optical Rx becomes saturated, while for dx >14 cm or dy >5 cm, the regulatory regime of the SNN is lost.

4 RESULTS AND DISCUSSIONS

Prior to conducting the experiments, we first evaluated the implementation of PAM on the μC which generates pulses on its PORT pins according to the level of the signal read using the ADC. In Figure 6, we exemplified the μC outputs (magenta, brown and green signals) when the SOMAs for both actuators and one sensor are active. The blue signal represents the output of the AGC connected to the ADC input, see Figure 2. To evaluate the performance of the electro-optical SNN in controlling SMA actuators, we captured the output signals of CLCs for 5s using a digital oscilloscope (Keysight Technologies model DSO-X 2004A). Figure 7a presents the signals for the input of the motor neuron when SOMAs are hardwired to the corresponding SYNs. We monitored the same connections for the LOS VLC link when the OA are used, and the signals are shown in Figure 7b. The magenta and blue signals represent the potential VOSC monitored on the SOMAs (see Figure 2a) of the motor neurons MN which drives SMA actuators used for the flexion of the thumb and index fingers. The brown and green signals represent the output of the CLCs for both fingers, respectively.

Details are in the caption following the image

Measured signal for pulse amplitude modulation. Synapses activated by pulse signals: (i) magenta and green signals used for SMA drivers, and (ii) a brown signal used by one of the two sensors.

Details are in the caption following the image

The captured signals during holding with: (a) hardwired axons, and (b) optical axons for line of sight and OPL of 10 cm.

Similarly, Figure 8a,b show the motor neurons' activity and the CLC output at the maximum dx = 14 cm and maximum deviation dy = 5 cm, when the hand can hold the object. Note that the neuron activations are represented by oscillating signals on both sides of the equilibrium voltage VEQU of the neuron, see Figure 2a.

Details are in the caption following the image

The captured signals during holding for: (a) a line of sight of 14 cm with an optical axon, and (b) an optical Rx moved to one side by dy of 5 cm and an OPL of 10 cm.

For the CLC output, the mean values are given in the following sections. The results show that the variation in received amplitude is higher with OA than with hardwired connections. This is due to varying link lengths and misalignment of the optical channel in OA. The variation can also be a consequence of the pulse superposition which affects the correct activation of SYNs. The voltage generated by the sensors is determined by the motor neuron's activity which is affected by the optical channel fading. However, the study of the correlation between the variations in the sensor's output and the neuron's firing rate is not the goal of this paper.

Despite the alteration in the mean value of the sensor output resulting from the implementation of OAs, deviation in the optical Rx within the specified range does not impede manual dexterity in grasping objects. The mean and maximum variation of the captured signals are calculated and are depicted in Figure 9. The mean of the sensors output changes with dx and dy implying that the force applied by each finger changes as the illuminance decreases. Also, the mean of the force of the thumb for dx = 16 cm reduces significantly because this finger is not actuated properly.

Details are in the caption following the image

(a) The mean, and (b) the maximum amplitude variation during steady state with optical axons when the OR was moved on dx and dy, respectively.

In addition, the received amplitude increases mainly when the Rx is moved to one side, showing that the regulatory performance of the SNN depends on illuminance. To further analyse the regulatory regime during steady state, the histograms of the CLC output for several positions of the OR relative to the PD panel are presented in Figure 10. Taking into account that illuminance depends on link distances, Figure 10 displays only the histograms for min and max dx, respectively.

Details are in the caption following the image

The histograms of sensor outputs for the index (left) and thumb (right) for line of sight (dy = 0 cm) and two values of dx for each measurement, the voltage variation range on the x-axis is split into 20 subintervals.

In Figure 11, detailed analysis results are provided for the deviation of the OR to one side, as this case exhibits changes in illuminance based on the characteristics of the LEDs. Note the significant difference between the histograms for both LOS and deviation, implying that changes in OPL and alignment impact the SNN regulatory performance. However, when dx and dy vary in the limits presented above, the robotic hand is able to hold the object despite alterations of the SNN behaviour.

Details are in the caption following the image

The side deviation for dx = 1 cm to dy = 5 cm with a resolution of 1 cm.

5 CONCLUSIONS

In this study, we investigated the possibility of using an enhanced PAM-based OA to connect the neuromorphic sensors and SMA drivers to the main neural control units of robotic hands. The OA provides the communication links between sensors and SNN on afferent paths as well as between SNN and actuator drivers on efferent paths. The OA is improved by using automatic gain control based on a reference optical signal and a single LED for information transmission. The proposed communication method based on VLC offers an elegant alternative to the traditional hardwired connections eliminating drawbacks associated with cable, such as inflexible connectivity, which is not suitable for systems with transceivers on mobile platforms. Moreover, multiplexing of the optical channels reduces the costs and complexity of WDM systems, while enabling relative mobility between the Tx and the Rx. The SNN performance to regulate the finger's force was evaluated by determining the mean and oscillation amplitudes of the sensors' outputs during the steady state. In addition, we present the histograms for these signals to get detailed information about the SNN behaviour with OA. The results demonstrated that the influence of the proposed OA on the regulatory performance of the SNN depends on the physical displacement of the OR relative to the transmitter. However, this influence did not affect the ability of the robotic arm to hold the object when OPL and deviation vary in certain limits. This implies that NOMA with PAM and automatic gain control is suitable for implementing VLC links between SNs in robotic applications where precise motion control is not necessary. For future work, we will explore possible improvements on OA, such as the detection and correction of pulses superposition. Additionally, we will investigate the use of LEDs with identical beam profiles for both the reference and main channels, as well as enhanced optical filters to enhance system performance further.

AUTHOR CONTRIBUTIONS

George-Iulian Uleru: Conceptualization; data curation; formal analysis; investigation; resources; software; visualization; writing—review and editing. Mircea Hulea: Conceptualization; investigation; methodology; validation; writing—original draft. Othman Isam Younus: Visualization; writing—review and editing. Zabih Ghassemlooy: Investigation; supervision; validation; writing—review and editing. Sujan Rajbhandari: Formal analysis; investigation; methodology; supervision; validation; writing—review and editing.

ACKNOWLEDGEMENTS

This work was supported by the EU COST Action CA19111 NEWFOCUS; GIU is supported by the doctoral school of TUIASI. SR work at the DSP centre of Excellence is part-funded by the European Regional Development Fund through Welsh Government and also by the North Wales Growth Deal through Ambition North Wales, Welsh Government and UK Government.

    CONFLICT OF INTEREST STATEMENT

    The authors declares no conflict of interest.

    DATA AVAILABILITY STATEMENT

    Data available on request from the authors.