Open Access
Add to BibSonomyAbstract
Maximizing the delivered capacity of optical transponders by mining the SNR margin to a near-zero level is of critical importance for the economic viability of future optical networks. We examine this trend from the equipment design point of view and apply artificial intelligence to the problem of capacity maximization. The bottom-up approach is focused on providing the key metrics that are built on the fundamentals of network performance, whereas the top-down approach uses machine learning and policy-driven actions that have the promise to achieve an unprecedented level of control. These methods show promise in delivering a practical near-zero margin network deployment.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. INTRODUCTION
Today’s developments in optical transport networks are a response to the continually increasing demand for network capacity. Recent trends that contribute to this demand include video streaming, 5G mobile deployment, data center interconnect, the Internet of Things, and other applications [1].
However, the revenue that is generated from a fixed number of bits being transported has eroded as quickly as the demand has grown, requiring network providers to find innovative ways to drive down the cost per transported bit. These include methods of mining capacity from a linear channel by means of digital signal processing (DSP) compensation of linear effects such as chromatic dispersion and optical filtering, noise mitigation using high gain forward error correction (FEC), linear modulation encoding, etc. [2]. Other advances address the nonlinear propagation environment of optical transport [3–5].
In a second transport-cost-reduction category are methods of fully utilizing available network capacity previously stranded by (a) fixed modem capacity and/or (b) margin guard bands based on generic, feed-forward estimates of required margin. Solutions in this category rely on flexible capacity modems, in concert with line equipment, being responsive to physical layer and network controls. Such controls are informed by modem and line monitors to meet and foresee network traffic requirements. Here there is the prospect of applying emergent capabilities of artificial intelligence and machine learning to transform in situ measurements merged with traffic requirements to provide forecasts, advice, or actionable decisions to network, line, or modem control. Thus, a goal of zero margin is approached within the practical capability of the implementation and delivering a required quality of service (QoS).
Figure 1 shows a representative analysis of the line card cost of network capacity over time and generations of transponder technologies. The growth for this example network was taken as 1.26 times per year in capacity [1]. This analysis shows that the cost per transported bit per second has been dropping while the total demand for transport increases. Multiplying these two quantities together yields the net cost for the total capacity (modems only). Interestingly, the total cost remains relatively flat. In other words, modem technology innovations have reduced the cost per bit of transport at a rate that has been keeping pace with this level of demand.
Fig. 1. Representative cost of optical transport capacity over time and transponder generations based on the historical average sales price (ASP) of DWDM line card data from Ovum showing (a) the decreasing cost per transported bit/s, (b) the growth in network capacity, (c) the net cost of transporting total capacity, and (d) the number of modems needed to satisfy capacity demand (all on an arbitrary linear scale).
To continue this trend, the network goal must be to deliver capacity as efficiently as possible. The traffic demand paths in an optical network deliver the capacity-carrying signals to their destination with a certain amount of noise as quantified by the signal-to-noise ratio (SNR). Today’s variable capacity modems are designed to convert as much of this SNR into capacity as possible given network constraints on QoS [6]. The QoS is a combination of the underlying quality of transmission (QoT) [7] of the signals and any additional layers of resilience built into the network through redundancy or excess throughput. Clearly, excess throughput is undesirable since it not only includes additional cost of transport but additional switching and routing equipment. Understanding the QoT is key to minimizing the amount of excess throughput required for the desired QoS, especially when operating in a near-zero margin regime.
In the following sections we address the issues of modernizing the approach to network design and deployment to achieve near-zero margin networks. We consider bottom-up and top-down analyses in this study. In the bottom-up analysis, we consider channel measures for telemetry and control as they pertain to network equipment design. In the top-down analysis, we apply machine learning and artificial intelligence that process these measures for control purposes.
Section 2 addresses the line system, focusing on minimizing noise contributions to the signal. Section 3 discusses modem design considerations for low margin deployments. In Section 4 is an outline of a margin prediction framework. Section 5 examines the control and monitoring of networks in the context of software-defined networking (SDN) and modern software tools leveraging the emergent capabilities of artificial intelligence and machine learning.
2. LINE SYSTEM
Line system design and deployments are evolving to adapt to variable capacity modems that can trade off reach and capacity to optimize for the delivered SNR. This contrasts with previous generations of line systems designed for fixed rate transponders, which had a corresponding fixed required SNR. At that time, the design of the line system was a cost minimization problem, where there was no benefit in exceeding the minimum required SNR of the corresponding fixed rate modem. Today, line system deployments attempt to maximize delivered SNR and therefore capacity. This is sometimes referred to as capacity-optimized or SNR-optimized design.
Different applications drive different criteria for line optimization. At two extremes of this space are terrestrial and submarine networks. In terrestrial designs, there is usually little or no power constraint at amplifier sites. There may also be many fibers available along any given path. In addition, the cost of the line is much lower than that of submarine cables. Submarine cables also have constraints in terms of electrical power delivered to each amplifier.
The following subsections will dive into some of these differences and the designs that are being chosen.
A. Terrestrial Line System SNR-Centric Design
Line systems deployed today value not only improving the delivered SNR but also reconfigurability. One of the best features of photonics is signal independence, i.e., the ability to support multiple generations of transponders independent of format, bit rate, symbol rate, etc. Many networks that were designed for 10 and 40 Gbps are now carrying 200 Gbps channels, and many that were deployed with flexible grid capability are now carrying 400 Gbps signals.
Typical deployments today include many new or updated technologies including hybrid Raman [a combination of distributed Raman amplification and erbium-doped fiber amplification (EDFA)]; switchable gain EDFAs; colorless, directionless, contentionless (CDC) add/drop [8]; and flexible grid [9] reconfigurable optical add/drop multiplexer (ROADM) technology. In addition to total-power monitors, spectral monitoring is provided through optical channel monitor (OCM) devices that act like optical spectrum analyzers. Optical time-domain reflectometers (OTDRs) are integrated into systems. OTDRs characterize the transmission fiber for point losses, loss per kilometer (km), reflections, core size, etc. Although initially introduced to assess risk of damage to the fiber plant by high-power Raman pumps prior to turn on, they have since become a requirement for most amplifier sites, even when Raman pumps are not in use. OTDRs are now used in-service to track drift in the fiber parameters over time, locate cuts, and compare before and after repair losses.
Line system instrumentation has great value in removing the uncertainty around the actual parameters of deployment. In the past this uncertainty had to be built into the planning process and remained as excess margin for the life of the network. Today this margin is reclaimed as available SNR and therefore capacity.
B. Submarine Line System Capacity-Centric Design
The constraints of submarine cable design are very different than those in terrestrial networks. For example, the amplifier output power is limited by the electrical power supplied over the length of the cable. This provides an opportunity to maximize capacity based on the power efficiency of the overall cable, which tends to drive the use of more fiber pairs at lower SNR. It is a natural consequence of the Shannon capacity equation wherein the capacity is proportional to the log of the SNR. Increasing the number of fibers in a cable reduces the power available to each amplifier, thereby reducing the SNR but increasing the amount of spectrum available through space division multiplexing (SDM). For fixed electrical power in some range of fiber pair count and amplifier total output power, the capacity reduction from a reduced SNR is smaller than the additional capacity obtained from lighting another fiber pair. Thus, there exists an optimum combination of optical power per amplifier and lit fiber pair count that maximizes the total delivered capacity [10].
The cost of cable in submarine deployments is a much larger fraction of the total system cost than in terrestrial deployments. It has been shown that the terminal equipment represented only 20% of the total submarine system cost [11]. Consequently, for cost reduction in a submarine deployment, efficient use of the available spectrum has more emphasis than does maximum throughput per transponder. This opens the possibility of tuning the spectral occupancy of each signal by adjusting the baud of the modem [12].
C. Line System Control and Optimization
The optimization of the line system requires adjustment of the various actuators, such as amplifier gains and tilts, and per channel attenuation using wavelength selective switch (WSS) pixels. The feedback from the system used in determining the actuator targets comes from the instrumentation including total-power taps and OCM spectral power data. Determining the appropriate targets requires an understanding of the SNR delivered in the current state and the change in SNR resulting from changes in the control. Previous generations of line system control were concerned with the optimization of the optical signal-to-noise ratio (OSNR), which is calculated from the linear noise contribution of amplified spontaneous emission (ASE) to the total SNR of the channel. The nonlinear interference (NLI) noise portion of the SNR had to be calculated separately owing to the complexity of the calculations in a low chromatic dispersion (CD) environment, which was typical of the line systems of the time.
With the advent of modem-based CD compensation, uncompensated line systems have become the norm. It is known to be a fair approximation, particularly for uncompensated links, to treat NLI as a noise having Gaussian statistics. This is often referred to as the Gaussian noise (GN) model [13]. This model can be used in the feedback loop of line system control to optimize the line contribution to the total SNR rather than just the linear portion or OSNR. It has also been shown that this approach can be used locally to minimize the incremental noise added by a single fiber or to a collection of fiber spans between ROADM nodes [14]. Optimization in real systems requires that the model works in the presence of all system effects, including wavelength-dependent loss (WDL), stimulated Raman scattering (SRS) [15,16], and the effects of coherent addition of NLI across incremental sections [17]. Algorithms for capacity maximization have been proposed that include these effects [18–20].
In the past, a major contributor to power fluctuations in a line system was the interaction of the signals with polarization-dependent loss (PDL) and polarization-dependent gain (PDG) [21]. Transponders prior to the introduction of dual-polarization coherent transmission transmitted single-polarization signals. Even when there are many single-polarization signals multiplexed together, the net degree of polarization (DOP) evolves over time and does not have a zero mean. Any non-zero net DOP interacts with the fixed PDL elements of the line and drives additional power fluctuation by inducing polarization-dependent gain (PDG) in the amplifiers [22]. The net DOP also interacts with the PDL of the total-power taps used for EDFA gain or power control, which tends to drive noise into the control loops themselves.
In contrast, starting with a set of zero DOP sources such as dual-polarization coherent transmitters, PDL and PDG interactions are far less pronounced. Figure 2 shows the difference in the stability of transmitted power for a single channel between single- and dual-polarization sources. In a system carrying dual-polarization signals, the penalty due to PDL is primarily limited to an SNR degradation of the channels themselves. The overall result is that modern transmission systems carrying exclusively dual-polarization coherent signals tend to be very stable, which is valuable when attempting to operate them at near-zero margin.
Fig. 2. Comparison of an individual signal transmission power standard deviation of a single-pol and a dual-pol source.
D. Monitoring the Line
Line system monitoring is an important input to understand the performance of the network. The modem feedback only monitors the net performance of the paths, whereas the line system can focus in on each ROADM section or individual fibers. Commonly reported metrics include EDFA gain, Raman gain, span loss, optical return loss (ORL), total input and output power, per channel power, or power spectral density (PSD), OSNR, etc.
The optimization methods discussed above require some estimate of the actual SNR of the paths in question. This can be a valuable monitor point for the line system, which should correlate strongly with margin. Comparing these results to modem estimates of margin can provide great value. When correlation is strong, the system can proceed to harvest SNR margin for additional capacity with confidence. When there is disagreement, it can point to issues that can be addressed through fault analysis. The incremental SNR can also be used to find “hot spots” in the network. These may be the fiber spans or sections that are contributing the most noise to the system. This allows routing engines to weight these links differently in path calculations, or the network operator may choose to perform maintenance on these hot spots to improve performance and therefore capacity.
3. MODEM
Achieving the maximum capacity requires minimizing the distance to the Shannon capacity. Achieving the minimum gap to Shannon in a deployed system involves real-time metrics that can be used by a controller to minimize residual margin and thereby maximize capacity. The following sections discuss methods for adjusting capacity to match the delivered SNR and how modems can report metrics related to this optimization for the purposes of network-level control.
A. Zero-Margin Modem Design
Figure 3 shows the building blocks for typical DSP-based coherent modems [23]. These DSP blocks function to mitigate channel effects and to flexibly tune capacity.
Constellation selection was the first capacity adjustment method to be employed in coherent modems. This method involves choosing from a selection of quadrature amplitude modulation (QAM) constellations, each with equal symbol visitation and a fixed baud, to generate a variable amount of capacity. The first generation of coherent transponders had fixed capacity. The second generation of coherent transponders included constellation selection, e.g., a 35 GBaud transponder could be programmed in fixed steps of 50 Gbps by selecting quadrature phase-shift keying (QPSK) for 100 Gbps, 8-QAM for 150 Gbps, or 16-QAM for 200 Gbps [24].
Finer capacity granularity requires a means of adjusting the average bits/symbol in non-integer steps. Two such methods in use today are time-domain hybrid QAM (TDHQ) [25,26] and probabilistic constellation shaping (PCS) [23,27].
TDHQ sends some fraction of its symbols in different QAM constellations, which allows a nearly continuous adjustment of the average number of bits per symbol. Figure 4 shows an example of TDHQ alternating between two symbols of 8-QAM and one of 16-QAM for an average of 3.33… bits per symbol. The required SNR (RSNR) is also somewhere between the fixed RSNR of the chosen QAMs.
Fig. 4. Time domain hybrid QAM interleaving 8-QAM and 16-QAM symbols to achieve an average of 3.33… bits per symbol.
PCS achieves capacity adjustment by employing a fixed-base QAM constellation but using non-equal visitation probability across the constellation points. The symbol probabilities are chosen to minimize the energy of the average symbol, which increases the energy per bit at the receiver and thus minimizes the RSNR for the given capacity as shown in Fig. 5.
Fig. 5. Probabilistic constellation shaping showing (a) unshaped 64-QAM at 6 bits per symbol and (b) shaped constellation at 5.75 bits per symbol.
Figure 6 shows a comparison of these two methods. The details of the TDHQ are as follows: the spectral efficiency regime from 3 to 4.8 b/s/Hz uses temporal interleaving of QPSK and 8-QAM, 4.8–6.4 b/s/Hz uses interleaving of 8-QAM and 16-QAM, and greater than 6.4 b/s/Hz uses interleaving of 16-QAM and 32-QAM. In each case, the energy of the higher cardinality format was adjusted to optimize the overall RSNR. Results are for a 25% FEC overhead (0.8 FEC rate) and 0.03 pre-FEC bit-error rate (BER) with a transceiver internal noise of 18 dB.
Fig. 7. Line system showing various sources of additive noise and the reference location of the ESNR metric.
For spectral efficiencies in the vicinity of 5 b/s/Hz, the RSNR for TDHQ exceeds PCS by 1.4 dB due to suboptimal Euclidean distance properties of the TDHQ encoding. At higher spectral efficiencies, the RSNR for TDHQ is approximately 2 dB higher than that for PCS. The differences in RSNR for a fixed delivered SNR translate directly into capacity.
The FEC overhead ratio can be used as a point of flexibility in maximizing the capacity through optimizing the SNR margin [28,29]. Lowering the baud of the modem is another means to approach minimum margin by reducing the symbol rate where the benefit is in reduced spectral occupancy. The lowest modem cost per transmitted bit is achieved by maximizing the baud or symbol rate [30] under the assumption that one can utilize the achieved capacity. However, there are situations where the modem cost alone is not dominant or the capacity requirement per modem is fixed. In these cases, it can be beneficial to vary the baud and/or the FEC overhead percentage in order to maximize the conversion of the combination of SNR and spectrum into capacity [31].
B. Monitoring the Modem
Referring again to Fig. 3, channel-dependent effects are compensated for through a series of DSP processing blocks, wherein each block provides a measure of the effect that it is compensating for. Corresponding metrics, which can be extracted from each of these blocks, are listed at the bottom of Fig. 3.
Perhaps the simplest of these metrics is the effective SNR or ESNR [32]. One method to calculate ESNR is from its relationship to the pre-FEC BER. Signal quality metrics other than ESNR are also being explored, including error vector magnitude (EVM) and modulation error ratio (MER) [33].
Knowledge of the ESNR and the RSNR of the modem provides a simple method to estimate the SNR margin. The benefit of margin calculated in this manner is that it includes all noise sources for the instance of transmitter, receiver, and channel as shown in Fig. 7. The calculated SNR margin represents the amount of additional additive white Gaussian noise (AWGN) that could be tolerated by the receiver before reaching the FEC threshold. It should be noted that this applies easily to systems dominated by AWGN, i.e., either dominated by linear noise or by high-net-dispersion environments where the nonlinear noise is dominantly Gaussian at detection [17].
C. Beyond Alarms and Performance Monitors
Traditionally, transport equipment was designed to signal only two states: nominal operation or failure. Failures were signaled by alarms in the network, and the ability to predict failures was very limited. The concept of performance monitors (PMs) existed, but PMs were typically used by human operators to troubleshoot an active issue or to determine the root cause after the fact. In addition, PMs are slow and passive, typically being collected in 15-min bins on the network elements, and due to local storage constraints could be limited in history to as little as 24 h.
There is a desire to move from failure detection to failure prediction. Long-term, in-service monitoring is essential to deliver on the zero-margin promise, and streaming telemetry is a key part of the solution. Streaming telemetry in conjunction with centralized control, storage, and analytics is anticipated to solve many of the issues of traditional PMs [34].
4. BOTTOM-UP MARGIN PREDICTION
Operating a network close to zero margin requires an understanding of the time variation of the noise. This can be quantified through an understanding of the statistics of the noise sources and their associated timescales. The goal of this analysis is to provide a means of categorizing noise sources as either static or time-varying in the assessment of QoS for connections with given durations.
The most important sources of noise include linear or ASE, NLI, modem implementation penalties, and other propagation-related penalties, e.g., linewidth penalties [35]. Partitioning noise sources is important in terms of system optimization. The ratio of linear and nonlinear noises as determined by the receiver can be used to augment the line system optimization. This partitioning has been shown to be possible through analytics applied directly in the modem [36]. Noise partitioning can then be applied to the prediction of the time evolution of margin.
For the purpose of this evaluation, impairments will be treated as independent sources of noise wherever appropriate, with the linear SNRs associated with each source, adding in the following reciprocal:
Since noise powers add, it is convenient to consider the noise-to-signal ratio (NSR) as follows:
Let ${\rm{nsr}_{\rm{line}}}$ be the total NSR contributed by the optical line, that is, any noise source occurring between the transmitter and receiver. The total line NSR contains contributions from ASE, nonlinear noise, linewidth/dispersion interactions, optical filtering, etc., and it can be expressed as
Modem aging can be included as an increase over life of the modem implementation penalty ${\rm{nsr}_{tx/rx}}$. Note that each NSR source may vary in time with dramatically different timescales possible between the different noise sources. For example, the NSR due to ASE, ${\rm{nsr}_{\rm{ase}}}$, can vary over the timescale of several minutes due to line system control or optical power variations, whereas the NSR due to aging may change over months/years.
To handle the different timescales of variation, each NSR source is treated as being composed of a static average contribution together with a zero-mean stochastic contribution that varies on the appropriate timescale as
The timescale of variation is ${\tau _{\rm{source}}}$. We can then estimate the distributions of NSR, ${f_{\rm{source}}}( {{\rm{\Delta }}\rm{nsr}} )$, from independent time-varying contributions about their mean values. Next, we group NSR distributions with similar timescales, ${\tau _{\rm{source}}}$. We can then convolve the zero-mean NSR distributions within each group and assign the group-average value of ${\tau _{\rm{source}}}$ to that group.
At this stage, contributions to the line NSR with comparable timescales of variation are grouped through convolution of their respective PDFs, resulting in a net stochastic NSR contribution for each timescale. Consequently, NSR contributions are grouped such that
NSR groups with timescales longer than the lifetime of the optical QoS will be treated as constant over the QoS lifetime (“frozen-in”) but may vary randomly from connection to connection or network to network. We observe that the impact of frozen-in NSR variations between connections in the same network may be at least partially offset by network layer control, such as SNR optimization, which will act to optimize channel powers and capacity allocation between channels to maximize overall network capacity [19]. Since this control is based on the ESNR as measured by each modem, control will attempt to optimally trade off channel powers (modifying the NLI) with any frozen-in statistics, while also possibly increasing capacity of channels experiencing lower NSR (higher SNR). This control action achieves a form of spectral averaging of the frozen-in per-channel statistics.
In addition to the statistics of SNR, the modems can inherently correct for polarization effects, and in doing so can report estimates of the state of polarization rate of evolution (SOP evolution), the expression of polarization mode dispersion (PMD) in terms of differential group delay (DGD), and PDL. Other metrics available from the modem may be of interest to the capacity routing engine. For example, round-trip delay of modem signaling overhead can provide a measurement of latency. Pre- and post-compensation in the DSP can estimate the net CD. Extracting information from clock and carrier recovery can give insight into phase noise statistics. Amplitude pre- and post-compensation can act as an intra-channel spectral monitor to optimize or troubleshoot optical filtering penalties. Each of these can be used to provide knowledge of the noise statistics and therefore the estimation of the QoS over the timescale of interest.
Visual diagnostics can also be provided, e.g., received constellation diagrams, but with the increased use of complex constellation design like multi-dimensional coding [23] the ability of a human to consume these is becoming limited. However, these metrics are well suited to deep-learning-based pattern recognition.
One challenge in implementing the bottom-up design is that of extracting the data required to characterize the processes that contribute to variations in system margin. This is especially true in multivendor environments where the northbound interfaces to the network controller have traditionally been proprietary. The industry is recognizing the need for normalized vendor-neutral data models, APIs, and protocols such as those using NETCONF and YANG [37].
The inherent risk in all bottom-up analyses is that some contributors to the margin or the statistical variation of margin are not modeled and therefore not accounted for. The following sections present top-down models, which can take advantage of a more global view.
5. NETWORK ANALYTICS
In developing network analytics, it is helpful to break the task down into simpler objectives as shown in Fig. 8. This serves as a roadmap for applying artificial intelligence and machine learning to the network [38].
The first step is Visibility, providing a meaningful analysis of the vast amount of data that networks provide. The next step is Insight, using machine learning to extract important information from the data that is available. The final step is Action, closing the loop through actions and polices defined in a centralized controller.
A. Visibility
The simplest form of top-down analytics is to provide processed data to experienced network engineers such that they can make informed decisions. The problem is the sheer volume of data. There are many monitor points to look at that fluctuate over time. It is difficult, if not impossible, for the unaided human to process this information.
This is the concept of Ciena’s Liquid Spectrum analytic apps like the Blue Planet MCP application Channel Margin Gauge [39], which performs a time-series analysis on the ESNR metric from the modems in a network. Figure 9 shows an example of a field deployed channel measured over a 90-day period where points were taken every 15 min. This channel shows noise-like fluctuations of approximately 0.5 dB on the order of hours and some drift of the approximately same magnitude on the order of days. There is also an event shown in the inset graph between day 30 and 40 where the margin dropped to near zero for one 15-min interval.
Fig. 9. Long-term performance data for a single channel showing ESNR data collected in 15-min intervals and a margin gauge metric for minimum margin estimation.
One of the purposes of the time-series analysis is to extract a lower envelope from this data to provide a short-term margin estimate valid over a period of days and to filter outlier events, which are identified by a larger-than-expected temporary excursion. Any single measurement does not provide adequate guidance, but using the processed data, an experienced network engineer may decide that an additional 2-dB margin could be taken from this channel safely under the current operating conditions with the proviso that the channel is not expected to survive the outlier events. This may be well within the QoT requirements of this channel, since these events are expected to be short lived.
The benefit of mining this margin can be quite high, as there are sometimes situations where a lot of excess margin exists, which can be turned into capacity [40]. However, this analysis is very reactive in nature and requires that some level of intelligence is used to apply this information.
Simple methods to provide prediction capability can be added to this type of time-series analysis, e.g., double exponential smoothing to detect changes in parameters, which correlate with the SNR margin [41]. However, these methods involve some level of intuition to provide the correlated inputs and to set the level of sensitivity. The following section explores methods of machine learning that can help to alleviate these constraints.
Fig. 10. The fiber identification app extracts correlation coefficients from the DSP of the receiver and coordinates the pre-/post-CD compensation to gather data about each of the fiber spans.
Fig. 11. Accuracy of fiber type predictions showing a baseline using a 5-layer ANN and deep learning, which uses a combination of ANNs with a total of 30 layers.
B. Insight
Machine learning provides value in determining parameters within the network that are not directly measurable by the equipment itself but can be discovered though analytics. Artificial neural networks (ANNs) are well suited to the problem of discovering hidden structures in data, for example, the extraction of the transmission fiber parameters.
Fiber type identification is important to ensure optimal system performance. Many cables have various fiber types among the fiber pairs they contain. Although it is usually assumed that fiber types are known at the time of system installation, in practice there is a small fraction of fibers that are either misidentified or the wrong fiber type is chosen. The optimal launch power for different fiber types can vary by several decibels, and setting the wrong target power can result in a significant SNR penalty [42].
Figure 10 shows the application of machine learning to determine fiber coefficients. The DSP in the coherent receiver performs computations on the received symbols to estimate the nonlinear noise imparted through propagation [43]. These computations result in a 2D array of correlation coefficients, which can be arranged into an image. The fiber identification app uses state-of-the-art pattern recognition applied to this image [44]. This work used a deep residual neural network (ResNet) with 25 layers combined with a second fully connected ANN with an additional five layers. The algorithm was implemented using the TensorFlow open source framework. A classic supervised learning methodology was used to train and test the algorithm on large datasets. The results are shown in Fig. 11. The system achieves better than 90% accuracy of fiber type prediction. The accuracy is better for those spans closer to the beginning and end of the path with a small degradation in the middle. There is a large improvement in the identification of the buried spans using the ResNet plus ANN compared to the ANN only.
Fiber-type identification is just one example of insight that can be provided. The goal is to provide a means to model the network so that future states can be evaluated. The following section explores the use of these insights in conjunction with other network data to take the next step in network intelligence.
C. Actions
In the case of applying AI to optical networks, the goal is to algorithmically determine the best network actions and the associated rewards in terms of, for example, capacity and QoT. In the context of closed-loop systems operating SDN controlled networks, the AI can be applied to enable more autonomous networks. The result is a software application that can learn when and how to perform actions on the network elements in order to reach an optimal state.
While this section focuses on a near-term potential commercialization of machine learning techniques for the purpose of network automation, researchers in the field are exploring many techniques including supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning (RL) [45].
Reinforcement learning has received a lot of attention through its success in solving complex goal-seeking problems, for example, the well-known AI AlphaGo [46]. At the high level, these algorithms require a reward function, a parameterization of the network state, and a list of possible actions. Although there are various RL techniques [47], the following example will focus on Q-learning applied to network optimization.
The process begins with a closed loop where, at each iteration, the state of the network $s$ is determined from the telemetry data and the value of the reward $r(s)$ associated with that state is calculated. Then the RL algorithm determines the action $a$ that could be taken on the network in order to bring it to the next state $s\!'$, which is expected to maximize the cumulative reward. Note that “doing nothing” is a valid action. In each iteration the RL algorithm updates the value of the expected cumulative reward $Q$ when being at state $s$ and action $a$ is taken, $Q( {s\!,a} )$, as follows:
After several iterations, the map of $Q( {s\!,a} )$ becomes an accurate description of the network states and their possible best actions. Parameter $\alpha $ affects how quickly the RL will adapt to changing conditions versus how much it will remember its lessons from the past when choosing an action. Although different policies can be used, one possibility is to select a valid action $a$ in a given state $s$ with probability according to the Boltzmann distribution
The choice of hyper-parameter $T$ can be varied to control the amount of exploration, which is done versus exploiting the current best possible action. It is best practice to start with a large $T$, allowing different actions to be chosen to map a larger state space; then, as $T$ tends to zero, it ensures the choice of the best possible action that maximizes the reward.
As for all machine learning, the RL algorithm requires training. In this case the learning provides an accurate model of the network states and its mapping to optimal actions.
In a green field deployment, the operator could let the network operate with control traffic (without real customer traffic) to let the RL learn by trial and error. If this is not possible, the RL can learn from historical data using an imitation learning strategy. Alternatively, if available, the RL algorithm can be trained using a network simulator.
Operating a network near zero margin increases the risk of short-term outages on individual channels in the network. The benefit is the fact that more net capacity can be delivered even when considering these intermittent dropped connections. Dealing with this intermittent connectivity requires that the network layers above the photonic and modem layer deal with temporarily dropped traffic. The Internet was designed to manage variable dial-up rates in modems, lost packets, etc. However, as data connection reliability increased, especially in the core of the network, applications have become increasingly intolerant to these types of interruptions. It is therefore beneficial to employ an intelligent multi-layer controller to predict and adapt to such changes, preventing as much packet loss as possible.
Figure 12 shows an example of RL applied to a packet optical network. Allocated bandwidth, transmitted bytes, and dropped bytes for three competing services are shown. Services 2 and 3 have priority 10, whereas service 1 has lower priority 1. To the left of the vertical dashed line is the time before activation of the RL application, while to the right is after activation. The RL algorithm is acting to minimize dropped packets from high-priority services while maximizing overall throughput. The algorithm is able to adapt to changing traffic patterns and bandwidth availability.
6. CONCLUSIONS
Modern optical network design is continuously evolving to adapt to the increasing demands for capacity and automation. This trend includes a flexible photonic line system designed to maximize capacity. Line system instrumentation is becoming increasingly important and, along with developments in optimization algorithms, is being used to extract more SNR from deployed systems.
Variable capacity modems provide the capability to tune capacity to the delivered SNR leaving the near-zero margin. Monitoring the residual margin is key to delivering the desired QoT. Metrics used in margin estimation are important inputs to characterizing the time-varying behavior of the network. The understanding of the statistical variation of performance and the associated timescales provides guidance for capacity assignment algorithms and higher-level network control.
In the context of centralized network control, or SDN, artificial intelligence is already playing an important role to provide analytics and insight from vast amounts of network data. There is great promise in the application of machine learning and closed-loop automation to achieve the goals of near-zero margin networking.
REFERENCES
1. “Cisco Visual Networking Index: Forecast and Trends, 2017–2022,” White Paper, 2018, http://www.cisco.com.
2. M. O’sullivan, “Progress of digital coherent optical communication systems,” in Conference on Lasers and Electro-Optics (2017), paper STu3M.5.
3. K. Roberts, M. O’Sullivan, M. Reimer, and M. Hubbard, “Nonlinear mitigation enabling next generation high speed optical transport beyond 100G,” in Optical Fiber Communication Conference (OFC) (2019), paper M3J.1.
4. D. Pilori, A. Nespola, P. Poggiolini, F. Forghieri, and G. Bosco, “Low-complexity non-linear phase noise mitigation using a modified soft-decoding strategy,” in Optical Fiber Communication Conference (OFC) (2019), paper M1I.2.
5. D. Krause, A. Awadalla, A. S. Karar, H. Sun, and K. Wu, “Design considerations for a digital subcarrier coherent optical modem,” in Optical Fiber Communication Conference (OFC) (2017), paper Th1D.1.
6. O. Gerstel, M. Jinno, A. Lord, and S. J. B. Yoo, “Elastic optical networking: a new dawn for the optical layer?” IEEE Commun. Mag.50(2), s12–s20 (2012). [CrossRef]
7. A. A. Díaz-Montiel, J. Yu, W. Mo, Y. Li, D. C. Kilper, and M. Ruffini, “Performance analysis of QoT estimator in SDN-controlled ROADM networks,” in International Conference on Optical Network Design and Modeling (ONDM), Dublin, Ireland, 2018, pp. 142–147.
8. J. Pedro and N. Costa, “Optimized hybrid Raman/EDFA amplifier placement for DWDM mesh networks,” J. Lightwave Technol.36, 1552–1561 (2018). [CrossRef]
9. “Spectral grids for WDM applications: DWDM frequency grid,” ITU-T Recommendation G.694.1, https://www.itu.int/rec/T-REC-G.694.1.
10. A. Pilipetskii, O. Sinkin, A. Turukhin, M. Bolshtyansky, and D. Foursa, “The role of SDM in future transoceanic transmission systems (Invited),” in European Conference on Optical Communication (ECOC), Gothenburg, Sweden, 2017.
11. H. Fevrier, S. Grubb, N. Harrington, A. Palmer-Felgate, E. Rivera-Hartling, and T. Stuch, “Facebook perspective on submarine wet plant evolution,” in Optical Fiber Communication Conference (OFC) (2019), paper M2E.4.
12. S. Grubb, P. Mertz, A. Kumpera, L. Dardis, J. Rahn, J. O’Connor, and M. Mitchell, “Real-time 16QAM transatlantic record spectral efficiency of 6.21 b/s/Hz enabling 26.2 Tbps capacity,” in Optical Fiber Communication Conference (OFC) (2019), paper M2E.6.
13. P. Poggiolini, “The GN model of non-linear propagation in uncompensated coherent optical systems,” J. Lightwave Technol.30, 3857–3879 (2012). [CrossRef]
14. P. Poggiolini, G. Bosco, A. Carena, R. Cigliutti, V. Curri, F. Forghieri, R. Pastorelli, and S. Piciaccia, “The LOGON strategy for low-complexity control plane implementation in new-generation flexible networks,” in Optical Fiber Communication Conference and Exposition and the National Fiber Optic Engineers Conference (OFC/NFOEC), Anaheim, California, 2013, paper OW1H.3.
15. M. Cantono, D. Pilori, A. Ferrari, C. Catanese, J. Thouras, J.-L. Augé, and V. Curri, “On the interplay of nonlinear interference generation with stimulated Raman scattering for QoT estimation,” J. Lightwave Technol.36, 3131–3141 (2018). [CrossRef]
16. D. Semrau, R. I. Killey, and P. Bayvel, “A closed-form approximation of the Gaussian noise model in the presence of inter-channel stimulated Raman scattering,” J. Lightwave Technol.37, 1924–1936 (2019). [CrossRef]
17. P. Poggiolini, M. R. Zefreh, G. Bosco, F. Forghieri, and S. Piciaccia, “Accurate non-linearity fully-closed-form formula based on the GN/EGN model and large-data-set fitting,” in Optical Fiber Communication Conference (OFC) (2019), paper M1I.4.
18. I. Roberts, J. M. Kahn, and D. Boertjes, “Convex channel power optimization in nonlinear WDM systems using Gaussian noise model,” J. Lightwave Technol.34, 3212–3222 (2016). [CrossRef]
19. I. Roberts and J. M. Kahn, “Efficient discrete rate assignment and power optimization in optical communication systems following the Gaussian noise model,” J. Lightwave Technol.35, 4425–4437 (2017). [CrossRef]
20. I. Roberts, J. M. Kahn, J. Harley, and D. W. Boertjes, “Channel power optimization of WDM systems following Gaussian noise nonlinearity model in presence of stimulated Raman scattering,” J. Lightwave Technol.35, 5237–5249 (2017). [CrossRef]
21. A. N. Pilipetskii and E. A. Golovchenko, “Performance fluctuations in submarine WDM systems,” J. Lightwave Technol.24, 4208–4214 (2006). [CrossRef]
22. M. Bolshtyansky and G. Cowle, “Polarization hole burning in EDFA,” in Optical Fiber Communication Conference (OFC) (2012), paper OW4D.2.
23. K. Roberts, Q. Zhuge, I. Monga, S. Gareau, and C. Laperle, “Beyond 100 Gb/s: capacity, flexibility, and network optimization [Invited],” J. Opt. Commun. Netw.9, C12–C24 (2017). [CrossRef]
24. K. Roberts, S. H. Foo, M. Moyer, M. Hubbard, A. Sinclair, J. Gaudette, and C. Laperle, “High capacity transport—100G and beyond,” J. Lightwave Technol.33, 563–578 (2015). [CrossRef]
25. Q. Zhuge, M. Morsy-Osman, X. Xu, M. Chagnon, M. Qiu, and D. Plant, “Spectral efficiency-adaptive optical transmission using time domain hybrid QAM for agile optical networks,” J. Lightwave Technol.31, 2621–2628 (2013). [CrossRef]
26. X. Zhou and L. Nelson, “Advanced DSP for 400 Gb/s and beyond optical networks,” J. Lightwave Technol.32, 2716–2725 (2014). [CrossRef]
27. M. Yankov, D. Zibar, K. Larsen, L. Christensen, and S. Forchhammer, “Constellation shaping for fiber-optic channels with QAM and high spectral efficiency,” IEEE Photon. Technol. Lett.26, 2407–2410 (2014). [CrossRef]
28. A. L. N. Souza, E. J. Mayoral Ruiz, J. D. Reis, L. H. H. Carvalho, J. R. F. Oliveira, D. S. Arantes, M. H. M. Costa, and D. A. A. Mello, “Parameter selection in optical networks with variable-code-rate superchannels,” J. Opt. Commun. Netw.8, A152–A161 (2016). [CrossRef]
29. T. Koike-Akino, K. Kojima, D. S. Millar, K. Parsons, T. Yoshida, and T. Sugihara, “Pareto optimization of adaptive modulation and coding set in nonlinear fiber-optic systems,” J. Lightwave Technol.35, 1041–1049 (2017). [CrossRef]
30. M. O’Sullivan, M. Reimer, and M. Bélanger, “Nascent line-side modem solutions for high capacity optical networks,” in Optical Fiber Communication Conference (OFC) (2016), paper M2J.1.
31. P. Poggiolini, A. Carena, Y. Jiang, G. Bosco, and F. Forghieri, “On the ultimate potential of symbol-rate optimization for increasing system maximum reach,” in European Conference on Optical Communication (ECOC), Valencia, Spain, 2015.
32. L. Berg, “Demystifying transceiver and line characterization metrics,” in Optical Fiber Communication Conference (OFC) (2019), paper W4H.3.
33. J. Lai, R. Tang, B. Wu, S. Li, X. Zhao, X. Tang, W. Zhao, and H. Zhang, “Research on optical performance monitoring (OPM) in coherent transmission system,” in Asia Communications and Photonics Conference (2016), paper AS2B.2.
34. A. Sadasivarao, S. Jain, S. Syed, K. Pithewan, P. Kantak, B. Lu, and L. Paraschis, “High performance streaming telemetry in optical transport networks,” in Optical Fiber Communication Conference (OFC) (2018), paper Tu3D.3.
35. Q. Zhuge, X. Xu, Z. A. El-Sahn, M. E. Mousa-Pasandi, M. Morsy-Osman, M. Chagnon, M. Qiu, and D. V. Plant, “Experimental investigation of the equalization-enhanced phase noise in long haul 56 Gbaud DP-QPSK systems,” Opt. Express20, 13841–13846 (2012). [CrossRef]
36. F. J. V. Caballero, D. Ives, Q. Zhuge, M. O’Sullivan, and S. J. Savory, “Joint estimation of linear and non-linear signal-to-noise ratio based on neural networks,” in Optical Fiber Communications Conference and Exposition (OFC) (2018), paper M2F.4.
37. P. Castoldi, A. Giorgetti, A. Sgambelluri, G. Cecchetti, A. L. Ruscelli, N. Sambo, F. Paolucci, A. Morea, I. Cerutti, and F. Cugini, “Optical white box: modeling and implementation,” in 20th International Conference on Transparent Optical Networks (ICTON), Bucharest, Romania, 2018, pp. 1–4.
38. D. Côté, “Using machine learning in communication networks [Invited],” J. Opt. Commun. Netw.10, D100–D109 (2018). [CrossRef]
39. “Liquid spectrum: transforming static optical networks into dynamic, responsive assets,” Application Note, Ciena Corporation, https://www.ciena.com/insights/app-notes/Liquid-Spectrum-Transforming-Optical-Networks.html.
40. D. W. Boertjes, A. Leong, and D. Attard, “Field trial of short term capacity optimization on a live system,” in Photonics and Fiber Technology (ACOFT, BGPP, NP) (2016), paper AW5C.3.
41. Z. Wang, M. Zhang, D. Wang, C. Song, M. Liu, J. Li, L. Lou, and Z. Liu, “Failure prediction using machine learning and time series in optical network,” Opt. Express25, 18553–18565 (2017). [CrossRef]
42. S. J. Savory, “Congestion aware routing in nonlinear elastic optical networks,” IEEE Photon. Technol. Lett.26, 1057–1060 (2014). [CrossRef]
43. F. J. V. Caballero, D. J. Ives, C. Laperle, D. Charlton, Q. Zhuge, M. O’Sullivan, and S. J. Savory, “Machine learning based linear and nonlinear noise estimation,” J. Opt. Commun. Netw.10, D42–D51 (2018). [CrossRef]
44. D. Cote, “Using machine learning in communication networks,” in IEEE Photonics Society Summer Topicals Meeting Series, Waikoloa, Hawaii, 9 –11 July 2018, paper TuB2.1.
45. F. Musumeci, C. Rottondi, A. Nag, I. Macaluso, D. Zibar, M. Ruffini, and M. Tornatore, “An overview on application of machine learning techniques in optical networks,” IEEE Commun. Surv. Tutorials21, 1383–1408 (2019). [CrossRef]
46. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of Go with deep neural networks and tree search,” Nature529, 484–489 (2016). [CrossRef]
47. G. A. Rummery and M. Niranjan, “On-line Q-learning using connectionist systems,” Technical Report CUED/F-INFENG/TR 166 (1994).
David W. Boertjes received his B.Sc. in physics from the University of New Brunswick, Fredericton, NB, Canada, in 1993; his M.Sc. in physics, from Dalhousie University, Halifax, NS, Canada, in 1995; and his Ph.D. in electrical engineering from the University of Alberta, Edmonton, AB, Canada, in 1998. From 1995 to 1998 he did research on active and passive polymer optics for low-cost photonic components for use in optical telecommunications at TRLabs in Edmonton, AB, Canada. In 1998 he joined Nortel’s Optical Networks division. In 2010 he was part of the Nortel MEN team acquired by Ciena. Currently he is a director in R&D working on liquid spectrum applications in Ottawa, Canada. He is currently a senior member of The Optical Society (OSA) and received Ciena’s Distinguished Engineer Award in 2014. He has also served on two OFC subcommittees and has served as the chair of each.
David Côté is co-founder and lead engineer of the Blue Planet Analytics (BPA) program at Ciena. He holds many patents in the field. His group is developing a big data and machine learning platform that supports software products and professional services for Ciena’s customers. He also participates in various research projects with Ciena’s Photonics Group and CTO office, as well as university partners. BPA was recognized by various industry innovation awards in 2017–2018. He received a Ph.D. in physics from Université de Montréal in 2007. Prior to joining Ciena, he worked as a particle physicist for 14 years at the Stanford Linear Accelerator Center (USA) and CERN (Switzerland), notably, where he got substantial hands-on experience with big data engineering, data science, and world-class research.
