Facebook Datacenter consists of large number of servers that run diverse Facebook services. Various services are aggregated to serve any given user request. To allow this aggregation, servers have to interact with each other via different traffic flows which are managed by networking fabric. The underlying connection powering this fabric consists of large number of pluggable optical interconnects and On Board Optical (OBO) modules carrying production data. This connectivity at scale requires fast and reliable detection of the link failures to ensure resolution. In the first generation of the deployments, detection of the link failure was sequential and a slow process. The troubleshoot process was equally tedious as the available tools required characterizing one optical transceiver at a time. Further, the failure analysis also presented majority of resolution with no failed optics as a root cause resulting in high No Trouble Found (NTF) rate. 
 
Here we introduce a novel link failure detection and resolution method that improves on the previous method on three dimensions: faster resolution, more reliable troubleshooting and scalable implementation. We introduce BER Illusion Methodology (BIM) that is highly scalable and a cost-effective solution that significantly reduces the time taken to troubleshoot pluggable optical interconnects. This is also scalable to next-gen OBO modules at Facebook datacenters aiming to lower the NTF rate and optimally utilizing the available resources. BIM, which is based on Open Compute Platform (OCP) network switches, can be used to troubleshoot 128 QSFP28 , 64 QSFP56 or 32 OBO modules simultaneously in under 30 minutes. The tool is easy to implement and capable of also reporting diagnostics on the transceiver such as Transmitter Power, Transmitter Bias Current, Receiver Power, Case Temperature, Bit Error Rate result per channel, Vendor information and Manufacturing part number. This additional test data report along with true failure indication helps optic suppliers gain confidence and build customer credibility. The open source nature and the universal applicability of this tool offers possibility for other users to adopt and further customize it for their networking needs. 
&nbsp;

BER Illusion Methodology: A Novel, Open Sourced and Scalable Approach to Troubleshooting High Radix Photonics Interconnects in a Modern Hyperscale Datacenter

Paper Proceedings: https://asmedigitalcollection.asme.org/InterPACK/InterPACK2021/volume/85505

ASME 2021 International Electronic Packaging Technical Conference and Exhibition (InterPACK2021)

Supporting the need among researchers worldwide for additional online and multimedia content, ASME now offers the ASME Conference Video Collection. This platform will collect and provide long-term access to conference presentation videos, encompassing all technology areas within ASME's technical conference and events program. About 30,000-40,000 engineers across all mechanical engineering and related disciplines participate in ASME conferences each year. This includes authors of approximately 6,000 to 8,000 technical papers published by ASME, the majority of whom also submit video presentations for our virtual conferences. The video collection will be available as an add-on to existing product packages or as a separate subscription.