Karel Hynek: Network Monitoring with ipfixprobe

The monitoring infrastructure Flox collects data aggregated into information about individual flows. Analytics on network traffic can then be performed on this data. The ipfixprobe tool is a modular flow exporter that can collect data from various sources such as PCAP, NDP, DPDK, or raw socket. NDP is used for acceleration FPGA cards, and raw socket allows data extraction from the Linux kernel.

The output of ipfixprobe is bidirectional flow, but it can be configured to export unidirectional flow. Classic information from NetFloxV5 is available, such as IP addresses, MAC addresses, ports, byte and packet counts. We started exporting data for machine learning as well, such as packet length and time sequences for the first thirty packets, and histograms of packet lengths and times. Documentation is available in the Git repository, and source codes are on GitHub. Packages for EPEL8 and EPEL9 are available, but the packages are also accessible in many different distributions. In practice, the tool is deployed on eight peering links of the CESNET network, which have a throughput of more than 100 Gbps. We use standard Dell servers with two processors and custom SmartNIC cards. Each server can monitor multiple links, with the probe's lossless throughput reaching 175 Gbps at worst. The probe is also deployed at the University of Dresden, in the backbone network of the Vysočina region, and testing is also ongoing in the Seznam.cz network.

In the past year, significant improvements have been made to the DPDK plugin, which can run as a secondary data extractor. Support for common SmartNIC cards has also been expanded; tested cards include Nvidia Mellanox ConnectX-6 and Broadcom N1400GD with a 400GE port. We cannot yet monitor full 400 gigabits due to PCI Express limitations. Thanks to buffer adjustments, we’ve achieved roughly a thousandfold improvement in packet loss at full saturation. The probe can also process packets intended for connection establishment using QUIC or HTTP/3. Initial packets are obfuscated using a known key, but we need to extract information from them, which slows us down. The current version can handle losslessly a 40Gbps peering with Google. Work is underway to support monitoring 400GE networks, but the challenge lies in transferring data from the card to the CPU. PCI Express can transfer up to 500 Gbps in its fifth generation, but that’s not enough for bidirectional transfer. The sixth generation can transfer up to 900 Gbps, but there are still very few servers available. Therefore, it is necessary to ensure the offload of flow records in NIC FPGA firmware. More than 70% of data is transferred in 10% of flows. The probe can then either receive everything, just trimmed headers, or only aggregated flow info. Everything is in the internal testing phase, and support should be available during the summer.

To simplify the creation and orchestration of the monitoring infrastructure, the web interface Panda is used, which guides you through everything. Infrastructures can be very complex, with several probes, collectors, and servers for analytics available, and Panda allows you to set everything up and manage it. Panda will be available under an open license during the summer.

Blažej Krajňák: sFlow Visualization Tool

The sFlow protocol collects packet headers and adds several metadata such as port or routing information. This data flows into GoFlow2 and then into a ClickHouse database, which stores column-oriented data. Usually, you don’t need to list all the information about every packet; you can choose specific columns. Additionally, the database is compressed, allowing you to store fifteen to twenty times more data.

ClickHouse can dynamically enrich data from other sources and can keep small tables in RAM to increase performance. The ALIAS data type is useful, as it can calculate a column using a specified logic, which isn’t actually stored in the database. For example, the DSCP value is calculated by shifting the ToS by two bits, and no additional data needs to be stored for that. When analyzing, it’s important to have real-time data access, which would take a long time if processing large datasets. We can create a reasonable sampling over the tables at the beginning, which selects just a sample of data and thus saves time during processing.

The output is then provided through a dashboard, where user-friendly filtering with an autocomplete function is available, and individual packets can be selected or an SQL query can be entered manually. The longer the time frame you choose for the output, the larger the sampling will be, but you can adjust it manually or turn it off completely. Visualization is then handled by Grafana.

Patrick Prangl: The Rise of Merchant Silicon

Merchant silicon refers to chips produced in bulk by wholesale suppliers and sold to network device manufacturers. These chips can be either FPGA or ASIC. The latter is more static, simpler, and has lower power consumption, which is enough for many deployments. They process packets so that the universal CPU doesn't have to. We offer 100GE and 400GE variants, have 800GE available, and are preparing for 1600GE.

The performance of these solutions is limited by the speed at which control chips are connected to switch chips. The speed has been gradually increasing from 10GE to 50GE, and up to 100GE, with possible further growth. Buffers are crucial for scenarios with many sources communicating with few targets or when the link speed needs to change along the route.
Merchant chips have been around since 2008, initially used for edge switching. Since about 2016, chips suitable for all parts of large networks have been available. Technological advancements have enabled the use of 3nm technology, improving chip performance and reducing power consumption. All chip manufacturers have access to the same technologies and production processes. The difference lies in the design for different deployments: from chips like Tofino to the more powerful Tomahawk, Trident, and Jericho, designed for high-performance routing. Tofino is now a dead project, though it is still available for purchase, and no further development is expected. Tomahawk 5 was originally designed for AI networks, with a throughput of up to 51.2 Tbps. Trident, used for campus networks, offers larger buffers and more features. The latest generation consumes 40% less power per bit and comes in a smaller package. Jericho focuses on large buffer capacities, ideal for service providers that need complex features.

Lukáš Šišmiš: Overcoming the IDS Stereotype

Suricata generates security alerts related to network traffic. Typically, a line is diverted from a router or switch to Suricata, which handles the preparation of security data. It can be deployed as an IDS or IPS, where it uses rules to drop certain traffic. The option of using Suricata simply for exporting traffic metadata in JSON format is often overlooked.

Suricata can also serve as a flow probe and, with the right hardware, can handle up to 400 Gbps. However, it should be noted that data is only exported after the flow ends. The upcoming Suricata 8 will support continuous export, and is expected to be released in the fall. Data export can be elevated, using Suricata as a Network Security Monitor to export records for individual application protocols such as DNS, HTTP, TLS, SMTP, and others. This allows obtaining detailed information on individual files moving across the network, which can be useful for audits. Suricata can also be used to detect misconfigured devices in the network. While switches can't view the content of specific communications, Suricata can. With basic rules installed, it can monitor events like corrupted HTTP traffic, DNS errors, asymmetric routing, TTL expiry, and more. These rules should be enabled when deploying Suricata, as they can generate a lot of data, and security events could get lost among them.

Suricata can also function as a firewall, which Amazon uses in its cloud services. Unlike traditional firewalls, Suricata defaults to allowing traffic and only blocks specific traffic at higher layers based on rules. This can be marketed as a next-generation firewall. With Suricata 8, a library will be available that allows integration into custom applications. This means your applications won't have to parse traffic and send it to Suricata manually; the library will make the process more efficient.

Michal Hažlinský: SDN on L0 with Open Hardware

In optical networks, communication requires several devices, especially in DWDM networks. These elements are called the zero network layer, typically including multiplexers/demultiplexers and optical amplifiers. These devices process the optical signal without converting it to electrical form. CESNET develops and operates its own optical devices called CzechLight.

The optical network can be controlled openly using SDN (Software Defined Network). At its core is ROADM (Reconfigurable Optical Add-Drop Multiplexer), which allows switching individual optical channels and terminating them locally. It’s similar to VLAN in regular switches but involves optical signals. The goal of configuration is to set all devices along the path so the signal reaches the destination from the input device through the infrastructure. In the rack, 1U boxes are placed, offering up to eight ports, along with optical amplifiers. These can be configured via API. Standard protocols like YANG, Netconf, and others are used for configuration. By using these common protocols, we can leverage modern tools, with DevOps practices like using Ansible and storing configurations in Git. The operation of switching a channel involves adding and removing port configurations in individual devices.

The openness of the optical layer can also support other services, not just data transmission. For example, CESNET works with the distribution of precise time, and time sources are available at certain points. For high accuracy, the same fiber must be used for both directions to avoid errors from asymmetry. Amplifiers need to be added to the network to filter, amplify, and return specific signals. This system can be managed using the software architecture.

Vladimír Bureš: Evolution to SRv6

Before 2000, the MPLS protocol was created, adding labels to traffic. Information about these labels is spread using the LDP protocol, which works with the local routing protocol. It seemed like a good idea because any routing protocol could be used. Between the second and third layer headers, a label stack can be stored, containing a list of labels for routing.

This solution was good at the time but showed its drawbacks over time. Notably, we need to maintain an additional protocol running parallel to the regular routing protocol. Synchronization between LDP and IGP must be addressed, and when issues occur, customer outages can be longer. As network elements converge at different times, loops can form. This can take a few tens of milliseconds, but today, we want recovery to be guaranteed within fifty milliseconds. The next step is segment routing, which aims to simplify the process and eliminate protocols with redundant functions. It’s also more scalable, as elements don’t need to store labels. The network doesn’t hold any state information and uses source routing, meaning the network behaves independently for each packet without a pre-established path.

For segment routing, two different data planes can be used: MPLS or native IPv6. In IPv6, the segment is an IPv6 address, providing 128 bits, and the segment list is stored in the extended header. In MPLS, labels are placed at the start of the packet, consumed, and removed along the path. In SRv6, the items are not physically removed from the header, but an indicator for the current label is stored. The header includes the destination address and a next-header item that specifies what follows. It could be TCP, UDP, another IPv4 packet, or an Ethernet frame. Essentially, anything can be encapsulated. Routing is then done through standard IPv6 routing. Not all nodes need to understand SRv6 since it uses regular routing.

Radim Roška, Vojtěch Šetina: Automation of Data Center Configuration at ČRA

For network automation, the key is the SOT (Source of Truth), which contains information about how network elements should behave. The network can be virtualized using tools like CINTAINERlab, which allows simulation and testing before deployment. This forms the first phase, which can gradually replace the original manual solution.

For automation, AVD templates from Arista can be used, containing pre-configured settings for common situations. Configuration is defined using variables; adjusting two lines changes many settings across devices. The output is fed into the CloudVision configuration tool, which applies the changes. In the second phase, additional Python tools were added to simplify management. For example, the Editor script allows users to prepare templates and version them in Git. When creating a new service, the Editor initiates a change branch, pushes it to the main branch, validates it automatically, and then compiles the data model to generate new configurations. The final step is deploying everything to the elements.

This is essentially the same DevOps approach that’s now a natural part of software development, gradually extending to network element configuration. The result is greater efficiency, reduced error risk, and the ability to configure the network without detailed knowledge. Full change history and the ability to revert to a previous state are also available.

Marian Rychtecký: How (Not) to Count Statistics

A traditional tool for graphing time series data is MRTG, which has been used for decades. Its drawback is limited flexibility, fixed time windows, and dependency on SNMP. However, it's simple and outputs static images. Due to averaging data over fixed intervals, short-term spikes can sometimes disappear from the graph.

NIX.CZ wanted to create a new portal with improved graphing. After testing several solutions, developers decided on their own system. Network data is stored in a database, enriched with additional data from Netbox, and then combined and stored in InfluxDB via an API. Data is gathered every 30 seconds and stored in different forms for different windows. Three values are saved: minimum, maximum, and average.

For more accurate data, sampling was moved to 20 seconds, and then every minute, the three values (maximum, minimum, and average) are averaged. New graphs better reflect reality and are smoother. Data was also enriched with state data and time intervals related to port availability, which can be done directly in InfluxDB. Data is computed in a cascade, where longer intervals are recalculated from higher-level data sets. For yearly graph summarization, it's better to use monthly data rather than raw input. 

Zdeněk Brůna: Automatic DNSSEC Management

DNSSEC is a security extension to DNS that uses asymmetric cryptography for signing records. It requires support from the domain registry, technical administrators, and connection providers. .CZ registry has supported DNSSEC since 2008, though resolver support wasn’t widespread. Therefore, the ODVR open resolver service was launched, and it now runs on the Knot Resolver.

The support rate for the .CZ domain grew to around 60%, but has stagnated for some time. The right registrar usually makes DNSSEC deployment seamless. There are large differences among registrars, but the biggest ones usually have no issues. For resolvers, support is high but fluctuates between 70% and 90%. You can check if anything is wrong in your network.
For a small number of domains, DNSSEC can be implemented manually by generating a key, signing records locally, and then publishing DS records via a registrar. Alternatively, the registrar can sign the zone for you. The third option involves generating a CDNSKEY record, with the registrar taking care of the rest. All this can be automated and is supported by most authoritative servers. 

Automatic DNSSEC management was introduced in RFC 7344 and 8078, allowing implementation without coordination with the domain holder, which is often complex. This has been supported in FRED since 2017, and Knot DNS has supported it since version 2.5.0. The registry scans domains for CDNSKEY records and initiates DNSSEC based on them.
This automatic management is considered the highest level of DNSSEC implementation, supported by only seven ccTLDs. Along with .cz, these include .sk, .ch, .li, .se, .nu, and .cr. Costa Rica uses our FRED system and could easily implement support. Initially, scan results were sent to technical administrators via automated emails, which was cumbersome. Now, scan results appear in WHOIS, where the entire process can be tracked.

In the future, the system could be improved by scanning from multiple locations, continuously evaluating scans throughout the day, and refining WHOIS display. Currently aimed at technicians, the goal is to make it more user-friendly. Consideration is being given to implementing RFC 9615, where the CDNSKEY record can be inserted into another already signed DNS operator zone.