Skip to content

Commit e240546

Browse files
mickel8mickel8
committed
Add "For Developers" guides - Debugging, Encryption and SDP
1 parent e339453 commit e240546

File tree

4 files changed

+342
-2
lines changed

4 files changed

+342
-2
lines changed

‎guides/for_developers/fd_debugging.md

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# WebRTC debugging
2+
3+
**It is also worth taking a look at [debugging](../advanced/debugging.md)**
4+
5+
6+
In most cases, when **something** does not work, we try to find the problem according to the following workflow:
7+
1. Check whether session has been negotiated successfully.
8+
1. Check whether connection (ICE and DTLS) has been established.
9+
1. Check whether RTP packets are demuxed, frames assembled and decoded.
10+
1. Check QoE statistics - freezes, jitter, packet loss, bitrate, fps.
11+
12+
13+
```mermaid
14+
flowchart TD
15+
S["Types of problems in WebRTC"] --> Session["Session negotiation (number of tracks, codecs)"]
16+
S --> Connection["Connection establishment (ICE and DTLS)"]
17+
S --> Playout["Playout (demuxing, packetization, decoding)"]
18+
S --> QoE["QoE (freezes, low quality, low fps)"]
19+
```
20+
21+
## Session Negotiation
22+
23+
Here, we just validate that SDP offer/answer looks the way it should.
24+
In particular:
25+
1. Check number of audio and video mlines.
26+
1. Check if any mlines are rejected (either by presence of port 0 in the m="" or a=inactive).
27+
In most cases port is set to 9 (which means automatic negotiation by ICE) or if ICE is already in progress or this is subsequent negotiation, it might be set to a port currently used by the ICE agent. Port 0 appears when someone stops transceiver via [`stop()`](https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpTransceiver/stop).
28+
1. Check mlines directions (a=sendrecv/sendonly/recvonly/inactive)
29+
1. Check codecs, their profiles and payload types.
30+
1. Number of mlines between offer and answer cannot change.
31+
This means that if one side offer that it is willing to only receive a single audio track,
32+
everything the other side can do is either confirm it will be sending or decline and say it won't be sending.
33+
If the other side also wants to send, additional negotiation has to be performed **in this case**.
34+
35+
SDP offer/answer can be easily checked in chrome://webrtc-internals (in chromium based browsers) or (about:webrtc in FF).
36+
37+
## Connection establishment
38+
39+
WebRTC connection state (PeerConnection or PC state) is a sum of ICE connection state and DTLS state.
40+
In particular, PC is in state connected when both ICE and DTLS is in state connected.
41+
42+
The whole flow looks like this:
43+
1. ICE searches for a pair of local and remote address that can be used to send and receive data.
44+
1. Once a valid pair is found, ICE changes its state to connected and DTLS handshake is started.
45+
1. Once DTLS handshake finishes, DTLS changes its state to connected and so the whole PC.
46+
1. In the meantime, ICE continues checking other pairs of local and remote address in case there is better path.
47+
If there is, ICE seamlessly switches to it - the transmission is not stopped or interrupted.
48+
49+
50+
More on ICE, its state changes, failures and restarts in section devoted to ICE.
51+
52+
In most cases, DTLS handshake works correctly. Most problems are related to ICE as it's pretty complex protocol.
53+
54+
Debugging ICE:
55+
56+
1. Check ICE candidates grid in chrome://webrtc-internals or about:webrtc
57+
1. Turn on debug logs in ex_ice or chromium (via command line argument). FF exposes all ICE logs in about:webrtc->Connection Log.
58+
Every implementation (ex_ice, chromium, ff) is very verbose.
59+
You can compare what's happening on both sides.
60+
1. Try to filter out some of the local network interfaces and remove STUN/TURN servers to reduce complexity of ICE candidate grid, amount of logs and number of connectivity checks.
61+
In ex_webrtc, this is possible via [configuration options](https://hexdocs.pm/ex_webrtc/0.14.0/ExWebRTC.PeerConnection.Configuration.html#t:options/0).
62+
1. Use Wireshark.
63+
Use filters on src/dst ip/port, udp and stun protocols.
64+
This way you can analyze whole STUN/ICE/TURN traffic between a single local and remote address.
65+
66+
Debugging DTLS:
67+
68+
This is really rare.
69+
We used Wireshark or turned on [debug logs in ex_dtls](https://hexdocs.pm/ex_dtls/0.17.0/readme.html#debugging).
70+
71+
## Playout
72+
73+
If both session negotiation and connection establishment went well, you can observe packets are flowing but nothing is visible in the web browser, the problem might be in RTP packets demuxing, frames assembly or frames decoding on the client side.
74+
75+
1. We heavily rely on chrome://webrtc-internals here.
76+
1. Check counters: packetsReceived, framesReceived, framesDecoded, framesDropped.
77+
1. E.g. if packetsReceived increases but framesReceived does not, it means that there is a problem in assembling video frames from RTP packets. This can happen when:
78+
1. web browser is not able to correctly demux incomming RTP streams possibly because sender uses incorrect payload type in RTP packets (different than the one announced in SDP) or does not include MID in RTP headers.
79+
Keep in mind that MID MAY be sent only at the beginning of the transmission to save bandwidth.
80+
This is enough to create a mapping between SSRC and MID on the receiver side.
81+
1. marker bit in RTP header is incorrectly set by the sender (although dependent on the codec, in case of video, marker bit is typically set when an RTP packet contains the end of a video frame)
82+
1. media is incorrectly packed into RTP packet payload because of bugs in RTP payloader
83+
1. E.g. if packetsReceived increases, framesReceived increases but framesDecoded does not, it probably means errors in decoding process.
84+
In this case, framesDropped will probably also increase.
85+
1. framesDropped may also increase when frames are assembled too late i.e. their playout time has passed.
86+
1. Check borwser logs.
87+
Some of the errors (e.g. decoder errors) might be logged.
88+
89+
## QoE
90+
91+
The hardest thing to debug.
92+
Mostly because it very often depends on a lot of factors (network condition, hardware, sender capabilities, mobile devices).
93+
Problems with QoE are hard to reproduce, very often don't occur in local/office environment.
94+
95+
1. We heavily rely on chrome://webrtc-internals here.
96+
1. Check coutners: nackCount, retransmittedPacketsSent, packetsLost.
97+
Retransmissions (RTX) are must have.
98+
Without RTX, even 1% of packet loss will have very big impact on QoE.
99+
1. Check incoming/outgoing bitrate and its stability.
100+
1. Check jitterBufferDelay/jitterBufferEmittedCount_in_ms - this is avg time each video frame spends in jitter buffer before being emitted for plaout.
101+
1. JitterBuffer is adjusted dynamically.
102+
103+
## Debugging in production
104+
105+
1. Dump WebRTC stats via getStats() into db for later analysis.
106+
1. getStats() can still be called after PC has failed or has been closed.
107+
1. Continous storage WebRTC stats as time series might be challenging.
108+
We don't have a lot of experience doing it.
109+
1. Come up with custom metrics that will allow you to observe the scale of a given problem or monitor how something changes in time.
110+
1. E.g. if you feel like you very often encounter ICE failures, count them and compare to successful workflows or to the number of complete and successful SDP offer/answer exchanges.
111+
This way you will see the scale of the problem and you can observer how it changes in time, after introducing fixes or new features.
112+
1. It's important to look at numbers instead of specific cases as there will always be someone who needs to refresh the page, restart the connection etc.
113+
What matters is the ratio of such problems and how it changes in time.
114+
1. E.g. this is a quote from Sean DuBois working on WebRTC in OpenAI:
115+
> We have metrics of how many people post an offer compared to how many people get to connected [state]. It’s never alarmed on a lot of users.
116+
117+
Watch the full interview [here](https://www.youtube.com/watch?v=HVsvNGV_gg8) and read the blog [here](https://webrtchacks.com/openai-webrtc-qa-with-sean-dubois/#h).
118+
1. Collect user feedback (on a scale 1-3/1-5, via emoji) and observe how it changes.
119+
120+
## MOS
121+
122+
Initially, MOS was simply asking people about their feedback on a scale from 1 to 5 and then computing avg.
123+
Right now, we have algorithms that aim to calculate audio/video quality on the same scale but using WebRTC stats: jitter, bitrate, packet loss, resolution, codecs, freezes, etc.
124+
An example can be found here: https://github.com/livekit/rtcscore-go
125+
126+
## chrome://webrtc-internals
127+
128+
1. Based on [getStats()](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/getStats) API
129+
1. getStats() does not return derivatives.
130+
They depend on the frequency of calls to getStats() and have to be calcualted by a user.
131+
1. chrome://webrtc-internals can be dumped and then analyzed using: https://fippo.github.io/webrtc-dump-importer/
132+

‎guides/for_developers/fd_encryption.md

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# WebRTC encryption
2+
3+
In WebRTC, there are two types of data:
4+
* Media
5+
* Arbitrary Data
6+
7+
## Media
8+
9+
Media is audio or video.
10+
It's sent using RTP - a protocol that adds timestamps, sequence numbers and other information
11+
that UDP lacks but is needed for retransmissions (RTX), correct audio/video demultiplexing, sync and playout, and so on.
12+
13+
## Arbitrary data
14+
15+
By arbitrary data we mean anything that is not audio or video.
16+
This can be chat messages, signalling in game dev, files, etc.
17+
Arbitrary data is sent using SCTP - a transport protocol like UDP/TCP but with a lot of custom features.
18+
In the context of WebRTC, two of them are the most important - reliability and transmission order.
19+
They are configurable and depending on the use-case, we can send data reliably/unreliably and in-order/unordered.
20+
SCTP has not been successfully implemented in the industry.
21+
A lot of network devices don't support SCTP datagrams and are optimized for TCP traffic.
22+
Hence, in WebRTC, SCTP is encapsulated into DTLS and then into UDP.
23+
Users do not interact with SCTP directly, instead they use abstraction layer built on top of it called Data Channels.
24+
Data Channels do not add additional fields/header to the SCTP payload.
25+
26+
## Encryption
27+
28+
```mermaid
29+
flowchart TD
30+
subgraph Media - optional
31+
M(["Media"]) --> R["RTP/RTCP"]
32+
end
33+
subgraph ArbitraryData - optional
34+
A["Arbitrary Data"] --> SCTP["SCTP"]
35+
end
36+
R --> S["SRTP/SRTCP"]
37+
D["DTLS"] -- keying material --> S
38+
I["ICE"] --> U["UDP"]
39+
SCTP --> D
40+
S --> I
41+
D --> I
42+
```
43+
44+
1. Media is encapsulated into RTP packets but not into DTLS datagrams.
45+
1. In the context of media, DTLS is only used to obtain keying material that is used to create SRTP/SRTCP context.
46+
1. RTP packet **payloads** are encrypted using SRTP.
47+
1. RTP headers are not encrypted - we can see and analyze them in Wireshark without configuring encryption keys.
48+
1. DTLS datagrams, among other fields, contain 16-bit sequence number in their headers.
49+
50+
51+
## E2E Flow
52+
53+
1. Establish ICE connection
54+
2. Perform DTLS handshake
55+
3. Create SRTP/SRTCP context using keying material obtained from DTLS context
56+
4. Encapsulate media into RTP, encrypt using SRTP, and send using ICE(UDP)
57+
5. Encapsulate arbitrary data into SCTP, encrypt using DTLS and send using ICE(UDP)
58+
59+
Points 1 and 2 are mandatory, no matter we send media, arbitrary data or both.
60+
WebRTC communication is **ALWAYS** encrypted.
61+
62+
## TLS/DTLS handshake
63+
64+
See:
65+
* https://tls12.xargs.org/
66+
* https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange#
67+
* https://webrtcforthecurious.com/docs/04-securing/
68+
* https://www.ibm.com/docs/en/cloud-paks/z-modernization-stack/2023.4?topic=handshake-tls-12-protocol
69+
70+
71+
1. TLS uses asymmetric cryptography but depending on TLS version and cipher suites, it is used for different purposes.
72+
1. In TLS-RSA, we use server's public key from server's cert to encrypt pre-master secret and send it from a client to the server.
73+
Then, both sides use client random, server random, and pre-master secret to create master secret.
74+
1. In DH-TLS, server's public key from server's cert is not used to encrypt anything.
75+
Instead, both sides generate priv/pub key pairs and exchange pub keys between each other.
76+
Pub key is based on a priv key and both of them are generated per connection.
77+
They are not related to e.g. server's pub key that's included in server's cert.
78+
All params are sent unecrypted.
79+
1. Regardless of the TLS version, server's cert is used to ensure server's identity.
80+
This cert is signed by Certificate Authority (CA).
81+
CA computes hash of the certificate and encrypts it using CA's private key.
82+
The result is known as digest and is included in server's cert.
83+
Client takes cert digest and verifies it using CA public key.
84+
1. In standard TLS handshake, server MUST send its certificate to a client but
85+
client only sends its certificate when explicitly requests by the server.
86+
1. In DTLS-SRTP in WebRTC, both sides MUST send their certificates.
87+
1. In DTLS-SRTP in WebRTC, both sides generate self-signed certificates.
88+
1. Alternatively, certs can be configured when creating a peer connection: https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection#certificates
89+
1. Fingerprints of these certs are included in SDP offer/answer and are checked once DTLS-SRTP handshake is completed i.e.
90+
we take fingerprint from the SDP (which is assumed to be received via secured channel) and check it against fingerprint
91+
of the cert received during DTLS-SRTP handshake.
92+
1. The result of DTLS-SRTP handshake is master secret, which is then used to create so called keying material.
93+
94+
## Keying material
95+
96+
See:
97+
* https://datatracker.ietf.org/doc/html/rfc5705#section-4
98+
* https://datatracker.ietf.org/doc/html/rfc5764#section-4.2
99+
100+
Keying material is used to create SRTP encryption keys and is derived from a master secret established during DTLS-SRTP handshake.
101+
102+
```
103+
keying_material = PRF(master_secret, client_random + server_random + context_value_length + context_value, label)
104+
```
105+
106+
107+
* PRF is defined by TLS/DTLS
108+
* context_value and context_value_length are optional and are not used in WebRTC
109+
* label is used to allow for a single master secret to be used for many different purposes.
110+
This is because PRF gives the same output for the same input.
111+
Using exactly the same keying material in different contexts would be insecure.
112+
In WebRTC this is a string "EXTRACTOR-dtls_srtp"
113+
* length of keying material is configurable and depends on SRTP profile
114+
* keying material is divided into four parts as shown below:
115+
116+
```mermaid
117+
flowchart TD
118+
K["KeyingMaterial"] --> CM["ClientMasterKey"]
119+
K --> SM["ServertMasterKey"]
120+
K --> CS["ClientMasterSalt"]
121+
K --> SS["ServerMasterSalt"]
122+
```
123+
124+
They are then fed into SRTP KDF (key derivation function), which is another PRF (dependent on SRTP protection profile), which produces actual encryption keys.
125+
Client uses ClientMasterKey and ClientMasterSalt while server uses ServerMasterKey and ServerMasterSalt.
126+
By client and server we mean DTLS roles i.e. client is the side that inits DTLS handshake.
127+
128+
### Protection profiles
129+
130+
Some of the protection profiles:
131+
* AES128_CM_SHA1_80
132+
* AES128_CM_SHA1_32
133+
* AEAD_AES_128_GCM
134+
* AEAD_AES_256_GCM
135+
136+
Meaning:
137+
* AES128_CM - encryption algorithm (AES in counter mode) with 128-bit long key
138+
* SHA1_80 - auth function for creating 80-bit long message authentication code (MAC)
139+
* AEAD_AES_128_GCM - modified AES, both encrypts and authenticates
140+
141+
Most of the SRTP protection profiles use AES_CM as KDF.

‎guides/for_developers/fd_sdp.md

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# WebRTC SDP
2+
3+
WebRTC uses SDP offer/answer to negotiate session parameters (numer of audio/video tracks, their directions, codecs, etc.).
4+
The way they are exchanged between both sides is not standardized.
5+
Very often it is a websocket.
6+
WebRTC was standardized by (among others) Google, Cisco and Mozilla.
7+
Cisco and Mozilla insisted on compatibility with SIP and telephone industry, hence a lot of strange things in WebRTC are present to allow for WebRTC <-> SIP interoperability (e.g. SDP, DTMF).
8+
9+
## General information
10+
11+
* an mline starts with `m=` and continues until the next mline or the end of the SDP
12+
* an mline represents a transceiver or data channel
13+
* audio/video mlines have direction - sendrecv, recvonly, sendonly, inactive
14+
* an mline can be rejected - in such case, its direction is set to inactive
15+
* when transceiver is stopped, port number in mline is set to 0
16+
* port number in mline set to 9 means that connection address will be set dynamically via ICE
17+
* SDP can include ICE candidates but it doesn't have to.
18+
In particular, when you create the first offer it won't have any ICE candidates, but if you wait a couple of seconds and read peerconnection.localDescription it will contain ICE candidates that were gatherd throughout this time.
19+
* offerer can offer to both send and receive
20+
* mline includes a list of supported codecs.
21+
They are sorted in preference order
22+
* sender can switch between negotiated codecs without informing receiver about this fact.
23+
Receiver has to be prepared for receiving any payload type accepted in SDP answer.
24+
This is e.g. used to switch between audio from microphone and DTMF.
25+
* each codec has its payload type - a number that identifies it and is included in RTP packet header
26+
* fmtp stands for format parameters and denotes additional codec parameters e.g. profile level, minimal packetization length, etc.
27+
* a lot of identifiers are obsolete (ssrc, cname, trackid in msid) but some implementations still rely on them (e.g. pion requries SSRC to be present in SDP to correctly demux incoming RTP streams). See RFC 8843, 9.2 for correct RTP demuxer algorithm.
28+
* rtcp-fb is RTCP feedback supported by offerer/answerer.
29+
Example feedbacks are used to request keyframes, retransmissions or to allow for congestion control implementation.
30+
31+
## Rules
32+
33+
1. Number of mlines in SDP answer MUST be the same as in the offer.
34+
1. Number of mlines MUST NOT decrease between subsequent offer/answers.
35+
1. SDP answer can exclude codecs, rtp header extensions, and rtcp feedbacks that were offered but are not supported by the answerer
36+
37+
38+
## Dictionary
39+
40+
* SDP munging - manual SDP string modification to enable/disable some of the WebRTC features.
41+
It happens inbetween createOffer/createAnswer and setLocalDescription.
42+
E.g. when experimental support for a new codec was introduced, it could be enabled via SDP munging.
43+
44+
45+
## Negotiating bidirectional P2P connection
46+
47+
See also: https://hexdocs.pm/ex_webrtc/mastering_transceivers.html
48+
49+
When the other side is a casual peer, in most cases we want to both send and receive a single audio and video.
50+
This is the most common case.
51+
Hence, when we add audio or video track via addTrack, and create offer via createOffer,
52+
this offer will have mlines with directions set to sendrecv to allow for immediate, bidirectional session establishment.
53+
54+
When the other side is an SFU, we have at least 3 options:
55+
* server sends, via signaling channel, information to the client on how many audio and video tracks there already are in the room.
56+
Client sends SDP offer including both its own tracks and server's tracks.
57+
This requires a single negotiation.
58+
* client sends SDP offer only including its own tracks.
59+
After this is negotiated succsessfully, server sends its SDP offer.
60+
This requries two negotiations.
61+
* we use two separate peer connections, one for sending and one for receiving.
62+
This way client and server can send their offers in parallel.
63+
This was used e.g. by LiveKit.

0 commit comments

Comments
 (0)