Skip to content

Commit 049cdd1

Browse files
mickel8mickel8
committed
Add "For Developers" guides - Debugging, Encryption and SDP
1 parent e339453 commit 049cdd1

File tree

4 files changed

+277
-2
lines changed

4 files changed

+277
-2
lines changed

‎guides/for_developers/fd_debugging.md

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# WebRTC debugging
2+
3+
**It is also worth taking a look at [debugging](../advanced/debugging.md)**
4+
5+
6+
In most cases, when **something** does not work, we try to find the problem according to the following workflow:
7+
1. Check whether session has been negotiated successfully.
8+
1. Check whether connection (ICE and DTLS) has been established.
9+
1. Check whether RTP packets are demuxed, frames assembled and decoded.
10+
1. Check QoE statistics - freezes, jitter, packet loss, bitrate, fps.
11+
12+
13+
```mermaid
14+
flowchart TD
15+
S["Types of problems in WebRTC"] --> Session["Session negotiation (number of tracks, codecs)"]
16+
S --> Connection["Connection establishment (ICE and DTLS)"]
17+
S --> Playout["Playout (demuxing, packetization, decoding)"]
18+
S --> QoE["QoE (freezes, low quality, low fps)"]
19+
```
20+
21+
## Session Negotiation
22+
23+
Here, we just validate that SDP offer/answer looks the way it should.
24+
In particular:
25+
1. Check number of audio and video mlines.
26+
1. Check if any mlines are rejected (either by presence of port 0 in the m="" or a=inactive).
27+
In most cases port is set to 9 (which means automatic negotiation by ICE) or if ICE is already in progress or this is subsequent negotiation, it might be set to a port currently used by the ICE agent. Port 0 appears when someone stops transceiver via [`stop()`](https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpTransceiver/stop).
28+
1. Check mlines directions (a=sendrecv/sendonly/recvonly/inactive)
29+
1. Check codecs, their profiles and payload types.
30+
1. Number of mlines between offer and answer cannot change.
31+
This means that if one side offer that it is willing to only receive a single audio track,
32+
everything the other side can do is either confirm it will be sending or decline and say it won't be sending.
33+
If the other side also wants to send, additional negotiation has to be performed **in this case**.
34+
35+
SDP offer/answer can be easily checked in chrome://webrtc-internals (in chromium based browsers) or (about:webrtc in FF).
36+
37+
## Connection establishment
38+
39+
WebRTC connection state (PeerConnection or PC state) is a sum of ICE connection state and DTLS state.
40+
In particular, PC is in state connected when both ICE and DTLS is in state connected.
41+
42+
The whole flow looks like this:
43+
1. ICE searches for a pair of local and remote address that can be used to send and receive data.
44+
1. Once a valid pair is found, ICE changes its state to connected and DTLS handshake is started.
45+
1. Once DTLS handshake finishes, DTLS changes its state to connected and so the whole PC.
46+
1. In the meantime, ICE continues checking other pairs of local and remote address in case there is better path.
47+
If there is, ICE seamlessly switches to it - the transmission is not stopped or interrupted.
48+
49+
50+
More on ICE, its state changes, failures and restarts in section devoted to ICE.
51+
52+
In most cases, DTLS handshake works correctly. Most problems are related to ICE as it's pretty complex protocol.
53+
54+
Debugging ICE:
55+
56+
1. Check ICE candidates grid in chrome://webrtc-internals or about:webrtc
57+
1. Turn on debug logs in ex_ice or chromium (via command line argument). FF exposes all ICE logs in about:webrtc->Connection Log.
58+
Every implementation (ex_ice, chromium, ff) is very verbose.
59+
You can compare what's happening on both sides.
60+
1. Try to filter out some of the local network interfaces and remove STUN/TURN servers to reduce complexity of ICE candidate grid, amount of logs and number of connectivity checks.
61+
In ex_webrtc, this is possible via [configuration options](https://hexdocs.pm/ex_webrtc/0.14.0/ExWebRTC.PeerConnection.Configuration.html#t:options/0).
62+
1. Use Wireshark.
63+
Use filters on src/dst ip/port, udp and stun protocols.
64+
This way you can analyze whole STUN/ICE/TURN traffic between a single local and remote address.
65+
66+
Debugging DTLS:
67+
68+
This is really rare.
69+
We used Wireshark or turned on [debug logs in ex_dtls](https://hexdocs.pm/ex_dtls/0.17.0/readme.html#debugging).
70+
71+
## Playout
72+
73+
If both session negotiation and connection establishment went well, you can observe packets are flowing but nothing is visible in the web browser, the problem might be in RTP packets demuxing, frames assembly or frames decoding on the client side.
74+
75+
1. We heavily rely on chrome://webrtc-internals here.
76+
1. Check counters: packetsReceived, framesReceived, framesDecoded, framesDropped.
77+
1. E.g. if packetsReceived increases but framesReceived does not, it means that there is a problem in assembling video frames from RTP packets. This can happen when:
78+
1. web browser is not able to correctly demux incomming RTP streams possibly because sender uses incorrect payload type in RTP packets (different than the one announced in SDP) or does not include MID in RTP headers.
79+
Keep in mind that MID MAY be sent only at the beginning of the transmission to save bandwidth.
80+
This is enough to create a mapping between SSRC and MID on the receiver side.
81+
1. marker bit in RTP header is incorrectly set by the sender (although dependent on the codec, in case of video, marker bit is typically set when an RTP packet contains the end of a video frame)
82+
1. media is incorrectly packed into RTP packet payload because of bugs in RTP payloader
83+
1. E.g. if packetsReceived increases, framesReceived increases but framesDecoded does not, it probably means errors in decoding process.
84+
In this case, framesDropped will probably also increase.
85+
1. framesDropped may also increase when frames are assembled too late i.e. their playout time has passed.
86+
1. Check borwser logs.
87+
Some of the errors (e.g. decoder errors) might be logged.
88+
89+
## QoE
90+
91+
The hardest thing to debug.
92+
Mostly because it very often depends on a lot of factors (network condition, hardware, sender capabilities, mobile devices).
93+
Problems with QoE are hard to reproduce, very often don't occur in local/office environment.
94+
95+
1. We heavily rely on chrome://webrtc-internals here.
96+
1. Check coutners: nackCount, retransmittedPacketsSent, packetsLost.
97+
Retransmissions (RTX) are must have.
98+
Without RTX, even 1% of packet loss will have very big impact on QoE.
99+
1. Check incoming/outgoing bitrate and its stability.
100+
1. Check jitterBufferDelay/jitterBufferEmittedCount_in_ms - this is avg time each video frame spends in jitter buffer before being emitted for plaout.
101+
1. JitterBuffer is adjusted dynamically.
102+
103+
## Debugging in production
104+
105+
1. Dump WebRTC stats via getStats() into db for later analysis.
106+
1. getStats() can still be called after PC has failed or has been closed.
107+
1. Continous storage WebRTC stats as time series might be challenging.
108+
We don't have a lot of experience doing it.
109+
1. Come up with custom metrics that will allow you to observe the scale of a given problem or monitor how something changes in time.
110+
1. E.g. if you feel like you very often encounter ICE failures, count them and compare to successful workflows or to the number of complete and successful SDP offer/answer exchanges.
111+
This way you will see the scale of the problem and you can observer how it changes in time, after introducing fixes or new features.
112+
1. It's important to look at numbers instead of specific cases as there will always be someone who needs to refresh the page, restart the connection etc.
113+
What matters is the ratio of such problems and how it changes in time.
114+
1. E.g. this is a quote from Sean DuBois working on WebRTC in OpenAI:
115+
> We have metrics of how many people post an offer compared to how many people get to connected [state]. It’s never alarmed on a lot of users.
116+
117+
Watch the full interview [here](https://www.youtube.com/watch?v=HVsvNGV_gg8) and read the blog [here](https://webrtchacks.com/openai-webrtc-qa-with-sean-dubois/#h).
118+
1. Collect user feedback (on a scale 1-3/1-5, via emoji) and observe how it changes.
119+
120+
## MOS
121+
122+
Initially, MOS was simply asking people about their feedback on a scale from 1 to 5 and then computing avg.
123+
Right now, we have algorithms that aim to calculate audio/video quality on the same scale but using WebRTC stats: jitter, bitrate, packet loss, resolution, codecs, freezes, etc.
124+
An example can be found here: https://github.com/livekit/rtcscore-go
125+
126+
## chrome://webrtc-internals
127+
128+
1. Based on [getStats()](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/getStats) API
129+
1. getStats() does not return derivatives.
130+
They depend on the frequency of calls to getStats() and have to be calcualted by a user.
131+
1. chrome://webrtc-internals can be dumped and then analyzed using: https://fippo.github.io/webrtc-dump-importer/
132+

‎guides/for_developers/fd_encryption.md

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# WebRTC encryption
2+
3+
In WebRTC, there are two types of data:
4+
* Media
5+
* Arbitrary Data
6+
7+
## Media
8+
9+
Media is audio or video.
10+
It's sent using RTP - a protocol that adds timestamps, sequence numbers and other information
11+
that UDP lacks but is needed for retransmissions (RTX), correct audio/video demultiplexing, sync and playout, and so on.
12+
13+
## Arbitrary data
14+
15+
By arbitrary data we mean anything that is not audio or video.
16+
This can be chat messages, signalling in game dev, files, etc.
17+
Arbitrary data is sent using SCTP - a transport protocol like UDP/TCP but with a lot of custom features.
18+
In the context of WebRTC, two of them are the most important - reliability and transmission order.
19+
They are configurable and depending on the use-case, we can send data reliably/unreliably and in-order/unordered.
20+
SCTP has not been successfully implemented in the industry.
21+
A lot of network devices don't support SCTP datagrams and are optimized for TCP traffic.
22+
Hence, in WebRTC, SCTP is encapsulated into DTLS and then into UDP.
23+
Users do not interact with SCTP directly, instead they use abstraction layer built on top of it called Data Channels.
24+
Data Channels do not add additional fields/header to the SCTP payload.
25+
26+
## Encryption
27+
28+
```mermaid
29+
flowchart TD
30+
subgraph Media - optional
31+
M(["Media"]) --> R["RTP/RTCP"]
32+
end
33+
subgraph ArbitraryData - optional
34+
A["Arbitrary Data"] --> SCTP["SCTP"]
35+
end
36+
R --> S["SRTP/SRTCP"]
37+
D["DTLS"] -- keying material --> S
38+
I["ICE"] --> U["UDP"]
39+
SCTP --> D
40+
S --> I
41+
D --> I
42+
```
43+
44+
1. Media is encapsulated into RTP packets but not into DTLS datagrams.
45+
1. In the context of media, DTLS is only used to obtain keying material that is used to create SRTP/SRTCP context.
46+
1. RTP packet **payloads** are encrypted using SRTP.
47+
1. RTP headers are not encrypted - we can see and analyze them in Wireshark without configuring encryption keys.
48+
1. DTLS datagrams, among other fields, contain 16-bit sequence number in their headers.
49+
50+
51+
## E2E Flow
52+
53+
1. Establish ICE connection
54+
2. Perform DTLS handshake
55+
3. Create SRTP/SRTCP context using keying material obtained from DTLS context
56+
4. Encapsulate media into RTP, encrypt using SRTP, and send using ICE(UDP)
57+
5. Encapsulate arbitrary data into SCTP, encrypt using DTLS and send using ICE(UDP)
58+
59+
Points 1 and 2 are mandatory, no matter we send media, arbitrary data or both.
60+
WebRTC communication is **ALWAYS** encrypted.
61+
62+
## TLS/DTLS handshake
63+
64+
See:
65+
* https://tls12.xargs.org/
66+
* https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange#
67+
* https://webrtcforthecurious.com/docs/04-securing/
68+
* https://www.ibm.com/docs/en/cloud-paks/z-modernization-stack/2023.4?topic=handshake-tls-12-protocol
69+
70+
71+
1. TLS uses asymmetric cryptography but depending on TLS version and cipher suites, it is used for different purposes.
72+
1. In TLS-RSA, we use server's public key from server's cert to encrypt pre-master secret and send it from a client to the server.
73+
Then, both sides use client random, server random, and pre-master secret to create master secret.
74+
1. In DH-TLS, server's public key from server's cert is not used to encrypt anything.
75+
Instead, both sides generate priv/pub key pairs and exchange pub keys between each other.
76+
Pub key is based on a priv key and both of them are generated per connection.
77+
They are not related to e.g. server's pub key that's included in server's cert.
78+
All params are sent unecrypted.
79+
1. Regardless of the TLS version, server's cert is used to ensure server's identity.
80+
This cert is signed by Certificate Authority (CA).
81+
CA computes hash of the certificate and encrypts it using CA's private key.
82+
The result is known as digest and is included in server's cert.
83+
Client takes cert digest and verifies it using CA public key.
84+
1. In standard TLS handshake, server MUST send its certificate to a client but
85+
client only sends its certificate when explicitly requests by the server.
86+
1. In DTLS-SRTP in WebRTC, both sides MUST send their certificates.
87+
1. In DTLS-SRTP in WebRTC, both sides generate self-signed certificates.
88+
1. Alternatively, certs can be configured when creating a peer connection: https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/RTCPeerConnection#certificates
89+
1. Fingerprints of these certs are included in SDP offer/answer and are checked once DTLS-SRTP handshake is completed i.e.
90+
we take fingerprint from the SDP (which is assumed to be received via secured channel) and check it against fingerprint
91+
of the cert received during DTLS-SRTP handshake.
92+
1. The result of DTLS-SRTP handshake is master secret, which is then used to create so called keying material.
93+
94+
## Keying material
95+
96+
See:
97+
* https://datatracker.ietf.org/doc/html/rfc5705#section-4
98+
* https://datatracker.ietf.org/doc/html/rfc5764#section-4.2
99+
100+
Keying material is used to create SRTP encryption keys and is derived from a master secret established during DTLS-SRTP handshake.
101+
102+
```
103+
keying_material = PRF(master_secret, client_random + server_random + context_value_length + context_value, label)
104+
```
105+
106+
107+
* PRF is defined by TLS/DTLS
108+
* context_value and context_value_length are optional and are not used in WebRTC
109+
* label is used to allow for a single master secret to be used for many different purposes.
110+
This is because PRF gives the same output for the same input.
111+
Using exactly the same keying material in different contexts would be insecure.
112+
In WebRTC this is a string "EXTRACTOR-dtls_srtp"
113+
* length of keying material is configurable and depends on SRTP profile
114+
* keying material is divided into four parts as shown below:
115+
116+
```mermaid
117+
flowchart TD
118+
K["KeyingMaterial"] --> CM["ClientMasterKey"]
119+
K --> SM["ServertMasterKey"]
120+
K --> CS["ClientMasterSalt"]
121+
K --> SS["ServerMasterSalt"]
122+
```
123+
124+
They are then fed into SRTP KDF (key derivation function), which is another PRF (dependent on SRTP protection profile), which produces actual encryption keys.
125+
Client uses ClientMasterKey and ClientMasterSalt while server uses ServerMasterKey and ServerMasterSalt.
126+
By client and server we mean DTLS roles i.e. client is the side that inits DTLS handshake.
127+
128+
### Protection profiles
129+
130+
Some of the protection profiles:
131+
* AES128_CM_SHA1_80
132+
* AES128_CM_SHA1_32
133+
* AEAD_AES_128_GCM
134+
* AEAD_AES_256_GCM
135+
136+
Meaning:
137+
* AES128_CM - encryption algorithm (AES in counter mode) with 128-bit long key
138+
* SHA1_80 - auth function for creating 80-bit long message authentication code (MAC)
139+
* AEAD_AES_128_GCM - modified AES, both encrypts and authenticates

‎guides/for_developers/fd_sdp.md

Whitespace-only changes.

‎mix.exs

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,14 +85,17 @@ defmodule ExWebRTC.MixProject do
8585

8686
deploying_guides = ["bare", "fly"]
8787

88+
for_developers_guides = ["fd_encryption", "fd_debugging"]
89+
8890
[
8991
main: "readme",
9092
logo: "logo.svg",
9193
extras:
9294
["README.md"] ++
9395
Enum.map(intro_guides, &"guides/introduction/#{&1}.md") ++
9496
Enum.map(advanced_guides, &"guides/advanced/#{&1}.md") ++
95-
Enum.map(deploying_guides, &"guides/deploying/#{&1}.md"),
97+
Enum.map(deploying_guides, &"guides/deploying/#{&1}.md") ++
98+
Enum.map(for_developers_guides, &"guides/for_developers/#{&1}.md"),
9699
assets: %{"guides/assets" => "assets"},
97100
source_ref: "v#{@version}",
98101
formatters: ["html"],
@@ -101,7 +104,8 @@ defmodule ExWebRTC.MixProject do
101104
groups_for_extras: [
102105
Introduction: Path.wildcard("guides/introduction/*.md"),
103106
Advanced: Path.wildcard("guides/advanced/*.md"),
104-
Deploying: Path.wildcard("guides/deploying/*.md")
107+
Deploying: Path.wildcard("guides/deploying/*.md"),
108+
"For Developers": Path.wildcard("guides/for_developers/*.md")
105109
],
106110
groups_for_modules: [
107111
MEDIA: ~r"ExWebRTC\.Media\..*",

0 commit comments

Comments
 (0)