SSI Troubleshooting Updates

content/en/tracing/trace_collection/automatic_instrumentation/single-step-apm/troubleshooting.md

@@ -9,19 +9,150 @@
 ## Overview
 
 Single Step Instrumentation (SSI) helps instrument applications by automatically loading application processes with the Datadog APM SDKs. SSI works for applications running on Linux hosts, in container environments such as Kubernetes and Docker, and for .NET applications served by Windows IIS—without requiring changes to application dependencies or images. If you encounter issues enabling APM with SSI, use this guide to troubleshoot and resolve common problems. For further assistance, contact [Datadog Support][1].
-## Host and Docker instrumentation
 
-### Host injection does not apply to existing processes
+## Troubleshooting methods
+
+You can troubleshoot injection issues in two ways: by using Fleet Automation in Datadog or by manually verifying at the container level.
+
+### Troubleshoot injection in Datadog Fleet Automation
+
+{{< callout btn_hidden="true" header="false">}}
+Instrumentation insights in Fleet Automation is available in Preview.
+{{< /callout >}}
+
+Using Datadog, you can identify and troubleshoot instrumentation issues across your infrastructure. You can see information like:
+- Which hosts have SSI enabled
+- Which processes failed instrumentation and why
+- Which services were instrumented successfully
+- Process-level details to help debug injection failures
+
+#### Prerequisites
+
+Instrumentation insights are available for:
+
+- **Languages**: Python, Java, Node.js, Ruby, PHP, .NET
+- **Environments**: Linux hosts, containers, Kubernetes
+- Datadog Agent v7.68.2+
+
+#### View instrumentation insights 
+
+To access instrumentation insights in Datadog:
+
+1. Navigate to [Fleet Automation][9].
+1. Use facets to filter down to relevant hosts:
+   - `single_step_instrumentation` shows which hosts have SSI enabled or disabled.
+   - `single_step_instrumentation_status` shows which hosts have encountered issues with service instrumentation.
+1. Select a host to open the Agent details panel.
+1. In the Agent panel, go to the **Services** tab.
+1. If SSI is enabled on the host, the tab shows:
+   - A banner with the message: "Single Step Instrumentation is enabled on this host."
+   - An **SDK Installations** section if there are issues to troubleshoot.
+
+### Manually verify injection in the application container
+
+If the Datadog UI does not show any instrumentation issues, or if you're troubleshooting a single service or container, you can manually verify whether injection occurred as expected. This method is helpful when debugging in environments where centralized visibility is limited or when a specific service isn't reporting traces.
+
+To confirm injection at the container level, check that:
+
+1. `/etc/ld.so.preload` includes the following entry: 
+   ```
+   /opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so
+   ``` 
+2. The `LD_PRELOAD` environment variable is set to the same value.
+3. The directory `/opt/datadog-packages/datadog-apm-inject` exists, with `stable` and `$version` subdirectories.
+4. Language-specific directories exist (for example, `/opt/datadog/apm/library/java/` for Java).
+
+To enable debug logs:
+1. Set `DD_APM_INSTRUMENTATION_DEBUG: true` in the pod spec.
+2. Delete the pod to enable debug logs during injection.
+
+## Configuration options that affect injection
+
+There are several configuration mechanisms that can block or alter injection behavior.
+
+### Injector version
+
+To set the injector version:
+- At the cluster level:
+
+  Set in `values.yaml` under `datadog.apm.instrumentation.injector.imageTag`.
+
+- At the pod level:
+
+  Set with the annotation `admission.datadoghq.com/apm-inject.version`.
+
+
+For host or Docker injection, modifying the `auto_inject` version is not recommended. 
+
+To manually update the injector version, run:
+```
+sudo datadog-installer install "oci://install.datadoghq.com/apm-inject-package: <VERSION>-1"
+```
+
+Alternatively, set the following in the installation script:
+```
+DD_INSTALLER_DEFAULT_PKG_VERSION_DATADOG_APM_INJECT=<VERSION>-1
+```
+
+### Allow and deny lists
+
+#### Default deny list
+
+Datadog maintains an internal deny list to prevent injection into certain processes (for example, IDEs or databases). If a process command or entrypoint is on this list, the injector skips the injection process.  
+
+#### Linux workload selection
+
+{{< callout url="https://docs.google.com/forms/d/e/1FAIpQLSdMu6WAsUCD3djkl_oN0Qh7fQmBCiKYyUvuqlYWRyObebAc6Q/viewform" header="Join the Preview!">}}
+Workload selection is available for Linux-based apps through a limited availability preview. To configure allow or deny rules for process injection, sign up for preview access.
+{{< /callout >}}
+
+#### Kubernetes workload selection  
+
+Workload selection enables injection based on Kubernetes labels and selectors. Rules to consider:
+
+1. `disabledNamespaces` always takes precedence.
+2. When a pod initializes, the target list is checked from top to bottom. Only the first matching rule applies per pod.
+
+#### Injection container flagged by security scanners
+
+Security tools may flag the `apm-inject` container because it runs an executable at startup, which can resemble malicious software. 
+
+The container's behavior is expected and safe; the executable configures the environment for auto-instrumentation.
+
+Datadog adheres to security best practices and is working with security vendors to allowlist this container.
+
+### Environments with strict pod security settings
+
+If pod security rules block the Datadog init container, you may see errors like:
+```
+Privilege escalation container is not allowed or violates PodSecurity "restricted: latest": allowPrivilegeEscalation is false
+```
+
+To resolve this, set one of the following Cluster Agent options:
+- `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_INIT_SECURITY_CONTEXT`
+- `admission_controller.auto_instrumentation.init_security_context`
+
+The value should be a JSON string that applies the necessary security context to the Datadog init containers.  
+
+### Custom instrumentation
+
+Custom instrumentation still requires you to import the tracing library. Configuration variables like .NET's `DD_TRACE_METHODS` remain available for defining custom spans.
+
+## Environment-specific troubleshooting
+
+### Host and Docker environments
+
+#### Host injection does not apply to existing processes
 
 The preload library only injects into newly launched processes. Start a new shell session or log out and log back in to apply instrumentation.
 
 **Note:** Docker-based injection does not have this limitation.
 
-### Injection fails on small instance types
+#### Injection fails on small instance types
 
 The preload library allows the analyzer one second to complete its work. On small VM instances running multiple services (for example, `t2.micro`), this time limit may be exceeded. To overcome this issue, use a larger instance size, such as `t2.small`.
 
-### Errors after manual uninstallation of agent files
+#### Errors after manual uninstallation of agent files
 
 If you manually delete agent files, you may see errors like:
 ```
@@ -33,23 +164,21 @@
 * [Kubernetes][1]
 * [Docker][2]
 * [Linux][3]
-* [Windows][3]
+* [Windows][4]
 
-### Injection not working with rootless Docker
+#### Injection not working with rootless Docker
 
 When using rootless Docker, set `docker_socket` in `/etc/datadog-agent/inject/docker_config.yaml` to the path of the Docker socket used by the current user (typically `/run/user/$UID/docker.sock`). No reboot is required.
 
-### Injection fails with statically linked launchers
+#### Injection fails with statically linked launchers
 
 If a custom launcher is statically linked (common with Go), the preload library might not be invoked. Injection can still succeed if:
 - The launcher's command line includes the language name
 - The launcher runs an intermediary dynamically linked program
 
 However, direct process launches from statically linked binaries are not injected.
 
-## Kubernetes instrumentation
-
-### Ensure the Admission Controller is configured and running
+### Kubernetes environments
 
 The Datadog Admission Controller must be deployed and configured before application pods are created; it cannot modify existing pods.
 
@@ -84,13 +213,13 @@
 
    Look for `admission_webhooks_library_injection_attempts` to see injection attempts by language.
 
-### Failed mutations
+#### Failed mutations
 
 The Cluster Agent logs warnings and errors for injection failures, typically from `admission/server.go`. For example, a warning might appear if `JAVA_TOOL_OPTIONS` is set using `valueFrom`.
 
 Use the metric `datadog.cluster_agent.admission_webhooks.library_injection_errors` for further debugging.
 
-#### Language annotation cannot be applied
+##### Language annotation cannot be applied
 
 During setup, SSI detects the application language of your service and applies a service label in the form `internal.dd.datadoghq.com/service-name.detected_langs`. If the label cannot be applied, injection fails.
 
@@ -101,93 +230,13 @@
 
 String limit violations are common if service tags are not explicitly set through [Unified Service Tagging][8], in which case default image names are used.
 
-#### Injection appears successful but traces are missing
+##### Injection appears successful but traces are missing
 
 If logs show no issues but traces are missing, there may be an application-side misconfiguration. Verify that:
 - Required annotations and labels are present.
 - [Unified Service Tagging][8] is set up correctly.
 - Allow/deny lists for workload selection are properly defined.
 
-## Debug the injection process
-
-After verifying webhook injection, verify the following in the application container:
-
-1. `/etc/ld.so.preload` includes the following entry:
-   ```
-   /opt/datadog-packages/datadog-apm-inject/stable/inject/launcher.preload.so
-   ```
-2. The `LD_PRELOAD` environment variable is set to the same value.
-3. The directory `/opt/datadog-packages/datadog-apm-inject` exists, with `stable` and `$version` subdirectories.
-4. Language-specific directories exist (for example, `/opt/datadog/apm/library/java/` for Java).
-
-To enable debug logs:
-1. Set `DD_APM_INSTRUMENTATION_DEBUG: true` in the pod spec.
-2. Delete the pod to enable debug logs during injection.
-
-## Environments with strict Pod Security settings
-
-If Pod Security Rules block the Datadog init container, you may see errors like:
-```
-Privilege escalation container is not allowed or violates PodSecurity "restricted: latest": allowPrivilegeEscalation is false
-```
-
-To resolve this, set one of the following Cluster Agent options:
-- `DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_INIT_SECURITY_CONTEXT`
-- `admission_controller.auto_instrumentation.init_security_context`
-
-The value should be a JSON string that applies the necessary security context to the Datadog init containers.
-
-## Upgrade or downgrade the injector version
-
-To set the injector version:
-- At the cluster level:
-
-  Set in `values.yaml` under `datadog.apm.instrumentation.injector.imageTag`.
-
-- At the pod level:
-
-  Set with the annotation `admission.datadoghq.com/apm-inject.version`.
-
-
-For host or Docker injection, modifying the `auto_inject` version is not recommended.
-
-To manually update the injector version, run:
-```
-sudo datadog-installer install "oci://install.datadoghq.com/apm-inject-package: <VERSION>-1"
-```
-
-Alternatively, set the following in the installation script:
-```
-DD_INSTALLER_DEFAULT_PKG_VERSION_DATADOG_APM_INJECT=<VERSION>-1
-```
-
-## Process deny lists
-
-### Default deny list
-
-Datadog maintains an internal deny list to prevent injection into certain processes (for example, IDEs or databases). If a process command or entrypoint is on this list, the injector exits the injection process.
-
-### Custom deny list entries (Linux only)
-
-{{< callout url="https://docs.google.com/forms/d/e/1FAIpQLSdMu6WAsUCD3djkl_oN0Qh7fQmBCiKYyUvuqlYWRyObebAc6Q/viewform" header="Join the Preview!">}}
-Workload selection is available for Linux-based apps through a limited availability preview. To configure allow or deny rules for process injection, sign up for preview access.
-{{< /callout >}}
-
-### Injection container flagged by security scanners
-
-Security tools may flag the `apm-inject` container because it runs an executable at startup, which can resemble malicious software.
-
-The container's behavior is expected and safe; the executable configures the environment for auto-instrumentation.
-
-Datadog adheres to security best practices and is working with security vendors to allow this container.
-
-### Kubernetes workload selection issues
-
-Workload selection enables injection based on Kubernetes labels and selectors. Rules to consider:
-
-1. `disabledNamespaces` always takes precedence.
-2. When a pod initializes, the target list is checked from top to bottom. Only the first matching rule applies per pod.
-
 ## Language-specific troubleshooting
 
 ### Java
@@ -225,54 +274,8 @@
 ### Python
 
 Versions <=2.7.5 contain a pre-packaged protobuf dependency that can conflict with system libraries.
-
-## Custom instrumentation
-
-Custom instrumentation still requires you to import the tracing library. Configuration variables like .NET's `DD_TRACE_METHODS` remain available for defining custom spans.
-
-## Use SSI with private registries
-
-To use SSI with a private container registry, follow the instructions in [Synchronize Datadog's images with a private registry][5].
-
-The required images vary depending on the language(s) you configure, but typically include:
-- `gcr.io/datadoghq/apm-inject`
-- `gcr.io/datadoghq/dd-lib-<lang>-init`
-  - `gcr.io/datadoghq/dd-lib-dotnet-init`
-  - `gcr.io/datadoghq/dd-lib-java-init`
-  - `gcr.io/datadoghq/dd-lib-js-init`
-  - `gcr.io/datadoghq/dd-lib-python-init`
-  - `gcr.io/datadoghq/dd-lib-ruby-init`
-
-### Version mirroring for private registries
-
-Each language-specific tracer image must be available in your private registry at the version specified during workload selection.
-
-By default, the injector uses version `0` unless a different version is configured in the Cluster Agent. If you are using annotations to specify tracer versions, those versions must also be mirrored and available in your private registry.
-
-For example:
-
-    apm:
-      instrumentation:
-        enabled: true
-        targets:
-          - name: "default-target"
-            ddTraceVersions:
-              java: "1"
-              python: "3"
-
-This configuration would require the following image tags to exist in the private registry:
-
-* `gcr.io/datadoghq/apm-inject:0`
-* `gcr.io/datadoghq/dd-lib-java-init:1`
-* `gcr.io/datadoghq/dd-lib-python-init:3`
-
-If these images are not mirrored in your private registry, you may run into an error like the following:
-
-```
-Failed to pull image "...." rpc error: code = Unknown
-```
-
-## Asking for support
+
+## Collect diagnostic information for support
 
 When contacting support about injection issues, collect the following information to assist troubleshooting:
 
@@ -333,3 +336,4 @@
 [6]: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set
 [7]: https://datatracker.ietf.org/doc/html/rfc1035
 [8]: /getting_started/tagging/unified_service_tagging/
+[9]: https://app.datadoghq.com/fleet