.NET Observability

docusaurus/docs/guides/dotnet/aws-native/logs.md

@@ -0,0 +1,127 @@
+# Logs
+
+Logs provide rich, contextual information about events in your application.  They are very essential for debugging and understanding why something went wrong. 
+
+This section provides recipes that can be used as best practices for emitting and sending logs from .NET applications to the AWS' native log service - Amazon CloudWatch Logs.
+
+### Stream logs from log files on Amazon EC2 instances or on-premises servers to Amazon CloudWatch Logs
+
+You can use this approach when your existing .NET applications write logs to log files and you want to use the Amazon CloudWatch logs for log storage and analysis without any changes to your code.
+
+**Step 1:** Install the CW Agent on the Amazon EC2 instance or the On-premises server where your application runs. Instructions for installing the CW Agent can be found [**here**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance.html).
+
+**Step 2:** Next, we need to provide permissions to the CloudWatch agent to write logs to CloudWatch. You create an IAM role, an IAM user, or both to grant permissions that the CloudWatch agent needs to write logs to CloudWatch. If you're going to use the agent on Amazon EC2 instances, you must create an IAM role. If you're going to use the agent on on-premises servers, you must create an IAM user. **CloudWatchAgentServerPolicy** is an AWS managed IAM Policy that includes necessary permissions to write logs to CloudWatch.
+
+[**Follow these instructions**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent-commandline.html) to provide the permissions to the CW Agent.
+
+**Step 3:** Before running the CloudWatch agent on any servers, you must create one or more CloudWatch agent configuration files. The agent configuration file is a JSON file that specifies the metrics, logs, and traces that the agent is to collect and where to send them (like which log group or namespace in CloudWatch). You can create it by [**using a wizard**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file-wizard.html) or by [**creating it yourself**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html) from scratch.
+
+The agent configuration file has four sections: agent, metrics, logs, and traces. You can provide the credentails you created previously (step 2) in the **agent** section. The **logs** section specifies what log files are published to CloudWatch Logs. This can include events from the Windows Event Log if the server runs Windows Server. Detailed instructions to configure the **agent** and the **logs** sections can be found [**here**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html).
+
+**Step 4:** Once you have all of the above things in place, you can [**start the CloudWatch agent**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-premise.html#start-CloudWatch-Agent-on-premise-SSM-onprem). 
+
+### Use AWS SDK for .NET to write logs from .NET applications to CloudWatch Logs
+
+If you want to write logs directly to Amazon CloudWatch Logs using the service's APIs, you can do so by using the AWS SDK for .NET. 
+
+[**AWS SDK for .NET**](https://docs.aws.amazon.com/sdk-for-net/) provides libraries that make it easier to develop .NET applications that interact with AWS Services. The libraries are provided in the form of NuGet packages.
+
+To interact with AmazonCloudWatch Logs, you'll need to use the AmazonCloudWatchLogsClient class provided by the AWSSDK.CloudWatchLogs NuGet package.
+
+**Step 1:** Install the AWS CloudWatch Logs NuGet package
+
+```csharp
+dotnet add package AWSSDK.CloudWatchLogs
+```
+
+**Step 2:** Set up AWS credentials
+
+Make sure your app has the necessary permissions to write to Cloud Watch Logs. Either through assigning an IAM role, using an AWS credentials file, or setting up environment variables. For example, the below policy provides permissions to create log groups and log streams as well as writing logs to them.
+
+```json
+{
+    "Effect": "Allow",
+    "Action": [
+        "logs:CreateLogGroup",
+        "logs:CreateLogStream",
+        "logs:PutLogEvents"
+    ],
+    "Resource": "*"
+}
+```
+
+**Step 3:** Initialize the CloudWatchLogs Client. The below namespaces and classes are part of the AWSSDK.CloudWatchLogs NuGet package.
+
+```csharp
+using Amazon.CloudWatchLogs;
+using Amazon.CloudWatchLogs.Model;
+
+var client = new AmazonCloudWatchLogsClient();
+```
+
+**Step 4:** Create a Log Group and Log Stream if needed. You can read more about these Amazon CloudWatch Logs concepts like Log groups and Log streams [**here**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatchLogsConcepts.html)
+
+```csharp
+string logGroupName = "MyAppLogGroup";
+string logStreamName = "MyAppLogStream";
+
+// Create Log Group (skip if already exists)
+try
+{
+    await client.CreateLogGroupAsync(new CreateLogGroupRequest
+    {
+        LogGroupName = logGroupName
+    });
+}
+catch (ResourceAlreadyExistsException) { }
+
+// Create Log Stream (skip if already exists)
+try
+{
+    await client.CreateLogStreamAsync(new CreateLogStreamRequest
+    {
+        LogGroupName = logGroupName,
+        LogStreamName = logStreamName
+    });
+}
+catch (ResourceAlreadyExistsException) { }
+```
+
+**Step 5:** Send your logs to Amazon CloudWatch Logs
+
+```csharp
+var logEvents = new List<InputLogEvent>
+{
+    new InputLogEvent
+    {
+        Message = "Hello CloudWatch from .NET!",
+        Timestamp = DateTime.UtcNow
+    }
+};
+
+var putLogRequest = new PutLogEventsRequest
+{
+    LogGroupName = logGroupName,
+    LogStreamName = logStreamName,
+    LogEvents = logEvents
+};
+
+await client.PutLogEventsAsync(putLogRequest);
+```
+
+### Use popular .NET logging frameworks to send structured logs to Amazon CloudWatch Logs
+
+Although using the AmazonCloudWatchLogsClient provides a lot of flexibility and low-level API access to CloudWatch Logs, it results in a significant amount of boilerplate code. Additionally, there are several popular third-party logging frameworks among the .NET developer community for structured logging that the AmazonCloudWatchLogsClient does not integrate with out-of-the-box.
+
+The [**aws-logging-dotnet**](https://github.com/aws/aws-logging-dotnet) repository contains plugins for many of these popular logging framework providers to integrate with Amazon CloudWatch Logs. The [**repository**](https://github.com/aws/aws-logging-dotnet) contains detailed information on how to wire up your application that uses the standard ASP.NET ILogger framework, NLog, Apache log4net, or Serilog to send logs to AmazonCloudWatch Logs.
+
+### Logging in AWS Lambda functions
+
+https://aws.amazon.com/blogs/compute/introducing-advanced-logging-controls-for-aws-lambda-functions/
+
+https://aws.amazon.com/blogs/developer/structured-logging-for-net-lambda/
+
+
+### PowerTools for Lambda
+
+https://docs.powertools.aws.dev/lambda/dotnet/core/logging/#using-aws-lambda-advanced-logging-controls-alc

docusaurus/docs/guides/dotnet/aws-native/metrics.md

@@ -0,0 +1,121 @@
+# Metrics
+
+Metrics are essential in Observability because they provide quantitative data about system performance and behavior. This enables trend analysis and supports proactive monitoring to detect issues before they impact users.
+
+To learn about Metrics in general and the features of Amazon CloudWatch for Metric collection and analysis visit [**Metrics in Amazon CloudWatch**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html)
+
+[**While many AWS services have the ability to publish infrastructure metrics out-of-the-box to Amazon CloudWatch**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-metrics-basic-detailed.html), this section will focus on capturing custom metrics from .NET applications and transporting them to Amazon CloudWatch metric monitoring systems for analysis.
+
+### Use CloudWatch PutMetricData API call through AWS SDK for .NET
+
+Include the Amazon.CloudWatch and Amazon.CloudWatch.Model NuGet packages in your code.
+
+```csharp
+using Amazon.CloudWatch;
+using Amazon.CloudWatch.Model;
+```
+
+Build the PutMetricDataRequest object that contains the namespace, metric name and value, dimensions and dimension values.
+
+```csharp
+var request = new PutMetricDataRequest
+{
+    Namespace = namespaceName,
+    MetricData = new List<MetricDatum>
+    {
+        new MetricDatum
+        {
+            MetricName = metricName,
+            Dimensions = new List<Dimension>
+            {
+                new Dimension
+                {
+                    Name = dimensionName,
+                    Value = dimensionValue
+                }
+            },
+            Value = metricValue
+        }
+    }
+};
+```
+
+Send the metric data to Amazon CloudWatch by using the PutMetricData API call.
+
+```csharp
+using var client = new AmazonCloudWatchClient();
+await client.PutMetricDataAsync(request);
+```
+
+### CloudWatch embedded metric format
+
+The [**CloudWatch embedded metric format (EMF)**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format.html) lets you create custom metrics asynchronously by writing logs to CloudWatch Logs. This approach allows you to:
+
+* Embed custom metrics alongside detailed log event data
+* Have CloudWatch automatically extract these metrics for visualization and alarming
+* Enable real-time incident detection
+* Query the associated detailed log events using CloudWatch Logs Insights
+* Gain deep insights into the root causes of operational events
+
+#### Use cases for using EMF
+
+* Generate custom metrics across compute environments
+
+  * Easily generate custom metrics from Lambda functions without requiring custom batching code, making blocking network requests or relying on 3rd party software. Other compute environments (EC2, On-prem, ECS, EKS, and other container environments) are supported by installing the [**CloudWatch Agent**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Generation_CloudWatch_Agent.html).
+
+* Linking metrics to high cardinality context
+
+    * Using the Embedded Metric Format, you will be able to visualize and alarm on custom metrics, but also retain the original, detailed and high-cardinality context which is queryable using [**CloudWatch Logs Insights**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html). For example, the library automatically injects environment metadata such as Lambda Function version, EC2 instance and image ids into the structured log event data.
+
+The [**aws-embedded-metrics-dotnet opensource repository**](https://github.com/awslabs/aws-embedded-metrics-dotnet) has everything you need to get started. 
+
+#### Installation
+
+Include the Amazon.CloudWatch.EMF NuGet package in your code
+
+```csharp
+using Amazon.CloudWatch.EMF
+```
+
+You can instantiate a MetricsLogger that implements IDisposable and use it as shown below. The metrics will be flushed to the configured sink when the logger is disposed.
+
+#### Usage
+
+```csharp
+using (var logger = new MetricsLogger()) {
+    logger.SetNamespace("Canary");
+    var dimensionSet = new DimensionSet();
+    dimensionSet.AddDimension("Service", "aggregator");
+    logger.SetDimensions(dimensionSet);
+    logger.PutMetric("ProcessingLatency", 100, Unit.MILLISECONDS,StorageResolution.STANDARD);
+    logger.PutMetric("Memory.HeapUsed", "1600424.0", Unit.BYTES, StorageResolution.HIGH);
+    logger.PutProperty("RequestId", "422b1569-16f6-4a03-b8f0-fe3fd9b100f8");
+
+}
+```
+#### ASP.NET Core
+
+We provide a helper package that helps with onboarding and provides default metrics for [**ASP.NET Core applications**](https://github.com/awslabs/aws-embedded-metrics-dotnet).
+
+1. Add the configuration to your Startup file.
+
+```csharp
+public void ConfigureServices(IServiceCollection services) {
+    // Add the necessary services. After this is done, you will have the
+    // IMetricsLogger available for dependency injection in your
+    // controllers
+    services.AddEmf();
+}
+```
+
+2. Add middleware to add default metrics and metadata to each request
+
+```csharp
+public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
+{
+    // Add middleware which will set metric dimensions based on the request routing
+    app.UseEmfMiddleware();
+}
+```
+
+In any environment, other than AWS Lambda, we recommend running an out-of-process agent (the [**CloudWatch Agent**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Generation_CloudWatch_Agent.html) or FireLens / Fluent-Bit) to collect the EMF events. When using an out-of-process agent, this package will buffer the data asynchronously in process to handle any transient communication issues with the agent. 

docusaurus/docs/guides/dotnet/aws-native/traces.md

@@ -0,0 +1,37 @@
+# Traces
+
+Traces track the request processing through complex distributed systems providing detailed information about the request flow through individual components including downstream AWS resources, microservices, databases, and APIs. This will help with performance optimization by identifying bottlenecks and latency issues.
+
+In this section, you will see links to AWS Documention and open source repositories that will provide information on using AWS X-Ray SDK for .NET to instrument .NET applications to create and send trace information to AWS X-Ray via X-Ray daemon.
+
+To learn about AWS X-Ray it's core concepts visit the [**What is AWS X-Ray**](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html) and [**Concepts**](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html) sections in the AWS X-Ray Developer Guide.
+
+The X-Ray SDK for .NET is a library for instrumenting C# .NET web applications, .NET Core web applications, and .NET Core functions on AWS Lambda. It provides classes and methods for generating and sending trace data to the X-Ray daemon. This includes information about incoming requests served by the application, and calls that the application makes to downstream AWS services, HTTP web APIs, and SQL databases.
+
+## Options for agents and SDKs
+
+You have the option to choose between AWS X-Ray daemon, Cloudwatch agent, and AWS Distro for OpenTelemetry (ADOT) collector to collect traces from Amazon EC2 instances and on-premise servers and send them to AWS X-Ray. Choose the right one for your usecase so you minimize the number of agents you have to manage. 
+
+To learn about configuring X-Ray daemon to collect and send traces from your application and infrastructure read the [**AWS X-Ray daemon**](https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon.html) guide. Instead, if your choice is to use CloudWatch agent, the [**Amazon CloudWatch user guide**](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html) will provide the instructions to setup and configure CloudWatch agent.
+
+To instrument your application to generate traces, you have the option to choose between OpenTelemetry and X-Ray SDK for .NET. The guidance to choose between these options is available [**here**](https://docs.aws.amazon.com/xray/latest/devguide/xray-instrumenting-your-app.html#xray-instrumenting-choosing). 
+
+## AWS X-Ray SDK for .NET
+
+The X-Ray SDK for .NET is an open source project. X-Ray SDK for .NET is supported for applications targeting .NET Framework 4.5 or later. For .NET Core applications, the SDK requires .NET Core 2.0 or later.
+
+Here are the links for you to get started.
+
+[**AWS X-Ray SDK for .NET developer guide**](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-dotnet.html) - The documentation explains installation via NuGet, configuration options, and various instrumentation capabilities including automatic HTTP request tracing and AWS service call monitoring. It covers how developers can create custom segments, add annotations, and utilize sampling rules to manage data collection. The guide offers comprehensive information for integrating X-Ray tracing into ASP.NET applications, helping developers gain visibility into application performance and troubleshoot issues effectively.
+
+[**The SDK open source project repo - aws-xray-sdk-dotnet**](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-dotnet.html) - The aws-xray-sdk-dotnet repository houses the open source code for Amazon's X-Ray SDK for .NET. Developers can see the implementation of this tracing tool which supports distributed application monitoring across .NET Core and .NET Framework environments. The repository contains the source code for automatic instrumentation of HTTP requests, AWS service calls, and custom instrumenting capabilities. You can review how the SDK integrates with ASP.NET frameworks and implements sampling rules. This GitHub project provides transparency into the SDK's functionality while allowing developers to report issues or contribute improvements to the codebase.
+
+The below are the API reference manuals that comprehensively describe the components of .NET X-Ray SDK. 
+
+[**The API Reference for .NET Framework**](https://docs.aws.amazon.com/xray-sdk-for-dotnet/latest/reference/html/d0b774be-200e-4897-9ce6-4e43c3f6f5de.htm)
+
+[**The API Reference for .NET (Core)**](https://docs.aws.amazon.com/xray-sdk-for-dotnetcore/latest/reference/html/bdf06719-4833-4e03-8ce5-31debb71506c.htm)
+
+Sample applications to learn using X-Ray SDK for .NET in your ASP.NET and ASP.NET Core applications are linked below
+
+[**Sample ASP.NET and ASP.NET core applications**](https://github.com/aws-samples/aws-xray-dotnet-webapp)