Skip to content

Commit c4bfada

Browse files
tvaron3tvaron3
authored
Fabric Entra Id Auth Integration (#45890)
* fix flaky test with endLSN * fix flaky test with endLSN * account data resolver fabric * move account data resolver to other project * revert unrelated changes * Changed pom file to work for spark 3.4 or 3.5, updated readme, removed logging for empty response * update changelog * update README.md * fix tests * fix analyze * fix analyze * fix analyze * fix analyze * fix analyze * react to comments * react to comments * fix analyze * removed old line from README * react to comments * fix bug where sometimes config key is lowercased * changes after testing * revert cosmos pom * revert cosmos pom * update dependencies * update useragent tests * react to comments * revert extra line and comma * react to comments * fix tests * Update Readme and react to comments * fix tests * depend on current version * react to comments * fix tests * fix tests * name change * remove changelog from unrelated runtimes * revert unnecessary comma * fix build issue * fix build issue * fix build issue
1 parent 95ec759 commit c4bfada

File tree

25 files changed

+1025
-60
lines changed

25 files changed

+1025
-60
lines changed

‎.vscode/cspell.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@
104104
"sdk/cosmos/azure-cosmos-spark_3-4_2-12/**",
105105
"sdk/cosmos/azure-cosmos-spark_3-5_2-12/**",
106106
"sdk/cosmos/azure-cosmos-spark-account-data-resolver-sample/**",
107+
"sdk/cosmos/fabric-cosmos-spark-auth_3/**",
107108
"sdk/cosmos/azure-cosmos-encryption/**",
108109
"sdk/cosmos/azure-cosmos-spark_3_2-12/**",
109110
"sdk/spring/azure-spring-data-cosmos/**",

‎eng/.docsettings.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ known_content_issues:
8181
- ['sdk/cosmos/azure-cosmos-spark_3-4_2-12/README.md', '#3113']
8282
- ['sdk/cosmos/azure-cosmos-spark_3-5_2-12/README.md', '#3113']
8383
- ['sdk/cosmos/azure-cosmos-spark-account-data-resolver-sample/README.md', '#3113']
84+
- ['sdk/cosmos/fabric-cosmos-spark-auth_3/README.md', '#3113']
8485
- ['sdk/cosmos/azure-cosmos-spark_3_2-12/dev/README.md', '#3113']
8586
- ['sdk/cosmos/azure-cosmos-spark_3_2-12/docs/catalog-api.md', '#3113']
8687
- ['sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md', '#3113']

‎eng/versioning/external_dependencies.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,7 @@ cosmos_org.mpierce.metrics.reservoir:hdrhistogram-metrics-reservoir;1.1.0
252252
cosmos_org.hdrhistogram:HdrHistogram;2.1.12
253253
cosmos_com.fasterxml.jackson.core:jackson-databind;2.15.2
254254
cosmos_com.fasterxml.jackson.module:jackson-module-scala_2.12;2.15.2
255+
cosmos_com.microsoft.azure.synapse:synapseutils_2.12;1.5.4
255256

256257
## Cosmos Spark connector under sdk\cosmos\azure-cosmos-spark_3-<version>_2-12\pom.xml
257258
# Cosmos Spark connector runtime dependencies - provided by Spark runtime/host

‎eng/versioning/version_client.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@ com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12;4.37.2;4.38.0-beta.1
110110
com.azure.cosmos.spark:azure-cosmos-spark_3-4_2-12;4.37.2;4.38.0-beta.1
111111
com.azure.cosmos.spark:azure-cosmos-spark_3-5_2-12;4.37.2;4.38.0-beta.1
112112
com.azure.cosmos.spark:azure-cosmos-spark-account-data-resolver-sample;1.0.0-beta.1;1.0.0-beta.1
113+
com.azure.cosmos.spark:fabric-cosmos-spark-auth_3;1.0.0-beta.1;1.0.0-beta.1
113114
com.azure:azure-cosmos-test;1.0.0-beta.13;1.0.0-beta.14
114115
com.azure:azure-cosmos-tests;1.0.0-beta.1;1.0.0-beta.1
115116
com.azure.cosmos.kafka:azure-cosmos-kafka-connect;2.4.0;2.5.0-beta.1

‎sdk/cosmos/azure-cosmos-encryption/README.md

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -43,10 +43,10 @@ A few important properties are defined at the level of the container, among them
4343

4444
## Examples
4545
The following section provides several code snippets covering some of the most common Cosmos Encryption API tasks, including:
46-
* [Create Cosmos Encryption Client](#create-cosmos-encryption-client "Create Cosmos Encryption Client")
47-
* [Create Cosmos Encryption Database](#create-cosmos-encryption-database "Create Encryption Database")
48-
* [Create Encryption Container](#create-cosmos-encryption-container "Create Encryption Container")
49-
* [CRUD operation on Items](#crud-operation-on-items "CRUD operation on Items")
46+
* [Create Cosmos Encryption Client](#create-cosmos-encryption-client)
47+
* [Create Cosmos Encryption Database](#create-cosmos-encryption-database)
48+
* [Create Encryption Container](#create-cosmos-encryption-container)
49+
* [CRUD operation on Items](#crud-operation-on-items)
5050

5151
### Create Cosmos Encryption Client
5252

@@ -225,6 +225,3 @@ or contact [[email protected]][coc_contact] with any additional questions o
225225
[sql_api_query]: https://learn.microsoft.com/azure/cosmos-db/sql-api-sql-query
226226
[getting_started_encryption]: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-encryption/src/samples/java/com/azure/cosmos/encryption/
227227
[quickstart]: https://learn.microsoft.com/azure/cosmos-db/create-sql-api-java?tabs=sync
228-
229-
230-

‎sdk/cosmos/azure-cosmos-spark_3-5_2-12/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
#### Bugs Fixed
1010

1111
#### Other Changes
12+
* Added compatibility with CosmosDB Fabric Native Accounts using the `FabricAccountDataResolver` for authentication. - See [PR 45890](https://github.com/Azure/azure-sdk-for-java/pull/45890)
1213

1314
### 4.37.2 (2025-05-14)
1415

‎sdk/cosmos/azure-cosmos-spark_3_2-12/docs/AAD-Auth.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ Due to the fact that support for system managed identities and token/secret APIs
8888
| `spark.cosmos.account.subscriptionId` | None | The `SubscriptionId` of the Azure Cosmos DB account resource specified under `spark.cosmos.accountEndpoint`. This parameter is required for all management operations when using AAD / Microsoft Entra ID authentication. |
8989
| `spark.cosmos.account.tenantId` | None | The `AAD TenantId` of the Azure Cosmos DB account resource specified under `spark.cosmos.accountEndpoint`. This parameter is required for all management operations when using AAD / Microsoft Entra ID authentication. |
9090
| `spark.cosmos.account.resourceGroupName` | None | The simple resource group name (not the full qualified one) of the Azure Cosmos DB account resource specified under `spark.cosmos.accountEndpoint`. This parameter is required for all management operations when using AAD / Microsoft Entra ID authentication. |
91-
| `your.own.custom.property` | | You can add add an duse custom properties for the configuration of your custom `AccountDataResolver` implementation. |
91+
| `your.own.custom.property` | | You can add and use custom properties for the configuration of your custom `AccountDataResolver` implementation. |
9292

9393
### Implementation of a custom `AccountDataResolver`
9494

‎sdk/cosmos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/CosmosClientCache.scala

Lines changed: 37 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,20 @@ private[spark] object CosmosClientCache extends BasicLoggingTrait {
157157
}
158158
}
159159

160+
private def validateAadConfigs(cosmosClientConfiguration: CosmosClientConfiguration): Boolean = {
161+
val shouldLog = !(cosmosClientConfiguration.resourceGroupName.isDefined &&
162+
cosmosClientConfiguration.subscriptionId.isDefined &&
163+
cosmosClientConfiguration.tenantId.isDefined)
164+
if (shouldLog) {
165+
logWarning(
166+
"To create Databases, Containers and other resources in Cosmos DB using Microsoft Entra ID, " +
167+
"you need to provide resourceGroupName, subscriptionId and tenantId in the configuration. " +
168+
"Otherwise, the Cosmos Catalog will not be able to create resources."
169+
)
170+
}
171+
shouldLog
172+
}
173+
160174
private[this] def syncCreate(cosmosClientConfiguration: CosmosClientConfiguration,
161175
cosmosClientStateHandle: Option[CosmosClientMetadataCachesSnapshot],
162176
ownerInfo: OwnerInfo)
@@ -170,17 +184,20 @@ private[spark] object CosmosClientCache extends BasicLoggingTrait {
170184
var sparkCatalogClient: CosmosCatalogClient = CosmosCatalogCosmosSDKClient(cosmosAsyncClient)
171185
// When using AAD auth, cosmos catalog will change to use management sdk instead of cosmos sdk
172186
cosmosClientConfiguration.authConfig match {
173-
case aadAuthConfig: CosmosServicePrincipalAuthConfig =>
174-
sparkCatalogClient =
175-
CosmosCatalogManagementSDKClient(
176-
cosmosClientConfiguration.resourceGroupName.get,
177-
cosmosClientConfiguration.databaseAccountName,
178-
createCosmosManagementClient(
179-
cosmosClientConfiguration.subscriptionId.get,
180-
new AzureEnvironment(cosmosClientConfiguration.azureEnvironmentEndpoints),
181-
aadAuthConfig),
182-
cosmosAsyncClient)
183-
case managedIdentityAuth: CosmosManagedIdentityAuthConfig =>
187+
case aadAuthConfig: CosmosServicePrincipalAuthConfig =>
188+
if (!validateAadConfigs(cosmosClientConfiguration)) {
189+
sparkCatalogClient =
190+
CosmosCatalogManagementSDKClient(
191+
cosmosClientConfiguration.resourceGroupName.get,
192+
cosmosClientConfiguration.databaseAccountName,
193+
createCosmosManagementClient(
194+
cosmosClientConfiguration.subscriptionId.get,
195+
new AzureEnvironment(cosmosClientConfiguration.azureEnvironmentEndpoints),
196+
aadAuthConfig),
197+
cosmosAsyncClient)
198+
}
199+
case managedIdentityAuth: CosmosManagedIdentityAuthConfig =>
200+
if (!validateAadConfigs(cosmosClientConfiguration)) {
184201
sparkCatalogClient =
185202
CosmosCatalogManagementSDKClient(
186203
cosmosClientConfiguration.resourceGroupName.get,
@@ -190,7 +207,9 @@ private[spark] object CosmosClientCache extends BasicLoggingTrait {
190207
new AzureEnvironment(cosmosClientConfiguration.azureEnvironmentEndpoints),
191208
managedIdentityAuth),
192209
cosmosAsyncClient)
193-
case accessTokenProviderAuth: CosmosAccessTokenAuthConfig =>
210+
}
211+
case accessTokenProviderAuth: CosmosAccessTokenAuthConfig =>
212+
if (!validateAadConfigs(cosmosClientConfiguration)) {
194213
sparkCatalogClient =
195214
CosmosCatalogManagementSDKClient(
196215
cosmosClientConfiguration.resourceGroupName.get,
@@ -200,7 +219,8 @@ private[spark] object CosmosClientCache extends BasicLoggingTrait {
200219
new AzureEnvironment(cosmosClientConfiguration.azureEnvironmentEndpoints),
201220
accessTokenProviderAuth),
202221
cosmosAsyncClient)
203-
case _ =>
222+
}
223+
case _ =>
204224
}
205225

206226
val epochNowInMs = Instant.now.toEpochMilli
@@ -822,7 +842,10 @@ private[spark] object CosmosClientCache extends BasicLoggingTrait {
822842
private[this] def createCosmosManagementClient(subscriptionId: String,
823843
azureEnvironment: AzureEnvironment,
824844
authConfig: CosmosAccessTokenAuthConfig): CosmosManager = {
825-
val azureProfile = new AzureProfile(authConfig.tenantId, subscriptionId, azureEnvironment)
845+
val tenantId = authConfig.tenantId.getOrElse(
846+
throw new IllegalArgumentException("Tenant ID must be provided for CosmosAccessTokenAuthConfig")
847+
)
848+
val azureProfile = new AzureProfile(tenantId, subscriptionId, azureEnvironment)
826849

827850
CosmosManager.authenticate(createTokenCredential(authConfig), azureProfile)
828851
}

‎sdk/cosmos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/CosmosClientConfiguration.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ private[spark] object CosmosClientConfiguration {
6565
if (runtimeInfo.isDefined) {
6666
applicationName = s"$applicationName|${runtimeInfo.get}"
6767
}
68+
applicationName = applicationName.replace("@", " ")
6869

6970
val customApplicationNameSuffix = cosmosAccountConfig.applicationName
7071
if (customApplicationNameSuffix.isDefined){

‎sdk/cosmos/azure-cosmos-spark_3_2-12/src/main/scala/com/azure/cosmos/spark/CosmosConfig.scala

Lines changed: 2 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -767,16 +767,6 @@ private object CosmosAccountConfig extends BasicLoggingTrait {
767767
assert(accountName.isDefined, s"Parameter '${CosmosConfigNames.AccountEndpoint}' is missing.")
768768
assert(azureEnvironmentOpt.isDefined, s"Parameter '${CosmosConfigNames.AzureEnvironment}' is missing.")
769769

770-
authConfig match {
771-
case _: CosmosServicePrincipalAuthConfig =>
772-
case _: CosmosManagedIdentityAuthConfig =>
773-
case _: CosmosAccessTokenAuthConfig =>
774-
assert(subscriptionIdOpt.isDefined, s"Parameter '${CosmosConfigNames.SubscriptionId}' is missing.")
775-
assert(resourceGroupNameOpt.isDefined, s"Parameter '${CosmosConfigNames.ResourceGroupName}' is missing.")
776-
assert(tenantIdOpt.isDefined, s"Parameter '${CosmosConfigNames.TenantId}' is missing.")
777-
case _ =>
778-
}
779-
780770
if (preferredRegionsListOpt.isDefined) {
781771
// scalastyle:off null
782772
var uri : URI = null
@@ -872,7 +862,7 @@ private case class CosmosManagedIdentityAuthConfig( tenantId: String,
872862
clientId: Option[String],
873863
resourceId: Option[String]) extends CosmosAuthConfig
874864

875-
private case class CosmosAccessTokenAuthConfig(tenantId: String, tokenProvider: List[String] => CosmosAccessToken)
865+
private case class CosmosAccessTokenAuthConfig(tenantId: Option[String], tokenProvider: List[String] => CosmosAccessToken)
876866
extends CosmosAuthConfig
877867

878868
private object CosmosAuthConfig {
@@ -970,7 +960,6 @@ private object CosmosAuthConfig {
970960
clientSecret,
971961
clientCert)
972962
} else if (authType.get == CosmosAuthType.AccessToken) {
973-
assert(tenantId.isDefined, s"Parameter '${CosmosConfigNames.TenantId}' is missing.")
974963
val accountDataResolverServiceName : Option[String] = CaseInsensitiveMap(cfg).get(CosmosConfigNames.AccountDataResolverServiceName)
975964
val accountDataResolver = CosmosConfig.getAccountDataResolver(accountDataResolverServiceName)
976965
if (accountDataResolver.isEmpty) {
@@ -987,7 +976,7 @@ private object CosmosAuthConfig {
987976
"returns an access token provider in the 'getAccessTokenProvider' method.")
988977
}
989978

990-
CosmosAccessTokenAuthConfig(tenantId.get, accessTokenProvider.get)
979+
CosmosAccessTokenAuthConfig(tenantId, accessTokenProvider.get)
991980
} else {
992981
throw new IllegalArgumentException(s"Unknown auth type '${authType.get}'.")
993982
}

0 commit comments

Comments
 (0)