access violation with System.Data.OleDb 4.7.1

[MaceWindu]

Description

After System.Data.OleDb 4.7.1 release, which fix transaction issues, we tried to enable our access oledb tests in .net core and faced another issue:
now tests crash (not always, in about 50% of runs) with access violation. This behavior wasn't observed before with 4.7.0 and is not observed in .net framework, so I would suspect something still is not right with System.Data.OleDb code (but also could be ACECORE.dll issue)

Configuration

dotnet --version
3.1.300

test command

dotnet vstest ./Tests/Linq/bin/Debug/netcoreapp2.1/linq2db.Tests.dll /Framework:.NETCoreApp,Version=v2.1 

Regression?

Probably, as 4.7.0 version had issues with transactions, but tests executed to the end.

Other information

dump file:
testhost_17220_46b7adf58c794c0f88f83d6f4ca647ba.zip

windbg outputs

CONTEXT:  (.ecxr)
rax=000000000003000a rbx=0000026dd69d09a0 rcx=00000250aadf1288
rdx=0000000000000001 rsi=00000250aadf1178 rdi=00000250aadf1230
rip=00007ffacbb0d7a8 rsp=00000065228bcb70 rbp=00000065228bcbf0
 r8=0000026dd69d1688  r9=0000000000000000 r10=0000000000000001
r11=0000026dd69d1608 r12=0000026dd69d0000 r13=0000000000000001
r14=00000250aadf16d0 r15=00000250aadf0010
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
ACECORE+0x6d7a8:
00007ffa`cbb0d7a8 4883780800      cmp     qword ptr [rax+8],0 ds:00000000`00030012=????????????????
Resetting default scope

FAULTING_IP: 
ACECORE+6d7a8
00007ffa`cbb0d7a8 4883780800      cmp     qword ptr [rax+8],0

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ffacbb0d7a8 (ACECORE+0x000000000006d7a8)
   ExceptionCode: c0000005 (Access violation)
  ExceptionFlags: 00000000
NumberParameters: 2
   Parameter[0]: 0000000000000000
   Parameter[1]: 0000000000030012
Attempt to read from address 0000000000030012

PROCESS_NAME:  testhost.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_PARAMETER1:  0000000000000000

EXCEPTION_PARAMETER2:  0000000000030012

READ_ADDRESS:  0000000000030012 

FOLLOWUP_IP: 
ACECORE+6d7a8
00007ffa`cbb0d7a8 4883780800      cmp     qword ptr [rax+8],0

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

APP:  testhost.exe

ANALYSIS_VERSION: 10.0.10240.9 amd64fre

MANAGED_CODE: 1

MANAGED_ENGINE_MODULE:  coreclr

MANAGED_ANALYSIS_PROVIDER:  SOS

BUGCHECK_STR:  INVALID_POINTER_READ_CLRSETUP_MSCORLIBMISMATCH

DEFAULT_BUCKET_ID:  INVALID_POINTER_READ_CLRSETUP_MSCORLIBMISMATCH

LAST_CONTROL_TRANSFER:  from 00007ffacbb0a3fa to 00007ffacbb0d7a8

STACK_TEXT:  
00000065`228bcb70 00007ffa`cbb0a3fa : 0000026d`ffffffff 0000026d`00000018 0000026d`d69d0000 0000026d`d69d1688 : ACECORE+0x6d7a8
00000065`228bcc00 00007ffa`cbaf62a4 : 00000250`00000010 00000250`aadf01f0 00000250`aadf0030 00000000`00000000 : ACECORE+0x6a3fa
00000065`228bcc80 00007ffa`cbb2bc15 : 00000000`00000000 00007ffa`cbac7ec2 00000000`00000000 00000000`00000000 : ACECORE+0x562a4
00000065`228bcf10 00007ffa`cbb2b8a2 : 00007ffa`00000000 00000250`95d6c508 00007ffa`cbc6b120 00000000`000000ff : ACECORE+0x8bc15
00000065`228bd180 00007ffa`cb7ad92d : 00000250`aa4bd688 00000000`00000000 00000250`ab743b00 00000250`95d6c8f0 : ACECORE+0x8b8a2
00000065`228bd200 00007ffa`cb7ad839 : 00000250`aa4bd688 00000000`00000000 00000250`00000005 00000000`000000ff : ACEOLEDB!DllCanUnloadNow+0x72e1
00000065`228bd250 00007ffa`cb7ad5f0 : 00000250`aa4bd688 00000250`aa4bd688 00000000`00000000 00000250`a8836a80 : ACEOLEDB!DllCanUnloadNow+0x71ed
00000065`228bd370 00007ffa`6f495132 : 00000250`93f6ca08 00000065`228bd5f0 00000250`95d6c508 00000065`228bd398 : ACEOLEDB!DllCanUnloadNow+0x6fa4
00000065`228bd4f0 00007ffa`6f494fb8 : 00000250`95d6c508 00000000`00000000 00000000`00000000 00000065`228bd6a0 : 0x00007ffa`6f495132
00000065`228bd600 00007ffa`6f49432c : 00000000`00000000 00000250`95b16ba8 00000000`00000000 00000250`95b16f50 : 0x00007ffa`6f494fb8
00000065`228bd640 00007ffa`6f491207 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffa`6f49432c
00000065`228bd6b0 00007ffa`cd1220d4 : 00000250`95b16f50 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffa`6f491207
00000065`228bd730 00007ffa`6f52a785 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : System_Data_Common+0x1f20d4
00000065`228bd760 00007ffa`6f52a69b : 00000250`95b16f50 00000000`00000000 00000250`93f64db8 00000000`00000000 : 0x00007ffa`6f52a785
00000065`228bd7d0 00007ffa`6f52a0ee : 00000250`95aa4718 00000250`95b16f50 00000001`00000000 00000250`95d586e0 : 0x00007ffa`6f52a69b
00000065`228bd810 00007ffa`6f529da2 : 00000250`95aa4718 00007ffa`00000000 00000065`228bdae0 00000000`00000000 : 0x00007ffa`6f52a0ee
00000065`228bdaa0 00007ffa`6f524b59 : 00000250`95aa4718 00000250`00000001 00000250`95d6b468 00000000`00000000 : 0x00007ffa`6f529da2
00000065`228bdae0 00007ffa`7096d845 : 00000250`95d69ae0 00000250`95d30078 00000000`00000000 00000250`95d2f098 : 0x00007ffa`6f524b59
00000065`228bdbc0 00007ffa`7096d49c : 00000250`95d69a40 00007ffa`7096d285 00000250`95d69a40 00000000`fffffffe : 0x00007ffa`7096d845
00000065`228bdc70 00007ffa`7096d349 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffa`7096d49c
00000065`228bdce0 00007ffa`7096d0c7 : 00000250`95d59358 00000250`95aa4718 00000250`95d2f098 00000000`00000000 : 0x00007ffa`7096d349
00000065`228bdd20 00007ffa`70969d95 : 00000250`95d68d80 00000250`95aa4718 00000250`95d2f098 00000000`00000000 : 0x00007ffa`7096d0c7
00000065`228bdd90 00007ffa`7096f5bb : 00000250`95d2e7f8 00000250`95d2f098 00000250`95d2e7d0 00000065`228be060 : 0x00007ffa`70969d95
00000065`228bde70 00007ffa`7095b047 : 00000250`95aa4718 40b9b3a5`7fb69985 40b9b3a5`7fb69985 00000000`00988569 : 0x00007ffa`7096f5bb
00000065`228be080 00007ffa`cdfb3053 : 00000250`95987128 00000250`8fe13d18 00000000`0000001d 00000065`228bf010 : 0x00007ffa`7095b047
00000065`228be1a0 00007ffa`cdefd39e : ffffffff`fffffffe 00000000`00000000 00000000`00000000 00000065`228be200 : coreclr!CallDescrWorkerInternal+0x83
00000065`228be1e0 00007ffa`cdefd7c7 : 00007ffa`cdc20b10 00000250`95aa4688 00000065`228be460 00000065`228be450 : coreclr!CallDescrWorkerReflectionWrapper+0x1a
00000065`228be230 00007ffa`cd8cf837 : 00007ffa`6edd19a8 00007ffa`70bbae20 00000000`00000000 00007ffa`cf197800 : coreclr!RuntimeMethodHandle::InvokeMethod+0x3e7
00000065`228be7a0 00007ffa`6f04e6da : 00000250`95a9a118 00000250`95a99ef0 00000065`228be940 00007ffa`6f077809 : System_Private_CoreLib+0x6df837
00000065`228be810 00007ffa`6f077669 : 00000250`95a99ef0 00007ffa`6f04e47b 00000250`90370210 00000250`95a9a698 : 0x00007ffa`6f04e6da
00000065`228be870 00007ffa`6f0775d8 : 00007ffa`6efea9d8 00000250`90373ac8 00000250`95a9a5d0 00000250`93584740 : 0x00007ffa`6f077669
00000065`228be8d0 00007ffa`6f0774d0 : 00000250`95a99e00 00000250`93d69e58 00000250`95a99ef0 00000065`228bf010 : 0x00007ffa`6f0775d8
00000065`228be910 00007ffa`6f077361 : 00000250`95a99ef0 00000250`93584740 00000250`95a99ef0 00000065`228bf010 : 0x00007ffa`6f0774d0
00000065`228be950 00007ffa`6f075f71 : 00000065`228bea30 00000250`93584740 00000250`95a99ef0 00000065`228bf010 : 0x00007ffa`6f077361
00000065`228be990 00007ffa`6f04b487 : 00000250`93584740 00000250`903b58c8 00000250`93584740 00007ffa`6f049c54 : 0x00007ffa`6f075f71
00000065`228be9d0 00007ffa`6f04ccaa : 00000250`93583d00 00007ffa`6f04bcc3 00000065`228bea30 00000000`ffffffff : 0x00007ffa`6f04b487
00000065`228bea40 00007ffa`6f04b487 : 00000065`228beba0 00000250`903b6058 00000250`93583d00 00007ffa`6f049c54 : 0x00007ffa`6f04ccaa
00000065`228beac0 00007ffa`6f04ccaa : 00000250`93581470 00007ffa`6f04bcc3 00000065`228beb20 00000000`ffffffff : 0x00007ffa`6f04b487
00000065`228beb30 00007ffa`6f04b487 : 00000250`000000a5 00000250`9036e590 00000250`93581470 00007ffa`6f04a6c4 : 0x00007ffa`6f04ccaa
00000065`228bebb0 00007ffa`6f04ac93 : 00000250`932e4980 00000250`932e7f48 00000250`93d67398 00000250`a8782b80 : 0x00007ffa`6f04b487
00000065`228bec20 00007ffa`cf677060 : 00000250`932e7f48 00000250`93fa0380 00000000`00000001 00000000`0000001a : 0x00007ffa`6f04ac93
00000065`228bec70 00007ffa`cd83d409 : 00000250`93fa03c0 00000250`93fa03e8 00000250`a880c7c0 00000250`a8782b80 : System_Threading_Thread+0x7060
00000065`228beca0 00007ffa`cdfb3053 : 00000250`93fa0498 00000000`00000001 00000065`228bef50 00000000`00000048 : System_Private_CoreLib+0x64d409
00000065`228bed20 00007ffa`cdea2763 : 00000250`8e2124c0 00000065`228befc8 00000250`8e210000 00000065`2092e000 : coreclr!CallDescrWorkerInternal+0x83
00000065`228bed60 00007ffa`cdfa18e1 : 00000065`228bf2d0 00000000`00000008 00000000`00000000 00000250`a87fea60 : coreclr!MethodDescCallSite::CallTargetWorker+0x273
00000065`228beea0 00007ffa`cdea2332 : 00000250`93fa0498 00000250`a87fea60 00000065`228bf0f0 00007ffb`2771ce16 : coreclr!ThreadNative::KickOffThread_Worker+0x131
00000065`228bf010 00007ffa`cdea2127 : ffffffff`00007e64 00000000`00000005 00000000`00000001 00007ffa`cdedf112 : coreclr!ManagedThreadBase_DispatchMiddle+0x176
00000065`228bf150 00007ffa`cdfabf5f : 00000000`00000001 ffffffff`ffffffff 00000250`a8782b80 00000000`00000000 : coreclr!ManagedThreadBase_DispatchOuter+0xaf
00000065`228bf1f0 00007ffa`cdedacdc : 00000250`a8782b80 00000065`00000001 00000000`00000008 00000065`228bf1d0 : coreclr!ManagedThreadBase_FullTransitionWithAD+0x2f
00000065`228bf250 00007ffa`cdedabba : 00000000`00000001 00000000`00000008 00000000`00000000 00000250`a8782b80 : coreclr!ThreadNative::KickOffThread+0xfc
00000065`228bf330 00007ffb`27a11fe4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : coreclr!Thread::intermediateThreadProc+0x8a
00000065`228bf8f0 00007ffb`2807ef91 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
00000065`228bf920 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21


SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  acecore+6d7a8

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ACECORE

IMAGE_NAME:  ACECORE.DLL

DEBUG_FLR_IMAGE_TIMESTAMP:  5e76d97c

STACK_COMMAND:  .ecxr ; kb

BUCKET_ID:  INVALID_POINTER_READ_CLRSETUP_MSCORLIBMISMATCH_acecore+6d7a8

PRIMARY_PROBLEM_CLASS:  INVALID_POINTER_READ_CLRSETUP_MSCORLIBMISMATCH_acecore+6d7a8

FAILURE_PROBLEM_CLASS:  INVALID_POINTER_READ_CLRSETUP_MSCORLIBMISMATCH

FAILURE_EXCEPTION_CODE:  c0000005

FAILURE_IMAGE_NAME:  ACECORE.DLL

FAILURE_FUNCTION_NAME:  Unknown

FAILURE_SYMBOL_NAME:  ACECORE.DLL!Unknown

FAILURE_BUCKET_ID:  INVALID_POINTER_READ_CLRSETUP_MSCORLIBMISMATCH_c0000005_ACECORE.DLL!Unknown

ANALYSIS_SOURCE:  UM

FAILURE_ID_HASH_STRING:  um:invalid_pointer_read_clrsetup_mscorlibmismatch_c0000005_acecore.dll!unknown

FAILURE_ID_HASH:  {6b70a4b5-25f9-c674-a177-061709309b2a}

More details could be provided if needed

[MaceWindu]

BTW, crashes not happen when running netcore 3.1

[MaceWindu]

Actually, 3.1 also affected. Getting random crashes like this one:

2020-07-09T13:30:40.6199640Z ##[error]The active test run was aborted. Reason: Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
2020-07-09T13:30:40.6209719Z ##[error]   at System.Data.Common.UnsafeNativeMethods+ICommandText.Execute(IntPtr, System.Guid ByRef, System.Data.OleDb.tagDBPARAMS, IntPtr ByRef, System.Object ByRef)
2020-07-09T13:30:40.6211456Z ##[error]   at System.Data.Common.UnsafeNativeMethods+ICommandText.Execute(IntPtr, System.Guid ByRef, System.Data.OleDb.tagDBPARAMS, IntPtr ByRef, System.Object ByRef)
2020-07-09T13:30:40.6212727Z ##[error]   at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForSingleResult(System.Data.OleDb.tagDBPARAMS, System.Object ByRef)
2020-07-09T13:30:40.6213993Z ##[error]   at System.Data.OleDb.OleDbCommand.ExecuteCommandText(System.Object ByRef)
2020-07-09T13:30:40.6215119Z ##[error]   at System.Data.OleDb.OleDbCommand.ExecuteCommand(System.Data.CommandBehavior, System.Object ByRef)
2020-07-09T13:30:40.6216221Z ##[error]   at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(System.Data.CommandBehavior, System.String)
2020-07-09T13:30:40.6217230Z ##[error]   at System.Data.OleDb.OleDbCommand.ExecuteReader(System.Data.CommandBehavior)
2020-07-09T13:30:40.6218618Z ##[error]   at System.Data.OleDb.OleDbCommand.ExecuteDbDataReader(System.Data.CommandBehavior)
2020-07-09T13:30:40.6219866Z ##[error]   at System.Data.Common.DbCommand.ExecuteReader(System.Data.CommandBehavior)
2020-07-09T13:30:40.6221761Z ##[error]   at LinqToDB.Data.DbCommandProcessor.DbCommandProcessorExtensions.ExecuteReaderExt(System.Data.IDbCommand, System.Data.CommandBehavior)
2020-07-09T13:30:40.6223045Z ##[error]   at LinqToDB.Data.DataConnection.ExecuteReader(System.Data.IDbCommand, System.Data.CommandBehavior)
2020-07-09T13:30:40.6224063Z ##[error]   at LinqToDB.Data.DataConnection.ExecuteReader(System.Data.CommandBehavior)
2020-07-09T13:30:40.6225038Z ##[error]   at LinqToDB.Data.DataConnection.ExecuteReader()
2020-07-09T13:30:40.6225961Z ##[error]   at LinqToDB.Data.DataConnection+QueryRunner.ExecuteReader()
2020-07-09T13:30:40.6227166Z ##[error]   at LinqToDB.Linq.QueryRunner+<ExecuteQuery>d__13`1[[System.Decimal, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
2020-07-09T13:30:40.6228660Z ##[error]   at System.Linq.Enumerable.TryGetFirst[[System.Decimal, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable`1<System.Decimal>, Boolean ByRef)
2020-07-09T13:30:40.6230184Z ##[error]   at System.Linq.Enumerable.First[[System.Decimal, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable`1<System.Decimal>)
2020-07-09T13:30:40.6232048Z ##[error]   at LinqToDB.Linq.Builder.FirstSingleBuilder+FirstSingleContext+<>c__DisplayClass3_0`1[[System.Decimal, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<GetFirstElement>b__0(LinqToDB.IDataContext, System.Linq.Expressions.Expression, System.Object[], System.Object[])
2020-07-09T13:30:40.6234052Z ##[error]   at LinqToDB.Linq.ExpressionQuery`1[[System.Decimal, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].System.Linq.IQueryProvider.Execute[[System.Decimal, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Linq.Expressions.Expression)
2020-07-09T13:30:40.6235780Z ##[error]   at System.Linq.Queryable.First[[System.Decimal, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Linq.IQueryable`1<System.Decimal>)
2020-07-09T13:30:40.6237600Z ##[error]   at Tests.Linq.ConvertTests.AssertConvert[[System.Decimal, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Decimal, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](Tests.Model.ITestDataContext, System.Decimal, System.Decimal)
2020-07-09T13:30:40.6238937Z ##[error]   at Tests.Linq.ConvertTests.ConvertFromOneToAnother(System.String)

https://dev.azure.com/linq2db/0dcc414b-ea54-451e-a54f-d63f05367c4b/_apis/build/builds/2628/logs/134

[MaceWindu]

What I find suspicious is that before COM issue with transactions were fixed (#32405) - we had no crashes (only failing tests due to transaction API broken).

[roji]

@saurabh500 does this ring any bells? Could this be connected to #32405 as suggested above?