Prepare for 1.2 release

Author: mShan0

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,37 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+There are some features which are not supported yet. Please check the [Limitations](https://github.com/microsoft/mssql-django#limitations) first to see if your bug is listed.
+
+**Software versions**
+* Django:
+* mssql-django:
+* python:
+* SQL Server:
+* OS:
+
+**Table schema and Model**
+<!-- Provide the table schema and model from models.py to reproduce the issue. -->
+
+**Database Connection Settings**
+`
+// Paste your database settings from Settings.py here.
+`
+
+**Problem description and steps to reproduce**
+<!-- Provide full details of the problem, including steps to reproduce the issue. -->
+
+**Expected behavior and actual behavior**
+<!-- A clear and concise description of what you expected to happen. -->
+
+**Error message/stack trace**
+<!-- Complete error message and stack trace. -->
+
+**Any other details that can be helpful**
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,23 @@
+---
+name: Feature Request
+about: Suggest an idea for this project
+title: "[FEATURE REQUEST]"
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? If so, please give a short summary of the problem and how the feature would resolve it**
+<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
+
+**Describe the preferred solution**
+<!-- A clear and concise description of what you want to happen. -->
+
+**Describe alternatives you've considered**
+<!-- A clear and concise description of any alternative solutions or features you've considered. -->
+
+**Additional context**
+<!-- Add any other context or screenshots about the feature request here. -->
+
+**Reference Documentations/Specifications**
+<!-- Provide links to any related documentation. -->

.github/ISSUE_TEMPLATE/question.md

@@ -0,0 +1,14 @@
+---
+name: Question
+about: Ask a question
+title: "[QUESTION]"
+labels: question
+assignees: ''
+
+---
+
+**Question**
+<!-- What is the question that you have? Please be detailed and give examples. -->
+
+**Relevant Issues and Pull Requests**
+<!-- If there are relevant issues and pull requests please list and link them here. -->

README.md

@@ -125,7 +125,7 @@ Dictionary. Current available keys are:
    for each database session. Valid values for this entry are
    `READ UNCOMMITTED`, `READ COMMITTED`, `REPEATABLE READ`,
    `SNAPSHOT`, and `SERIALIZABLE`. Default is `None` which means
-   no isolation levei is set to a database session and SQL Server default
+   no isolation level is set to a database session and SQL Server default
    will be used.
 
 -  dsn
@@ -204,6 +204,27 @@ Dictionary. Current available keys are:
             },
     ```
 
+- return_rows_bulk_insert
+
+  Boolean. Sets if backend can return rows from bulk insert.
+  Default value is False which doesn't allows for the backend to
+  return rows from bulk insert. Must be set to False if database
+  has tables with triggers to prevent errors when inserting.
+
+  ```python
+  # Examples
+  "OPTIONS": {
+      # This database doesn't have any triggers so can use return
+      # rows from bulk insert feature
+      "return_rows_bulk_insert": True
+  }
+
+  "OPTIONS": {
+      # This database has triggers so leave return_rows_bulk_insert as blank (False)
+      # to prevent errors related to inserting and returning rows from bulk insert
+  }
+  ```
+
 ### Backend-specific settings
 
 The following project-level settings also control the behavior of the backend:
@@ -248,11 +269,9 @@ The following features are currently not fully supported:
 - Timezones, timedeltas not fully supported
 - Rename field/model with foreign key constraint
 - Database level constraints
-- Math degrees power or radians
-- Bit-shift operators
 - Filtered index
 - Date extract function
-- Hashing functions
+- Bulk insert into a table with a trigger and returning the rows inserted
 
 JSONField lookups have limitations, more details [here](https://github.com/microsoft/mssql-django/wiki/JSONField).
 

azure-pipelines.yml

@@ -11,6 +11,9 @@ schedules:
     - dev
   always: true
 
+variables:
+- group: DjangoTestApp
+
 jobs:
   - job: Windows
     pool:
@@ -21,6 +24,16 @@ jobs:
 
     strategy:
       matrix:
+        Python3.10 - Django 4.1:
+          python.version: '3.10'
+          tox.env: 'py310-django41'
+        Python 3.9 - Django 4.1:
+          python.version: '3.9'
+          tox.env: 'py39-django41'
+        Python 3.8 - Django 4.1:
+          python.version: '3.8'
+          tox.env: 'py38-django41'
+
         Python3.10 - Django 4.0:
           python.version: '3.10'
           tox.env: 'py310-django40'
@@ -44,39 +57,6 @@ jobs:
           python.version: '3.6'
           tox.env: 'py36-django32'
 
-        Python 3.9 - Django 3.1:
-          python.version: '3.9'
-          tox.env: 'py39-django31'
-        Python 3.8 - Django 3.1:
-          python.version: '3.8'
-          tox.env: 'py38-django31'
-        Python 3.7 - Django 3.1:
-          python.version: '3.7'
-          tox.env: 'py37-django31'
-        Python 3.6 - Django 3.1:
-          python.version: '3.6'
-          tox.env: 'py36-django31'
-
-        Python 3.9 - Django 3.0:
-          python.version: '3.9'
-          tox.env: 'py39-django30'
-        Python 3.8 - Django 3.0:
-          python.version: '3.8'
-          tox.env: 'py38-django30'
-        Python 3.7 - Django 3.0:
-          python.version: '3.7'
-          tox.env: 'py37-django30'
-        Python 3.6 - Django 3.0:
-          python.version: '3.6'
-          tox.env: 'py36-django30'
-
-        Python 3.7 - Django 2.2:
-          python.version: '3.7'
-          tox.env: 'py37-django22'
-        Python 3.6 - Django 2.2:
-          python.version: '3.6'
-          tox.env: 'py36-django22'
-
     steps:
       - task: CredScan@2
         inputs:
@@ -102,14 +82,18 @@ jobs:
           Invoke-Sqlcmd @"
               EXEC xp_instance_regwrite N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'LoginMode', REG_DWORD, 2
               ALTER LOGIN [sa] ENABLE;
-              ALTER LOGIN [sa] WITH PASSWORD = 'MyPassword42', CHECK_POLICY=OFF;
+              ALTER LOGIN [sa] WITH PASSWORD = '$(TestAppPassword)', CHECK_POLICY=OFF;
           "@
         displayName: Set up SQL Server
 
       - powershell: |
           Restart-Service -Name MSSQLSERVER -Force
         displayName: Restart SQL Server
 
+      - powershell: |
+          (Get-Content -ReadCount 0 testapp\settings.py) -replace 'MyPassword42', '$(TestAppPassWord)' | Set-Content testapp\settings.py
+        displayName: Change PASSWORD in settings.py
+
       - powershell: |
           python -m pip install --upgrade pip wheel setuptools
           python -m pip install tox
@@ -127,6 +111,16 @@ jobs:
 
     strategy:
       matrix:
+        Python3.10 - Django 4.1:
+          python.version: '3.10'
+          tox.env: 'py310-django41'
+        Python 3.9 - Django 4.1:
+          python.version: '3.9'
+          tox.env: 'py39-django41'
+        Python 3.8 - Django 4.1:
+          python.version: '3.8'
+          tox.env: 'py38-django41'
+
         Python3.10 - Django 4.0:
           python.version: '3.10'
           tox.env: 'py310-django40'
@@ -150,48 +144,15 @@ jobs:
           python.version: '3.6'
           tox.env: 'py36-django32'
 
-        Python 3.9 - Django 3.1:
-          python.version: '3.9'
-          tox.env: 'py39-django31'
-        Python 3.8 - Django 3.1:
-          python.version: '3.8'
-          tox.env: 'py38-django31'
-        Python 3.7 - Django 3.1:
-          python.version: '3.7'
-          tox.env: 'py37-django31'
-        Python 3.6 - Django 3.1:
-          python.version: '3.6'
-          tox.env: 'py36-django31'
-
-        Python 3.9 - Django 3.0:
-          python.version: '3.9'
-          tox.env: 'py39-django30'
-        Python 3.8 - Django 3.0:
-          python.version: '3.8'
-          tox.env: 'py38-django30'
-        Python 3.7 - Django 3.0:
-          python.version: '3.7'
-          tox.env: 'py37-django30'
-        Python 3.6 - Django 3.0:
-          python.version: '3.6'
-          tox.env: 'py36-django30'
-
-        Python 3.7 - Django 2.2:
-          python.version: '3.7'
-          tox.env: 'py37-django22'
-        Python 3.6 - Django 2.2:
-          python.version: '3.6'
-          tox.env: 'py36-django22'
-
     steps:
       - task: UsePythonVersion@0
         inputs:
           versionSpec: "$(python.version)"
         displayName: Use Python $(python.version)
 
       - script: |
-          docker pull mcr.microsoft.com/mssql/server:2019-latest
-          docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=MyPassword42' -p 1433:1433 -d mcr.microsoft.com/mssql/server:2019-latest
+          docker pull mcr.microsoft.com/mssql/server:2022-latest
+          docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=$(TestAppPassword)' -p 1433:1433 -d mcr.microsoft.com/mssql/server:2022-latest
           curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
           curl https://packages.microsoft.com/config/ubuntu/20.04/prod.list | sudo tee /etc/apt/sources.list.d/mssql-release.list
           sudo apt-get update
@@ -204,6 +165,10 @@ jobs:
           git clone https://github.com/django/django.git
         displayName: Install requirements
 
+      - script: |
+          sed -i 's/MyPassword42/$(TestAppPassword)/g' testapp/settings.py
+        displayName: Change PASSWORD in settings.py
+
       - script: tox -e $(tox.env)
         displayName: Run tox
 

mssql/base.py

@@ -8,6 +8,7 @@
 import re
 import time
 import struct
+import datetime
 
 from django.core.exceptions import ImproperlyConfigured
 
@@ -35,12 +36,12 @@
 from .client import DatabaseClient  # noqa
 from .creation import DatabaseCreation  # noqa
 from .features import DatabaseFeatures  # noqa
-from .introspection import DatabaseIntrospection  # noqa
+from .introspection import DatabaseIntrospection, SQL_TIMESTAMP_WITH_TIMEZONE  # noqa
 from .operations import DatabaseOperations  # noqa
 from .schema import DatabaseSchemaEditor  # noqa
 
 EDITION_AZURE_SQL_DB = 5
-
+EDITION_AZURE_SQL_MANAGED_INSTANCE = 8
 
 def encode_connection_string(fields):
     """Encode dictionary of keys and values as an ODBC connection String.
@@ -80,6 +81,13 @@ def encode_value(v):
     return v
 
 
+def handle_datetimeoffset(dto_value):
+    # Decode bytes returned from SQL Server
+    # source: https://github.com/mkleehammer/pyodbc/wiki/Using-an-Output-Converter-function
+    tup = struct.unpack("<6hI2h", dto_value)  # e.g., (2017, 3, 16, 10, 35, 18, 500000000)
+    return datetime.datetime(tup[0], tup[1], tup[2], tup[3], tup[4], tup[5], tup[6] // 1000)
+
+
 class DatabaseWrapper(BaseDatabaseWrapper):
     vendor = 'microsoft'
     display_name = 'SQL Server'
@@ -184,6 +192,7 @@ class DatabaseWrapper(BaseDatabaseWrapper):
         13: 2016,
         14: 2017,
         15: 2019,
+        16: 2022,
     }
 
     # https://azure.microsoft.com/en-us/documentation/articles/sql-database-develop-csharp-retry-windows/
@@ -234,6 +243,9 @@ def __init__(self, *args, **kwargs):
                     ops[op] = '%s COLLATE %s' % (sql, collation)
             self.operators.update(ops)
 
+        if (settings.USE_TZ):
+            self.data_types['DateTimeField'] ='datetimeoffset'
+
     def create_cursor(self, name=None):
         return CursorWrapper(self.connection.cursor(), self)
 
@@ -364,6 +376,9 @@ def get_new_connection(self, conn_params):
                 if not need_to_retry:
                     raise
 
+        # Handling values from DATETIMEOFFSET columns
+        # source: https://github.com/mkleehammer/pyodbc/wiki/Using-an-Output-Converter-function
+        conn.add_output_converter(SQL_TIMESTAMP_WITH_TIMEZONE, handle_datetimeoffset)
         conn.timeout = query_timeout
         if setencoding:
             for entry in setencoding:
@@ -410,6 +425,11 @@ def init_connection_state(self):
         datefirst = options.get('datefirst', 7)
         cursor.execute('SET DATEFORMAT ymd; SET DATEFIRST %s' % datefirst)
 
+        # Let user choose if driver can return rows from bulk insert since
+        # inserting into tables with triggers causes errors. See issue #130
+        if (options.get('return_rows_bulk_insert', False)):
+            self.features_class.can_return_rows_from_bulk_insert = True
+
         val = self.get_system_datetime()
         if isinstance(val, str):
             raise ImproperlyConfigured(
@@ -464,7 +484,8 @@ def to_azure_sql_db(self, _known_azures={}):
         if self.alias not in _known_azures:
             with self.temporary_connection() as cursor:
                 cursor.execute("SELECT CAST(SERVERPROPERTY('EngineEdition') AS integer)")
-                _known_azures[self.alias] = cursor.fetchone()[0] == EDITION_AZURE_SQL_DB
+                edition = cursor.fetchone()[0]
+                _known_azures[self.alias] = edition == EDITION_AZURE_SQL_DB or edition == EDITION_AZURE_SQL_MANAGED_INSTANCE
         return _known_azures[self.alias]
 
     def _execute_foreach(self, sql, table_names=None):