An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block #3777

basapuram-kumar · 2023-12-08T06:17:33Z

basapuram-kumar
Dec 8, 2023

Is there an existing issue for this?

I have searched the existing issues

Description

We are running Impala queries from Hue -> impala Editor. getting the below issue frequenlty with no pattern of occurance.

Here is the stacktrace.

[05/Dec/2023 12:02:42 -0800] decorators   ERROR    Error running execute
Traceback (most recent call last):
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/decorators.py", line 120, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 236, in execute
    response = _execute_notebook(request, notebook, snippet)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 198, in _execute_notebook
    history.save()
  File "/usr/lib/hue/desktop/core/src/desktop/models.py", line 1361, in save
    super(Document2, self).save(*args, **kwargs)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 905, in _save_table
    forced_update)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/query.py", line 667, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/sql/compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

Steps To Reproduce

This issue is not repro everytime.
There is an issue reported earlier(hue/issues/3035), but no solution provided, provided cloudera link solution didnt help.

Logs


544555&startedTimeBegin=1701328796000&doAs=rf544555  returned in 7ms 200 11 {"apps":{}}
[06/Dec/2023 23:19:56 -0800] access       INFO     10.217.45.86 rf544555 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 21ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:05 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=ss408403&startedTimeBegin=1701328805000&doAs=ss408403  returned in 5ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:05 -0800] access       INFO     10.44.42.3 ss408403 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 17ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:10 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=hxn24009&startedTimeBegin=1701328810000&doAs=hxn24009  returned in 7ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:10 -0800] access       INFO     10.44.50.255 hxn24009 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 19ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:24 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=pxk06977&startedTimeBegin=1701328824000&doAs=pxk06977  returned in 6ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:24 -0800] access       INFO     10.44.41.65 pxk06977 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 17ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:25 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=axy92614&startedTimeBegin=1701328825000&doAs=axy92614  returned in 5ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:25 -0800] access       INFO     10.45.138.138 axy92614 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 17ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:27 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=rf544555&startedTimeBegin=1701328827000&doAs=rf544555  returned in 4ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:27 -0800] access       INFO     10.217.45.86 rf544555 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 16ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:36 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=ss408403&startedTimeBegin=1701328836000&doAs=ss408403  returned in 8ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:36 -0800] access       INFO     10.44.42.3 ss408403 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 19ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:41 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=hxn24009&startedTimeBegin=1701328841000&doAs=hxn24009  returned in 4ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:41 -0800] access       INFO     10.44.50.255 hxn24009 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 14ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:55 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=pxk06977&startedTimeBegin=1701328855000&doAs=pxk06977  returned in 7ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:55 -0800] access       INFO     10.44.41.65 pxk06977 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 19ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:56 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=axy92614&startedTimeBegin=1701328856000&doAs=axy92614  returned in 8ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:56 -0800] access       INFO     10.45.138.138 axy92614 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 22ms 200 12 (mem: 341mb)
[06/Dec/2023 23:20:58 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=rf544555&startedTimeBegin=1701328858000&doAs=rf544555  returned in 6ms 200 11 {"apps":{}}
[06/Dec/2023 23:20:58 -0800] access       INFO     10.217.45.86 rf544555 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 16ms 200 12 (mem: 341mb)
[06/Dec/2023 23:21:07 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=ss408403&startedTimeBegin=1701328867000&doAs=ss408403  returned in 44ms 200 11 {"apps":{}}
[06/Dec/2023 23:21:07 -0800] access       INFO     10.44.42.3 ss408403 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 55ms 200 12 (mem: 341mb)
[06/Dec/2023 23:21:08 -0800] thrift_util  INFO     Thrift exception; retrying: unexpected exception
[06/Dec/2023 23:21:08 -0800] thrift_util  DEBUG    Thrift call: <class 'ImpalaService.ImpalaHiveServer2Service.Client'>.ExecuteStatement(args=(TExecuteStatementReq(confOverlay={'QUERY_TIMEOUT_S': '900'}, sessionHandle=TSessionHandle(sessionId=THandleIdentifier(secret=a04bd5f37c76ca04:7b5d3683011010a3, guid=f2433f8712c4d49f:6b01ca5cbd13f8b7)), runAsync=False, statement='USE `gene`', queryTimeout=0),), kwargs={})
[06/Dec/2023 23:21:12 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=hxn24009&startedTimeBegin=1701328872000&doAs=hxn24009  returned in 5ms 200 11 {"apps":{}}
[06/Dec/2023 23:21:12 -0800] access       INFO     10.44.50.255 hxn24009 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 18ms 200 12 (mem: 341mb)
[06/Dec/2023 23:21:21 -0800] api          ERROR    Autocomplete data fetching error
Traceback (most recent call last):
  File "/usr/lib/hue/apps/beeswax/src/beeswax/api.py", line 128, in _autocomplete
    table = db.get_table(database, table)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/dbms.py", line 478, in get_table
    return self.client.get_table(database, table_name)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 1549, in get_table
    table = self._client.get_table(database, table_name, partition_spec)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 894, in get_table
    (res, session) = self.call(self._client.GetTables, req)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 742, in call
    return self.call_return_result_and_session(fn, req, status, session=session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 767, in call_return_result_and_session
    return self._call_return_result_and_session(fn, req, status=status, session=session)
  File "/usr/lib/hue/apps/beeswax/src/beeswax/server/hive_server2_lib.py", line 795, in _call_return_result_and_session
    session.save()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 905, in _save_table
    forced_update)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/query.py", line 667, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/sql/compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')
[06/Dec/2023 23:21:21 -0800] access       INFO     10.45.138.138 axy92614 - "POST /api/editor/autocomplete/gene_pipeline/ext_ld_req_64_pip/ HTTP/1.1" returned in 301740ms 200 105 (mem: 341mb)
[06/Dec/2023 23:21:24 -0800] decorators   ERROR    Error running execute
Traceback (most recent call last):
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/decorators.py", line 120, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 236, in execute
    response = _execute_notebook(request, notebook, snippet)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 198, in _execute_notebook
    history.save()
  File "/usr/lib/hue/desktop/core/src/desktop/models.py", line 1361, in save
    super(Document2, self).save(*args, **kwargs)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 905, in _save_table
    forced_update)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/query.py", line 667, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/sql/compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
[06/Dec/2023 23:21:24 -0800] access       INFO     10.45.138.138 axy92614 - "POST /notebook/api/execute/impala HTTP/1.1" returned in 302629ms 200 137 (mem: 341mb)
[06/Dec/2023 23:21:25 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=pxk06977&startedTimeBegin=1701328885000&doAs=pxk06977  returned in 6ms 200 11 {"apps":{}}

another occurance stacktrace is the below.


[07/Dec/2023 00:33:40 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=axy92614&startedTimeBegin=1701333220000&doAs=axy92614  returned in 6ms 200 11 {"apps":{}}
[07/Dec/2023 00:33:40 -0800] access       INFO     10.45.138.138 axy92614 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 20ms 200 12 (mem: 341mb)
[07/Dec/2023 00:33:43 -0800] base         DEBUG    Selected interpreter impala interface=hiveserver2 compute=None
[07/Dec/2023 00:33:43 -0800] dbms         DEBUG    Query via ini impala
[07/Dec/2023 00:33:43 -0800] dbms         DEBUG    Query Server: {'dialect': 'impala', 'SESSION_TIMEOUT_S': 900, 'QUERY_TIMEOUT_S': 900, 'server_name': 'impala', 'transport_mode': 'socket', 'server_host': 'us1ndlm001-vip09018.corpnet2.com', 'querycache_rows': 50000, 'server_port': 21050, 'use_sasl': True, 'auth_password_used': False, 'http_url': 'https://us1ndlm001-vip09018.corpnet2.com:21050', 'impersonation_enabled': True, 'auth_username': 'hue', 'principal': 'impala/us1ndlm001-vip09018.corpnet2.com'}
[07/Dec/2023 00:33:43 -0800] dbms         DEBUG    Query Server: {'dialect': 'impala', 'SESSION_TIMEOUT_S': 900, 'QUERY_TIMEOUT_S': 900, 'server_name': 'impala', 'transport_mode': 'socket', 'server_host': 'us1ndlm001-vip09018.corpnet2.com', 'querycache_rows': 50000, 'server_port': 21050, 'use_sasl': True, 'auth_password_used': False, 'http_url': 'https://us1ndlm001-vip09018.corpnet2.com:21050', 'impersonation_enabled': True, 'auth_username': 'hue', 'principal': 'impala/us1ndlm001-vip09018.corpnet2.com'}
[07/Dec/2023 00:33:43 -0800] thrift_util  DEBUG    Thrift client <desktop.lib.thrift_util.ConnectionPooler object at 0x7f4585768810> got connection 39 after 0.00 seconds
[07/Dec/2023 00:33:43 -0800] thrift_util  DEBUG    Thrift client <desktop.lib.thrift_util.ConnectionPooler object at 0x7f4585768810> got connection 39 after 0.00 seconds
[07/Dec/2023 00:33:43 -0800] thrift_util  DEBUG    Thrift call: <class 'ImpalaService.ImpalaHiveServer2Service.Client'>.GetOperationStatus(args=(TGetOperationStatusReq(operationHandle=TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret=5446d892674f8d9a:1cb60f20dbacd5b5, guid=2f42297b408ab9f4:a9660eaf00000000)), getProgressUpdate=None),), kwargs={})
[07/Dec/2023 00:33:43 -0800] thrift_util  DEBUG    Thrift call: <class 'ImpalaService.ImpalaHiveServer2Service.Client'>.GetOperationStatus(args=(TGetOperationStatusReq(operationHandle=TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret=5446d892674f8d9a:1cb60f20dbacd5b5, guid=2f42297b408ab9f4:a9660eaf00000000)), getProgressUpdate=None),), kwargs={}) returned in 2ms: TGetOperationStatusResp(status=TStatus(errorCode=None, errorMessage='Invalid or unknown query handle: 2f42297b408ab9f4:a9660eaf00000000.\n', sqlState='HY000', infoMessages=None, statusCode=3), errorCode=None, operationCompleted=None, progressUpdateResponse=None, errorMessage=None, taskStatus=None, operationState=None, hasResultSet=None, sqlState=None, numModifiedRows=None, operationStarted=None)
[07/Dec/2023 00:33:43 -0800] decorators   ERROR    Error running check_status
Traceback (most recent call last):
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/decorators.py", line 120, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 260, in check_status
    response = _check_status(request, notebook=notebook, snippet=snippet, operation_id=operation_id)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 274, in _check_status
    response['query_status'] = get_api(request, snippet).check_status(notebook, snippet)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/connectors/hiveserver2.py", line 113, in decorator
    raise QueryError(message)
QueryError: Invalid or unknown query handle: 2f42297b408ab9f4:a9660eaf00000000.

[07/Dec/2023 00:33:43 -0800] access       INFO     10.44.58.183 hxf78985 - "POST /notebook/api/check_status HTTP/1.1" returned in 534ms 200 97 (mem: 341mb)
[07/Dec/2023 00:33:45 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=rf544555&startedTimeBegin=1701333225000&doAs=rf544555  returned in 7ms 200 11 {"apps":{}}
[07/Dec/2023 00:33:45 -0800] access       INFO     10.217.45.86 rf544555 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 23ms 200 12 (mem: 341mb)
[07/Dec/2023 00:33:49 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=ss408403&startedTimeBegin=1701333229000&doAs=ss408403  returned in 6ms 200 11 {"apps":{}}
[07/Dec/2023 00:33:49 -0800] access       INFO     10.44.42.3 ss408403 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 19ms 200 12 (mem: 341mb)
[07/Dec/2023 00:34:00 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=pxk06977&startedTimeBegin=1701333240000&doAs=pxk06977  returned in 11ms 200 11 {"apps":{}}
[07/Dec/2023 00:34:00 -0800] access       INFO     10.44.41.65 pxk06977 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 32ms 200 12 (mem: 341mb)
[07/Dec/2023 00:34:05 -0800] decorators   ERROR    Error running execute
Traceback (most recent call last):
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/decorators.py", line 120, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 236, in execute
    response = _execute_notebook(request, notebook, snippet)
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 198, in _execute_notebook
    history.save()
  File "/usr/lib/hue/desktop/core/src/desktop/models.py", line 1361, in save
    super(Document2, self).save(*args, **kwargs)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 905, in _save_table
    forced_update)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/query.py", line 667, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/sql/compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
[07/Dec/2023 00:34:05 -0800] access       INFO     10.44.58.183 hxf78985 - "POST /notebook/api/execute/impala HTTP/1.1" returned in 306089ms 200 137 (mem: 341mb)
[07/Dec/2023 00:34:06 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=hxf78985&startedTimeBegin=1701333246000&doAs=hxf78985  returned in 8ms 200 11 {"apps":{}}
[07/Dec/2023 00:34:06 -0800] access       INFO     10.44.58.183 hxf78985 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 27ms 200 12 (mem: 341mb)
[07/Dec/2023 00:34:08 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=hxn24009&startedTimeBegin=1701333248000&doAs=hxn24009  returned in 8ms 200 11 {"apps":{}}
[07/Dec/2023 00:34:08 -0800] access       INFO     10.44.50.255 hxn24009 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 23ms 200 12 (mem: 341mb)
[07/Dec/2023 00:34:11 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=axy92614&startedTimeBegin=1701333251000&doAs=axy92614  returned in 6ms 200 11 {"apps":{}}
[07/Dec/2023 00:34:11 -0800] access       INFO     10.45.138.138 axy92614 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 18ms 200 12 (mem: 341mb)
[07/Dec/2023 00:34:17 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=rf544555&startedTimeBegin=1701333257000&doAs=rf544555  returned in 8ms 200 11 {"apps":{}}
[07/Dec/2023 00:34:17 -0800] access       INFO     10.217.45.86 rf544555 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 21ms 200 12 (mem: 341mb)
[07/Dec/2023 00:34:20 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=ss408403&startedTimeBegin=1701333260000&doAs=ss408403  returned in 6ms 200 11 {"apps":{}}
[07/Dec/2023 00:34:20 -0800] access       INFO     10.44.42.3 ss408403 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 22ms 200 12 (mem: 341mb)
[07/Dec/2023 00:34:31 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=pxk06977&startedTimeBegin=1701333271000&doAs=pxk06977  returned in 7ms 200 11 {"apps":{}}
[07/Dec/2023 00:34:31 -0800] access       INFO     10.44.41.65 pxk06977 - "POST /jobbrowser/jobs/ HTTP/1.1" returned in 25ms 200 12 (mem: 341mb)
[07/Dec/2023 00:34:37 -0800] resource     DEBUG    GET None https://us1salxhpm0031.corpnet2.com:8090/ws/v1/cluster/apps?finalStatus=UNDEFINED&limit=1000&user.name=hue&user=hxf78985&startedTimeBegin=1701333277000&doAs=hxf78985  returned in 7ms 200 11 {"apps":{}}

Hue version

Open source release 4.10

Answered by basapuram-kumar

Jun 21, 2024

The below configs worked to overcome this issue.
"serv_conn_timeout" in hue

innodb_lock_wait_timeout=300
innodb_rollback_on_timeout=ON
wait_timeout=28800
interactive_timeout=28800

Changed the DB host

Increase the Mem_limit (80 GB -> 100 GB) in Impala

The above are helped to fix the issue. Thanks.

View full answer

bjornalm · 2023-12-11T13:26:57Z

bjornalm
Dec 11, 2023
Maintainer

Hi @basapuram-kumar and thanks for reporting. @Harshg999 can you take a look here since you already worked investigated #3035

0 replies

2024-01-11T01:49:17Z

github-actions[bot]
bot Jan 11, 2024

This issue is stale because it has been open 30 days with no activity and is not labeled "Prevent stale". Remove "stale" label or comment or this will be closed in 10 days.

0 replies

basapuram-kumar · 2024-01-11T10:51:11Z

basapuram-kumar
Jan 11, 2024
Author

Hello @Harshg999,
Have you had an opportunity to review the issue related to the 'atomic block'? Your insights on this matter would be greatly appreciated. Thank you.

0 replies

Harshg999 · 2024-01-31T11:45:00Z

Harshg999
Jan 31, 2024
Collaborator

Hey @basapuram-kumar, this issue is a bit hard to reproduce and gut feeling is that maybe this depends on how the db instance is configured.

If the links to the previous similar issue didn't help you out, then can you try using latest Hue version to see if that helps? You're using py2 + Django1.11 based Hue, try using latest py3.8 + Django3.2 Hue if possible.

0 replies

2024-03-02T01:46:14Z

github-actions[bot]
bot Mar 2, 2024

This issue is stale because it has been open 30 days with no activity and is not labeled "Prevent stale". Remove "stale" label or comment or this will be closed in 10 days.

0 replies

basapuram-kumar · 2024-06-21T10:43:37Z

basapuram-kumar
Jun 21, 2024
Author

The below configs worked to overcome this issue.
"serv_conn_timeout" in hue

innodb_lock_wait_timeout=300
innodb_rollback_on_timeout=ON
wait_timeout=28800
interactive_timeout=28800

Changed the DB host

Increase the Mem_limit (80 GB -> 100 GB) in Impala

The above are helped to fix the issue. Thanks.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block #3777

Uh oh!

{{title}}

Uh oh!

Replies: 6 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block #3777

Uh oh!

basapuram-kumar Dec 8, 2023

Is there an existing issue for this?

Description

Steps To Reproduce

Logs

Hue version

Replies: 6 comments

Uh oh!

bjornalm Dec 11, 2023 Maintainer

Uh oh!

github-actions[bot] bot Jan 11, 2024

Uh oh!

basapuram-kumar Jan 11, 2024 Author

Uh oh!

Harshg999 Jan 31, 2024 Collaborator

Uh oh!

github-actions[bot] bot Mar 2, 2024

Uh oh!

basapuram-kumar Jun 21, 2024 Author

basapuram-kumar
Dec 8, 2023

bjornalm
Dec 11, 2023
Maintainer

github-actions[bot]
bot Jan 11, 2024

basapuram-kumar
Jan 11, 2024
Author

Harshg999
Jan 31, 2024
Collaborator

github-actions[bot]
bot Mar 2, 2024

basapuram-kumar
Jun 21, 2024
Author