diff --git a/.gitattributes b/.gitattributes index 94d9dd83..dea1f140 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1 +1,3 @@ -infra/ansible-docker/group_vars/all/vault.yml diff=ansible-vault +infra/ansible/group_vars/all/vault.yml diff=ansible-vault +infra/ansible/inventories/dev/group_vars/all/vault.yml diff=ansible-vault +infra/ansible/inventories/qa/group_vars/all/vault.yml diff=ansible-vault diff --git a/.github/workflows/warehouses_e2e_tests.yml b/.github/workflows/warehouses_e2e_tests.yml index 4a6b309c..97b9b5db 100644 --- a/.github/workflows/warehouses_e2e_tests.yml +++ b/.github/workflows/warehouses_e2e_tests.yml @@ -32,10 +32,10 @@ jobs: uses: actions/checkout@v6 - name: Bring up Docker Compose services - uses: hoverkraft-tech/compose-action@v2.3.0 + uses: hoverkraft-tech/compose-action@v2.5.0 with: compose-file: infra/local/docker-compose.yml - up-flags: --quiet-pull --wait --wait-timeout 300 + up-flags: --wait --wait-timeout 300 down-flags: --volumes --remove-orphans - name: Add adp-router to /etc/hosts diff --git a/AGENTS.md b/AGENTS.md index 91fcf85d..d31b2624 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,8 +33,8 @@ common developer tasks. ## Cloud deployment -- Use the Ansible playbooks in `infra/ansible-docker/` together with the - `inventory*.yml` files. See the `infra/ansible-docker/readme.md` for role and +- Use the Ansible playbooks in `infra/ansible/` together with the + `inventories/**` files. See the `infra/ansible/readme.md` for role and variable guidance. ## Pull Request Guidelines @@ -55,7 +55,7 @@ common developer tasks. - Docker resource issues: the local compose stack can be resource heavy. Ensure Docker Desktop has enough CPU/memory. - Ansible role errors: ensure you have required galaxy roles (see - `infra/ansible-docker/ansible-galaxy-requirements.yaml`) and the correct Python + `infra/ansible/ansible-galaxy-requirements.yaml`) and the correct Python and Ansible versions installed. ## Where to go next diff --git a/docs-devel/deployment/index.md b/docs-devel/deployment/index.md index 9ecdfed7..72361c4b 100644 --- a/docs-devel/deployment/index.md +++ b/docs-devel/deployment/index.md @@ -9,78 +9,31 @@ _It is not yet a production-grade, HA system._ There are several prerequisites steps required before deployment can begin. Please read [them here](./prerequisites.md). -## Ansible +## Provision VMs -Ansible playbooks in `/ansible-docker/playbooks` control the deployment -of the system. All commands in this section assume that the current working directory -is `infra/ansible-docker`. +Choose the environment you are configuring for, `dev` or `qa`, and provision the cloud resources: -### Networking - -Create the private VM network: - -```sh -ansible-playbook playbooks/cloud/private_network_create.yml -``` - -Create a node for Traefik (also acts as an SSH jump node): - -```sh -ansible-playbook \ - -e openstack_cloud_name= \ - -e openstack_key_name= \ - -e vm_config_file=$PWD/playbooks/traefik/group_vars/traefik.yml - playbooks/cloud/vm_create.yml +```bash +> cd infra/ansible/terraform +> tofu init +> tofu plan -var-file -var cloud_name= +> tofu apply -var-file -var cloud_name= ``` -where _cloud\_name_ and _ssh\_key_ are described in the [prerequisites section](./prerequisites.md#openstack-api--vm-credentials). -Take a note of the new node ip address and create a new inventory file: +Move the newly generated inventory `.ini` file to `infra/ansible/inventories/`. -```sh -cp inventory-sample.yml inventory.yml -``` +## Services -Fill in the new Traefik ip address and deploy Traefik: +Deploy the services using Ansible: -```sh -ansible-playbook -i inventory.yml playbooks/traefik/deploy.yml +```bash +> cd infra/ansible +> ansible-playbook -i inventories//inventory.ini site.yml ``` -Once deployed check the Traefik dashboard is available at `https:///traefik/dashboard/.` -The passwords are in Keeper. - -### Services - -Now we deploy the remaining services. The deployment order is important as some -services depend on others being available. Each service has a single VM with the -exception of Superset that hosts multiple instances on a single node. - -First create the VMs: - -```sh -for svc in keycloak lakekeeper trino elt; do - ansible-playbook \ - -e openstack_cloud_name= \ - -e openstack_key_name= \ - -e vm_config_file=$PWD/playbooks/$svc/group_vars/$svc.yml - playbooks/cloud/vm_create.yml -done -# Now Superset -ansible-playbook \ - -e openstack_cloud_name= \ - -e openstack_key_name= \ - -e vm_config_file=$PWD/playbooks/superset/vm_vars.yml - playbooks/cloud/vm_create.yml -``` - -Gather the new ip addresses of each VM and fill in the appropriate section of the new `inventory.yml` created above. - -Now deploy the services: - -```sh -for svc in keycloak lakekeeper trino elt superset; do - ansible-playbook -i inventory.yml playbooks/$svc/deploy.yml -done -``` +Once deployed the services are available at: -Superset should be available at `https:///workspace/accelerator`. +- Keycloak: /iceberg> +- Lakekeeper: /authn> +- Superset instances: + - /workspace/accelerator> diff --git a/docs-devel/deployment/prerequisites.md b/docs-devel/deployment/prerequisites.md index 0298e222..b1917698 100644 --- a/docs-devel/deployment/prerequisites.md +++ b/docs-devel/deployment/prerequisites.md @@ -2,6 +2,7 @@ The following resources are required before deployment can proceed: +- [OpenTofu](#terraformopentofu) - [Python environment configured for running Ansible](#python-environment) - [Ansible vault password](#ansible-vault) - [Openstack clouds.yaml](#openstack-api--vm-credentials) @@ -9,6 +10,18 @@ The following resources are required before deployment can proceed: - [Object storage](#object-store) - [Networking](#networking) +## Terraform/OpenTofu + +[Terraform](https://developer.hashicorp.com/terraform) is used to provision resources on an Openstack +cloud. See the resources definitions in [terraform](../../infra/ansible/terraform/). + +*Note: Terraform no longer has an open-source license. [OpenTofu](https://opentofu.org/) is a* +*drop-in replacement supported by CNCF, the `terraform` command can be replaced by `tofu`* +*wherever it appears in external documentation.* + +- Install OpenTofu using their documented method for your platform: +- Run `tofu init` in the `../../infra/ansible/terraform/` directory. + ## Python environment Ansible requires a Python environment. These instructions assume the use of the @@ -46,40 +59,40 @@ stored locally in a `/ansible/.vault_pass` file. **Do not share this ## Manila share -_Used for: Persistent storage for running system services, e.g. database data. Not used for user data._ +*Used for: Persistent storage for running system services, e.g. database data. Not used for user data.* A Manila/CephFS share of at least 5TB is required. Once a quota has been assigned to the project: -- Create a new share, under _Project->Share->Shares_, and mark it private. -- Click on the share, make note of the _Export locations.Path_ value. +- Create a new share, under *Project->Share->Shares*, and mark it private. +- Click on the share, make note of the *Export locations.Path* value. - Edit the `vault_cephfs_export_path` variable to match the value from the previous step. -- On the main _Shares_ page click the down arrow on the side of the _EDIT SHARE_ - button and go to _Manage Rules_. +- On the main *Shares* page click the down arrow on the side of the *EDIT SHARE* + button and go to *Manage Rules*. - Add a new rule and once created make note of the _Access Key` value. - Edit the `vault_cephfs_access_secret` variable to match the value from the previous step. ## Object store -_Used for: Persistent storage of user data._ +*Used for: Persistent storage of user data.* This is currently expected to be configured to use the Echo object store. The S3 endpoint is configured through the `s3_endpoint` ansible variable -in `/ansible/group_vars/all/s3.yml`. +in [infra/ansible](inventories/qa/group_vars/all/all.yml). An access key and secret are configured in the vault. They cannot be managed through the Openstack web interface, instead new keys and secrets are created using the `openstack ec2 credentials` command. -In the `/ansible` directory run `uv run openstack --os-cloud= ec2 credentials create` +In the [infra/ansible](../../infra/ansible) directory run `uv run openstack --os-cloud= ec2 credentials create` to create a new access key/secret pair. Update the Ansible vault accordingly. ## Networking -A floating IP is required for the Traefik load balancer node. - -Using the web interface create one from _Project->Network->Floating IPs_, using _ALLOCATE IP TO PROJECT_, ensuring -a description is provided. +Requirements: -Update the value of `openstack_reverse_proxy_fip` in `/ansible/group_vars/all/openstack.yml`. -The `openstack_reverse_proxy_fip` value must match the value configured -in the DNS record for the domain defined in `/ansible/group_vars/all/domains.yml` +- floating IP: + - Using the web interface create one from *Project->Network->Floating IPs*, + using *ALLOCATE IP TO PROJECT*, ensuring a description is provided. + - Place the value in the Terraform [environments tf vars file](../../infra/ansible/terraform/environments). +- DNS record pointing at the above floating IP + - Place the value in [inventories/dev/group_vars/all/all.yml](../../infra/ansible/inventories/dev/group_vars/all/all.yml). diff --git a/docs-devel/index.md b/docs-devel/index.md index 7ef6b848..ad39c351 100644 --- a/docs-devel/index.md +++ b/docs-devel/index.md @@ -24,7 +24,7 @@ the future. ├── elt-common/ # Reusable Python package with common ETL/ELT helpers used by the warehouses ├── docs/ # User and developer documentation using MkDocs. See `docs/src` for content used in the published docs site. ├── infra/ -│ ├── ansible-docker/ # Ansible playbooks/roles to deploy the system to the STFC (OpenStack) cloud. +│ ├── ansible/ # Ansible playbooks/roles to deploy the system to the STFC (OpenStack) cloud. │ ├── container-images/ # Container definitions deployed services │ └── local/ # docker-compose configuration for a local development environment and end-to-end CI tests. └── warehouses/ # One subdirectory per (Lakekeeper) warehouse. Each contains ELT code to extract, transform and load data from external sources into Iceberg tables. diff --git a/elt-common/src/elt_common/iceberg/maintenance/__init__.py b/elt-common/src/elt_common/iceberg/maintenance/__init__.py index c4dd46c7..9b8c3295 100644 --- a/elt-common/src/elt_common/iceberg/maintenance/__init__.py +++ b/elt-common/src/elt_common/iceberg/maintenance/__init__.py @@ -2,15 +2,29 @@ from typing import Sequence import click -from elt_common.iceberg.trino import TrinoCredentials, TrinoQueryEngine +from elt_common.iceberg.trino import TrinoQueryEngine +from pydantic_settings import BaseSettings, SettingsConfigDict -ENV_PREFIX = "ELT_COMMON_ICEBERG_MAINT_TRINO_" + +ENV_PREFIX = "" LOG_FORMAT = "%(asctime)s:%(module)s:%(levelname)s:%(message)s" LOG_FORMAT_DATE = "%Y-%m-%dT%H:%M:%S" LOGGER = logging.getLogger(__name__) +class TrinoCredentials(BaseSettings): + model_config = SettingsConfigDict(env_prefix=ENV_PREFIX) + + host: str + port: str + catalog: str + user: str | None + password: str | None + http_scheme: str = "https" + verify: bool = True + + class IcebergTableMaintenaceSql: """See https://trino.io/docs/current/connector/iceberg.html#alter-table-execute""" @@ -66,7 +80,8 @@ def cli(table: Sequence[str], retention_threshold: str, log_level: str): ) LOGGER.setLevel(log_level) - trino = TrinoQueryEngine(TrinoCredentials.from_env(ENV_PREFIX)) + trino_creds = TrinoCredentials() # type: ignore + trino = TrinoQueryEngine(**trino_creds.model_dump(mode="python")) iceberg_maintenance = IcebergTableMaintenaceSql(trino) if not table: diff --git a/elt-common/src/elt_common/iceberg/trino.py b/elt-common/src/elt_common/iceberg/trino.py index a948de46..c0132207 100644 --- a/elt-common/src/elt_common/iceberg/trino.py +++ b/elt-common/src/elt_common/iceberg/trino.py @@ -1,7 +1,5 @@ import contextlib -import dataclasses import logging -import os import re from typing import Sequence @@ -15,33 +13,6 @@ LOGGER = logging.getLogger(__name__) -@dataclasses.dataclass -class TrinoCredentials: - host: str - port: str - catalog: str - user: str | None - password: str | None - http_scheme: str = "https" - - @classmethod - def from_env(cls, env_prefix: str) -> "TrinoCredentials": - def _get_env(field: dataclasses.Field): - key = f"{env_prefix}{field.name.upper()}" - val = os.getenv(key) - if val is not None: - return val - elif field.default is not dataclasses.MISSING: - return field.default - elif getattr(field, "default_factory", dataclasses.MISSING) is not dataclasses.MISSING: - return field.default_factory() # type: ignore - else: - raise KeyError(f"Missing required environment variable: {key}") - - kwargs = {f.name: _get_env(f) for f in dataclasses.fields(cls)} - return cls(**kwargs) - - class TrinoQueryEngine: @property def engine(self) -> Engine: @@ -51,10 +22,19 @@ def engine(self) -> Engine: def url(self) -> str: return self._url - def __init__(self, credentials: TrinoCredentials): + def __init__( + self, + host: str, + port: str, + catalog: str, + user: str, + password: str, + http_scheme="https", + verify=True, + ): """Initlialize an object and create an Engine""" - self._url = f"trino://{credentials.host}:{credentials.port}/{credentials.catalog}" - self._engine = self._create_engine(credentials) + self._url = f"trino://{host}:{port}/{catalog}" + self._engine = self._create_engine(user, password, http_scheme=http_scheme, verify=verify) def execute(self, stmt: str, connection: Connection | None = None): """Execute a SQL statement and return the results. @@ -103,17 +83,13 @@ def validate_retention_threshold(cls, retention_threshold: str): raise ValueError(f"Invalid retention threshold format: {retention_threshold}") # private - def _create_engine(self, credentials: TrinoCredentials) -> Engine: - if credentials.user is None or credentials.password is None: + def _create_engine(self, user: str, password: str, **connect_args) -> Engine: + if user is None or password is None: auth = BasicAuthentication("trino", "") else: - auth = BasicAuthentication(credentials.user, credentials.password) + auth = BasicAuthentication(user, password) return create_engine( self.url, - connect_args={ - "auth": auth, - "http_scheme": credentials.http_scheme, - "verify": False, - }, + connect_args=dict(auth=auth, **connect_args), ) diff --git a/elt-common/tests/unit_tests/iceberg/maintenance/test_table_maintenance.py b/elt-common/tests/unit_tests/iceberg/maintenance/test_table_maintenance.py index 187e5bf5..72613a8e 100644 --- a/elt-common/tests/unit_tests/iceberg/maintenance/test_table_maintenance.py +++ b/elt-common/tests/unit_tests/iceberg/maintenance/test_table_maintenance.py @@ -1,10 +1,13 @@ from typing import Dict import re - from click.testing import CliRunner -from elt_common.iceberg.trino import TrinoCredentials, TrinoQueryEngine -from elt_common.iceberg.maintenance import cli, IcebergTableMaintenaceSql +from elt_common.iceberg.maintenance import ( + cli, + IcebergTableMaintenaceSql, + TrinoCredentials, +) +from elt_common.iceberg.trino import TrinoQueryEngine import pytest from pytest_mock import MockerFixture @@ -40,8 +43,19 @@ def test_iceberg_maintenance_commands_run_expected_trino_alter_table_command( def test_iceberg_maintenance_cli_runs_successfully(mocker: MockerFixture): - mock_from_env = mocker.patch.object(TrinoCredentials, "from_env", spec=TrinoCredentials) - mock_from_env.return_value = TrinoCredentials("host", "1234", "catalog", "user", "password") + mock_credentials_instance = mocker.MagicMock(spec=TrinoCredentials) + mock_credentials_instance.model_dump.return_value = { + "host": "localhost", + "port": 8080, + "catalog": "test_catalog", + "user": "test_user", + "password": "default", + } + mocker.patch.object( + TrinoCredentials, + "__new__", + return_value=mock_credentials_instance, + ) mock_trino_list_tables = mocker.patch.object( TrinoQueryEngine, "list_iceberg_tables", spec=TrinoQueryEngine ) @@ -54,7 +68,7 @@ def test_iceberg_maintenance_cli_runs_successfully(mocker: MockerFixture): assert result.stderr == "" assert result.exit_code == 0 - mock_from_env.assert_called_once() + mock_credentials_instance.model_dump.assert_called_once() # 4 calls per table, 1 per routine assert mock_trino_execute.call_count == 8 diff --git a/elt-common/tests/unit_tests/iceberg/test_trino.py b/elt-common/tests/unit_tests/iceberg/test_trino.py index 5af81d49..ecfc2a56 100644 --- a/elt-common/tests/unit_tests/iceberg/test_trino.py +++ b/elt-common/tests/unit_tests/iceberg/test_trino.py @@ -8,9 +8,10 @@ def test_trino_query_engine_list_tables_returns_only_iceberg_tables( mocker: MockerFixture, ): - mock_creds = mocker.MagicMock() mocker.patch.object(TrinoQueryEngine, "_create_engine", mocker.MagicMock()) - trino = TrinoQueryEngine(mock_creds) + trino = TrinoQueryEngine( + host="localhost", port="8088", catalog="default", user="test_user", password="default" + ) trino_execute_spy = mocker.spy(trino, "execute") trino.list_iceberg_tables() diff --git a/infra/ansible-docker/README.md b/infra/ansible-docker/README.md deleted file mode 100644 index 542d5850..00000000 --- a/infra/ansible-docker/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# Ansible Docker - -The playbooks defined here define the deployment of the system. - -See [deployment docs](../../docs-devel/src/deployment/index.md) for further information. diff --git a/infra/ansible-docker/group_vars/all/ceph-mount.yml b/infra/ansible-docker/group_vars/all/ceph-mount.yml deleted file mode 100644 index e57636e8..00000000 --- a/infra/ansible-docker/group_vars/all/ceph-mount.yml +++ /dev/null @@ -1,5 +0,0 @@ ---- -cephfs_mount_export_path: "{{ vault_cephfs_export_path }}" -cephfs_mount_mount_path: "/mnt/lakehouse_data" -cephfs_mount_access_name: isis -cephfs_mount_access_secret: "{{ vault_cephfs_access_secret }}" diff --git a/infra/ansible-docker/group_vars/all/docker.yml b/infra/ansible-docker/group_vars/all/docker.yml deleted file mode 100644 index a4aa9741..00000000 --- a/infra/ansible-docker/group_vars/all/docker.yml +++ /dev/null @@ -1,7 +0,0 @@ ---- -docker_daemon_options: - mtu: 1450 - registry-mirrors: ["https://dockerhub.stfc.ac.uk"] - -docker_users: - - "{{ openstack_vm_user }}" diff --git a/infra/ansible-docker/group_vars/all/domains.yml b/infra/ansible-docker/group_vars/all/domains.yml deleted file mode 100644 index 035d1af6..00000000 --- a/infra/ansible-docker/group_vars/all/domains.yml +++ /dev/null @@ -1,2 +0,0 @@ ---- -top_level_domain: analytics.isis.cclrc.ac.uk diff --git a/infra/ansible-docker/group_vars/all/keycloak.yml b/infra/ansible-docker/group_vars/all/keycloak.yml deleted file mode 100644 index c4d62bf0..00000000 --- a/infra/ansible-docker/group_vars/all/keycloak.yml +++ /dev/null @@ -1,8 +0,0 @@ ---- -keycloak_http_port: 8080 -keycloak_http_management_port: 9000 -keycloak_base_path: /auth -keycloak_url: "https://{{ top_level_domain }}{{ keycloak_base_path }}" -keycloak_realm: "isis" -keycloak_realm_url_isis: "{{ keycloak_url }}/realms/{{ keycloak_realm }}" -keycloak_token_endpoint_url_isis: "{{ keycloak_realm_url_isis }}/protocol/openid-connect/token" diff --git a/infra/ansible-docker/group_vars/all/lakekeeper.yml b/infra/ansible-docker/group_vars/all/lakekeeper.yml deleted file mode 100644 index 8f6fd23b..00000000 --- a/infra/ansible-docker/group_vars/all/lakekeeper.yml +++ /dev/null @@ -1,16 +0,0 @@ ---- -lakekeeper_http_port: 8181 -lakekeeper_base_path: /iceberg -lakekeeper_url: "https://{{ top_level_domain }}{{ lakekeeper_base_path }}" - -lakekeeper_catalog: - uri: "{{ lakekeeper_url }}/catalog" - warehouses: - accelerator: - storage: - bucket_name: isis-dataarchitecture-prototypes-warehouse-accelerator - key_prefix: "" - # management: - # storage: - # bucket_name: isis-dataarchitecture-prototypes-warehouse-management - # key_prefix: "" diff --git a/infra/ansible-docker/group_vars/all/openstack.yml b/infra/ansible-docker/group_vars/all/openstack.yml deleted file mode 100644 index 5d698b87..00000000 --- a/infra/ansible-docker/group_vars/all/openstack.yml +++ /dev/null @@ -1,18 +0,0 @@ ---- -openstack_project_name: ISIS-DataArchitecture-Prototypes -openstack_image: ubuntu-noble-24.04-nogui -openstack_python_interpreter: /usr/bin/python3 -openstack_vm_user: ubuntu - -openstack_virtual_network: - name: lakehouse - subnets: - - name: lakehouse-1 - cidr: 192.168.42.0/24 - gateway_ip: 192.168.42.1 - -openstack_reverse_proxy_fip: 130.246.214.124 - -openstack_apt_keys: - - url: https://dl.igtf.net/distribution/igtf/current/GPG-KEY-EUGridPMA-RPM-4 - filename: eugridpma-rpm-4.asc diff --git a/infra/ansible-docker/group_vars/all/python.yml b/infra/ansible-docker/group_vars/all/python.yml deleted file mode 100644 index 43ad4992..00000000 --- a/infra/ansible-docker/group_vars/all/python.yml +++ /dev/null @@ -1,4 +0,0 @@ ---- -pip_install_packages: - - name: docker - extra_args: --break-system-packages diff --git a/infra/ansible-docker/group_vars/all/stfc.yml b/infra/ansible-docker/group_vars/all/stfc.yml deleted file mode 100644 index 62ff6234..00000000 --- a/infra/ansible-docker/group_vars/all/stfc.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -stfc_ldap_server: "{{ vault_stfc_ldap_server }}" -stfc_ca_cert: "{{ vault_stfc_ca_cert }}" diff --git a/infra/ansible-docker/group_vars/all/superset.yml b/infra/ansible-docker/group_vars/all/superset.yml deleted file mode 100644 index 1210557a..00000000 --- a/infra/ansible-docker/group_vars/all/superset.yml +++ /dev/null @@ -1,5 +0,0 @@ -superset_url_prefix: /workspace -superset_instances: - superset_accelerator: - app_root: "{{ superset_url_prefix }}/accelerator" - http_port: 8088 diff --git a/infra/ansible-docker/group_vars/all/trino.yml b/infra/ansible-docker/group_vars/all/trino.yml deleted file mode 100644 index 46d79cc7..00000000 --- a/infra/ansible-docker/group_vars/all/trino.yml +++ /dev/null @@ -1,11 +0,0 @@ ---- -trino_http_port: 8060 -trino_https_port: 8444 - -trino_superset_username: superset -trino_ingestion_username: ingestion -trino_users: - - username: "{{ trino_superset_username }}" - password: "{{ vault_trino_superset_passwd }}" - - username: "{{ trino_ingestion_username }}" - password: "{{ vault_trino_ingestion_passwd }}" diff --git a/infra/ansible-docker/group_vars/all/vault.yml b/infra/ansible-docker/group_vars/all/vault.yml deleted file mode 100644 index e38a8243..00000000 --- a/infra/ansible-docker/group_vars/all/vault.yml +++ /dev/null @@ -1,873 +0,0 @@ -$ANSIBLE_VAULT;1.1;AES256 -38363061386537313532303139313463656366323266303638353532396133373539636230356661 -6433373562646664363533376562383263383462626264660a643436303662633138633436316363 -30343761636363396635616663303066393965346165636663323036303566303934646265613038 -3864353135623737310a383464326434333366643938623439393564643335373764663561316262 -39363165306166343338373231343764396131323132366336333966303066303338393065663062 -65633534656234636263396134343434353561653836393039626637356365323666633135306165 -61323564343636313061613232316264376265326230373463333561633036383732353162383362 -38356430303836343530303766363532646565393961663162646139316136396262646138633932 -66323233396639636565663536373736393139343135616464323862636366653333323765333266 -38656563653139316433323132323530346433363238336532356661316162313066646235623164 -65386665386634623336643938616162363763666338393931353339343262633632613634313232 -37643465636262313638396664343432373037383266363465316331643863613634643430646639 -61366234666232633038383463656131636638343339663461333163626634383562376361326362 -38336166653039323064623261313138303562336133383530316361333739383737636638366330 -64333730323561306465383437643064633631366636373963323261633233633662666530396338 -38643165633231336439656366343836653633643831346662653938663636363661333431653236 -30306336656639653038396466366163386262363937356562616234663236643262646336303937 -63326432326133636635366264353763376665366233626334353665653534303861626262383538 -33663664373830313833363539333663643065313864613865663432393437393264316165353630 -33316537623337316665363361356661313563636237626463646433383135393834636564643965 -33346164333338643935333232613561323766313062303262336366316563343033363036643231 -36633237386231313261386462346333336464326363393330616564623931666436326231343237 -32343266336536333938386535366234353330313034313064636534326239353162633332336434 -62633331386637313538316366623133356533653731366364336465343862313161306362373237 -37343630336133396139373366383530306535633833306562643239643661626437363962386630 -65396264363632353331623638356239316466393732643833326339306235383265653164633465 -62336461633565343731313861643735663534343533616436383331353731353165366238653330 -61303163373038613033343334373031306536663638663532646162646464326532616530386165 -33656534636336313964333165323761316663313636393238393831643364663233656661613438 -38396536633061626332333235383835626639663732363836343566653731626266663863393834 -35623062646461343063623232643039656161366261366638396134366161616230313537333131 -66643533636439623766393163396431623164623735633532303936633665626561366632356366 -61316164363063663637326236323236343733636534313664323834343063323862623535653733 -36313564346233316435313939363262313139623935366638316565336336326633303232313837 -33323039396263353731303663613231633830333361343031313736616466653036663637346661 -61633032356339333236303339343830343065663736336538346333663162323761323965396431 -63313638313936346635323361303361386166393436373063656334363762303035303064643136 -62353633303431373931653036336239386566373635646636613765366534343763333633323162 -66343731343439633766353533323637383062343032343832343061306237643936653965313563 -61656236366438623865376435353765366534343133393336316538306461646237613964306433 -64313366636562366336386437393634353935653764333635336666613932343265366535346136 -31363331633233613361653631343239333034633637663330643734333738386135666666353761 -66613962363336383732646436666333623163393639653663666330393564616166353335666436 -33623939373566363764623738363831613864363965306563653738336164373738326538323332 -63306234303635346138646537326562383162383235653932363737336632376436326236616234 -65363664653030376338306366626466386432643566663136313939393730376634383737343766 -34626266336532366565353364643563393730633630336336356664313266316137623864653365 -63373065396461643839346465663935373232636165656561353839646262636662636336323066 -39353563323463373336363662313938393461613366623636323066313732353935653664353263 -39306330333430343966393266336634386239366135653932396134373661393732656466393065 -34653837316465623461633863303539366465393839646563356566663266613539333361376563 -65393162636363656563383139616266383333386238376564633839313762333361653066313937 -65313634386637383261376262326663633438303637323864373466316235623434373130376661 -61383036626238633666656338313234653538393834393764393930313838623037653765333030 -35386339656661353662316134653163313961313564313665353335663063636134623730666231 -65316439336231626236616365336330353135663361376531333932306331353033646131623939 -65643265373334366331393436343236323262666237326564663035316262383662386338373937 -65616466656635333963633431623232396363323461313532383136386666643038636364636439 -66363739303139353066663534316164366362346265643137663666646134366636323035333636 -65396133343039613636363032353634653434326664663136636236663832326232626638663164 -66376134653962343162626135383239393739656332303636306134613930316633666537326665 -36303139376530383964313164643838616365323164613931363032306162643732363939373331 -38323161323165633432383165656464326435376230303132363636353435343731313337666664 -62303336636530633334633464386461313634333635376330333639396361623539396664666465 -35316161313137383535666135646331646633346430303836306436343832336561636562616431 -39323237633164636434656532323462383032343336613130383331356139303861376564303831 -38613936303266326562643163613161356234663365396430623630333830353764306436336139 -61646265623962616239303831376132343366396131376466643434343665366162326133613763 -38313631326337336432646462363966633732366130353639376561643265633063666164326564 -61656431383462393936323063333335373763363964623864333261396132343064336261653732 -64393466336234313239643838316430646637663036336438326230313139343564663061326631 -39326665316530326332656436656363393164656563316138663065343831666265323733303365 -34326137643833313137383064633934333565613665623563303636393036343966323964623433 -63386538646338333733363733646265303366383133643633626132633734393738333233326264 -35646235616130393938343966373433366266373837383038633639643663386339376135663961 -65656632323333616262396537346136353430663434313562363438376665343261386164313239 -63306661333634653563393831343532336564626462636135616237333062613839313034333964 -31653439353530663435396333643135636361633132653537353531376630646432383565303236 -36326165343335383364376261326164346532346163343334323465363832303362663162396630 -33373834383266326562303833373534343538333436616333333131303337373265303935613762 -31333635623764623839343563633666376632386130353538383638356536353634613137356139 -33303037396564353739313634383934393463346662373730353933356561663934383664643930 -66313266356263363962616531306538363466336564643461346639316230333239363863653062 -30303833373965396164316337643331663438353363663963343163346636663264653932363938 -62313835343932383331636634313865303035633630623534613566663066313063386138626465 -66313937383930356163313465623138383437636137636437636639643364656234323966653739 -65373336353363663931336332303664396238336438386530303130336161386134363630303534 -63353230646466663066306233346139666430393232363533303333346234323266653131353264 -39326164663462633935323438333735613739636333333161346632653130386138383630373632 -64393161373063626333613830333165373561336366656439396238636165336364643731643931 -37333165316433613365666666653133323865323439323864663065326338353834663436346337 -38303230356461396165363963376639323932326230376461613866616137343965653963383831 -32653364386635623563393165373463663261303133363030373536343831333536363334613262 -39313235343962613462353034383861303233396432306466653738366439373532623131383331 -66346664613334383031623837306330346530383332323730316362663634356431373138343833 -63323232633934393937646238343335343039316635333530663339613831353961303830326263 -30663363666465353766363530323630623433663761323335643930616131373065363738396233 -63313235373533356162326631616639383830313164323762333639326365653366373136623539 -63323130656139633863613261373235663234383139633766373633643166383864653762636265 -38366532363131663438363662656564323861663535333161613235396537363632326161363438 -37666361353863623034643665626636373361643262346635383431323564636663653265353932 -37613231336630303665636662313164333434346636613562396139656537363634306264333964 -65653666653637646161396565383762653161303564323331643063373532313030313566396363 -66393461633161383339666339373035353064383564343137306161666636316562653862353231 -38323065373133333766656433653334316334626263626464643838663032356632383463303233 -33353336623831656464326331646131333633663635303064343536663361316662333263393734 -63663761303264613736643739383337393530336262376531643966353030343438653261303839 -64386436626430336366323430643866633666323233613461326333653063666539613337636138 -32373731356635316662623032386665313832623764303364616637306562386536613862613433 -31313761633365353833656161626265363065373535616463643465656563316539383635373639 -30393239313663663132353262383364346563643638633537613734643439343430626538373639 -62623732383834346233396531626632343237663332336130613536373065623132633265383463 -30386138616638306135323265353930303838653331306133303636343636653131306461303939 -34306533386231663032613361666535353239333463376163616233373761646134626439653264 -30333265643365663061333730666362333930313562366266303161373163373365396437336534 -33316336633632326431633830636230323735623535356538613338646237366437343938653536 -38383366356164336365623038373539383136386562613037663836303436346665363530623037 -64343663633262613634353963653336666331656333396437323566313661306130383739396432 -30613662373666653236656365393636366233396234386161313433383166326666316233323362 -66303163356265353536623737646538636637613331633533376636633061663262393963343966 -32393366656462666533636463653564346337386339356566323136383066663937363463633066 -34666232313266663037663362653535326566396332303939376135373230343533343436633062 -31323066303936653039646162646139303032623531643433663732383962363830353733613664 -33333631363934303062616261376336356132666534636139373037343439633437346632306366 -30643033333665383031333565306636383264313138646132343136313938356138366137343539 -37633531623064303661376532656364663161656237323232313363343265623737373935313861 -38336362393337633436393134633264653739323732316162633530643335666265383234353963 -62323535333162636363626131613566663266333634623964616130613662653832383333626534 -34363032616436616139626533396134326164373238376331383966343766663435363861633732 -35623737373465356335653164373066393630623066373035386436336630616565343666333437 -63356232353336646336333833353934633631653064376432326131373935613636616164316264 -64613062393437646336333434623962353230393934626565663837633363663365323831323332 -34333036613138346632663336636537626365396439623666383636643963303130633565393739 -30613461396164343235376336353861306262333338663634643261326231666663343835316463 -32636637323439316563343261363839623961313766353064366235343332316666396436666630 -38363265616637366230343937333662313234336336636462653838383266663430313364363830 -36386431666139306431343532326566616637326438616137636464313561363431653633386239 -35633630653933303633613962663864323265323339313338353164366434316231323236616534 -32376339323339343264393165356365383430323662663564613134333364363136323431366363 -30393165626361373137303130386531313065303464656433363837666334666534386562383461 -39336261383039643035663234363030626437336138393561353366666130386261363239386463 -61366566616432386132643965613761613036393764363035613537376338653631643965393161 -36613461303866613938396466333962386136366432343231613931333239363664396132663339 -30643939333533393835653161653762616563353535306231323033653737393166623931393138 -61346630613162323261663164666134383064633635396130356536373431353466353964356331 -64636133636137333963383165376234363039343731373538633534363762643835313839356538 -61356434313631653239633862376136303830353133343765353730313234636538343533323030 -37373963646265373535633436376162636262643463626466393161346231353231353661636366 -64366366653462623635626532663231646532613732653039636335353236653166326437613635 -38636637356134626132313438663233633730356534313236306333643964343937353733393366 -35623365353561353338663162326333363333336566373934656437303463386137623139616532 -64366666666466303066316436623435323430306136646337646239386134373761653630663632 -64383232383065336539306235336631353764653761323336616565643966323432353435356161 -32616234363166393961316636373132356439343263643738363161633439323935663735623464 -32336432656664366338373731336139346438636162653338316430373665656565616663366664 -33613562373231633838653637386238633430613435663238653438653630346363353636663032 -64623165636535653161303532383638633634303661326337623864666136633163653661656461 -32633262613237356138323632626535633839306330636265616166363763303761313833386230 -61613137616332333633653030306566306165626364633933336537643761626334623633323537 -33366465323434626236313134313630343962633636643737306664643830303765633530656233 -38383035346634326331653464626261343663666162643535663439366134383639356361616330 -61616663343266653530313331663062343435396266333333306166376138616338626436323865 -30323133623438343761306265373839306532393764386331396364363930643563366161656665 -37383332646533346336323636643035643634356562363930633430333131656130376462366138 -61306534333166393736623036396635386332663133333032316266626637336235323763636530 -62633866323561656666306666353766623335333466613937663139666131386331376539643138 -37326366643231363232343831643036356237383831373537316365663564666134346166366663 -61313432373332336463316135363566313237383266613930363531363164366464393733376536 -33303832346430363334366531666466653365376534333565613631616232653161666433353165 -66313162326635643431393966353066303266643962393737656335616238336330323435643130 -34313138613439336163396661653664656535666464313763656135363437333362643030306563 -30623234373162623934663964343863386563343165363964326664666434663366376232643362 -37356132613063646531366266613764306137326336366338646164306335383138376333623739 -61383532333562363434633234386132303836343164663935623462633463616166363265323761 -30626261626536656432313761386333613339303438643435666465316437393037316534303136 -30376535346630613133396165666131306636363965663839383031353835643931666534386436 -34323165333262383837363836646262303339353939323138623033323534373762366231363037 -33333230626234336461616133393037633433373661336266393864336432613430326466653234 -34656666333462393965333630323366663561643065363733663963633265633334623364313966 -38323836636561653237323238626564333330396633386431396662616433626530613030633034 -39303733316335313762313632356237353438313235623764346635363466363836396139366465 -35343032623035616366323530313338313238383234366231366662383032376461616465396131 -33626431633335336661353035623666363562626461353362666235396434353338376661326330 -33303263363838353430616361646163663736343839633338663831363039343335353131303830 -66313065303132396564613761356462633764313161383739383934633036613665623364396333 -31643532306131626234313135373932313235663163363762356130323164376563623966346239 -31323731353861663736383165666564353163633430346634643637343964396463613934323666 -34376562636638653363313862363435303266323161616131663264613462343761663735356130 -30393764376333316137333734383264623932353162643638323732333431633030616263356263 -36316231333739643638666630303564326631663637393666663137643236636265326337336433 -61316334613039306563353135376162633437333032333137613131313534343235303732323735 -31336639303166306137323837663835666337316334666261613564313361326632623861316261 -39636538333665616537313732383465306135626638313562386464613962326266323739343331 -37303862386535333466616135353031636534653831633932623961623665613834363936396139 -37306334363266373639626336336633303032383030616663326532393438343062613235653638 -35623832363634353634393363306237616334376335383039623366356233333633616433363831 -62373765313232366130386230383737326237393437306337396130343964313239363336346631 -35353061386534653234363461383466336538616634383265393563376434303661663135316434 -62336230666337353061303561653561626536613530323039353262323761663462623165326136 -65636330663430323531666331316532646431326138383736393330343934396263333064666261 -37323939623035653634336630346366653834306337666566366362646635653935346638386235 -32386236343963633734373566323662376339663739636130333134336334386661303039613734 -36333236376333326634313638623738396463353763333137326637323032393631613534383139 -34333062616635353065393137313464383564616261333836613031333533646365633164623961 -62643432373961386363383965663738336165333063396264343235623364333531376165333531 -31663439353130636561386430666163306563613366363938376230653434643634386531666437 -30396639303831383638316633356237346134623037316139313137626439626334633934653936 -30626330616364386630393231646461306639396663383239373064316536613263643531663237 -35303734353163343437316338323132396430643063306139653262386265363464613431373333 -66643532653439396638663561363265393863333466616564333131396363653163613332346136 -64633733623932633737623836316531323564323030613339373337393534663232623631663731 -62306663633538383161376131396131303738366265313364383466626236633263383930303433 -34316633316263626231316234623161326266383838303365363634303536373862616231303365 -37353662633565666161316531646532613533376630303137393966363965396536393639666534 -65343265643730323130313839336636316665346436356333623063353738336465623735346365 -37386332363661653661663531616464313663313139663865663239663566393538623466643363 -36313562343736623566616438363534383738383563616534383230613664616138626662383465 -35323964323364336138316636643666613462666639396332643331393464373537623730333435 -33633966633864383963653261663135643138643539346133326232386533316438333332373033 -37666332396438623366363365663930653238303430303632626631373465313863633035373835 -61666162626161646366353034333537326136343932373735613762616662323031323365653836 -31373864323065646237383132363034643439333138346139303665613235363161316532316632 -36323431623664336630356637333665313333643061313363386165383938613664373664363330 -37356162646265643561623737643133623561396634643436613162396364656666663162303966 -34643139356630613362386464353034313461663830626434613334326562386165353534613161 -31313635303033623837316632353238613738643836613433366263373866616432373966343033 -61636335636663636531303432386535363763333630636162393765383733343137663364346330 -39343830393965623536326633323830313830353033383238656235323430386432643662383939 -30623362653336393134626337373939313939376664323935653637373963663264306165326662 -65336661616130643463386530613530383739333165656263316138353230663938383330656636 -33613339363864386536663030343863333038333833303039666261306465303365333939356537 -30373766316236623961653935653633353633343939653135633532656636326333363930376632 -61613338623263636135356132356539663433626334326633613963613139656335653537353036 -38313630383065656661613834613963363637346539643261633033396438323663653630313461 -63323535373364643066656361663866353266303361323230366430613564666161303939303531 -62353838663163346635366262613739366138333636366331623363636432643635363931366563 -63646632323730666564666163306361356261303263303364393437316466623363616434323835 -31656361316361383133616661616461346262386532313661626136616563366234643230633664 -66343862303362333132613331343232636234326232373830343736643739633835333337626336 -65343737323436653566663065353634633163336134363034356363343334666131663463323336 -62356230353231303065643962383266633333356336306264323635626234373831323161623433 -65356663376439316264323465613733623062653265363639373234653834663666343133633730 -39396538363331646239336439653637353431613737316236333436656634613064643932636530 -63663630303838313936376361643234363162313436623166356537363562313062636238616632 -37346237373838656535646433643135363761643139313462303961373231373233653932363334 -65386133356165356332383839373561343336613830653831616164396563663765663062363630 -35396162626463306565613238633733383831306666626237386135623734663539666165613463 -33353234663138386265653632336263396332313931646365356465663339646530663565626562 -37303962356263356165326133346361653765303663373432353932623431346339366137333864 -62333030343639653430643634333661666335363766333263323937323834636563303237363438 -30633232353763323238376636306134663165626631326135346637326232643865383632313435 -34373736376136666565623233313364623839326562616264313434323132663164326335643939 -63643431323734373438303962366666333631346165393535646532396364633165643634316336 -66666432363030366431346538373537393534303839666231346137633438366631363134353331 -38316664336463643732306637643861326637316232386465393535373832643736386533366365 -62666163383461383062393434656537356666353334656638373863396134373139633133656463 -39326262353435326233303739636639393934353130613064623935623862666535373065616638 -61393038323262646636386361383462303764646365396566356438373938626433393431643966 -64313831376431663032343737643738643365373931636435333364616537383636383633386464 -37343939316330393532366262363632323666643033313530623339343065376331383136666131 -66356236313634366635383731363239626266653837383137366362333264396238373839356333 -35326435393030643264613236373134663365363635326633666161653862623232303736376230 -62623833303930383632613230653162646534613662633333633264623439376237653430336661 -63336232303964646132663766336237366136303838386562323166613834373933306436626233 -62356435396134363064303862373231353031316632666535363433343931613962666134306133 -33613963373366386461616632346239653735386433386631616462666432373031306239363532 -63346537363135393562663035626235373335346265383338346562663336623336353032333030 -38656665376563373232316338663133656638316562373735326335663433656365343239646635 -30653338303135333164623337643563653639363634336434373366316537356434326365326663 -62396465613161306231616163323161323366666264313664663337653536643666393633333337 -61323865323333646661313061303863393864353732363565633965353636353064303130633964 -62636563636439626238626462346664626365363862623533616634376165383138366435336435 -36343133363063326632366533303665656337306132653565613637623932663564303232396166 -31333730333563303434393063646437333935653634636464393865643034666635376239323038 -33396432346134646132303564363636663032393638373864623333386337313766383962333565 -63666139396630663561393162333239383539353961663263383863323139383632326163333138 -38623865323233346262633765373263323738326265633832373866346334316565626137343139 -64613462323138643263323364613031343962333961633063663537393030383338386230653936 -65396461646165626332373738336133313765326262656435316230303462316434663232366162 -34653238363061653537333739313038326438383565303930393934616566373836653130616438 -63343961303234323138663266366362646361343665353737326131386239396237613936386366 -61633864363263303037613237623332306530633862626132656633613562386235386434383364 -34376562623437666163363130316137623637663031356232653764306338326163363335373936 -37636236656664313538336465333233323033366133636234333630373731346334306664386230 -34363562333336626434323432646565326362643766313330373534373434343961303831336266 -64326335386336326537343031303762336330366262303738626234336134323438346633303734 -32646437373533343930396363636463363339386531373263623138653234363761316136623062 -66306332613236663363383236643464353265326664393135343335346133336232666538356636 -61313033656239613466623630653331343662636364363432383365363638373133633034323162 -62353331373563656439343736303934346138393130656634383961373661656330653031303337 -31393933383138383431323438616336653366633739343933326230326662393364363463336239 -38633261393239343931383862323136373936396534643230353933313065666163653637636133 -37313031353330356664396234613037363530393363646330626539393566636432616333373462 -39663836643137396238393165306662376434626433626463613439313265326431633933613338 -30633261336664353137613035643336643361313666343238383061323130376534343730313765 -35353130666139336237366134656230316436313836623465326164393732383566356664363865 -66623062613532666437373639366662323665396238643634646137383562663363636232626235 -36613763363533383531343639383163386466336232343961646565633461623730343438636538 -33306335313832626239623836326132623661313134323063306462353530656435396535646565 -63646465633136383636663265633139336632326135343566663861326566633263316333353866 -31633132323561616262613631623236623461366463616238336232623738313561656461666165 -31656266326238666665313938613837326632376636313166616161303031343965633666336431 -36653033333638373731373065366435393430363035383662393061366435356532633736333836 -38613232386333303865343134393038316136376636313166346235383331653734323835636563 -64306336363762623934336538356137636234346337616335383664666439366235323330336338 -34663034343835383965383066353365386533363833343530643361316266393035336561646232 -37306263373366313531633439323730646233343438306665666665356163323363343963303534 -39366335303265633939356336666336373366623361383966663062356234616135383936656237 -35303165343465356437376239666332326135333536363062643937363638623662653433333638 -38663630623336313830393565613666633133303431636432373665633464353733663134386131 -65306565373035663837623035343737313564646663333664373136656363643636323163643035 -63366239336664653839666431353366366535366162346662373662316666663263636565383065 -63393539333666663630643231656434633465393665613132613065623038346163393837313630 -64363439373137373535393865656434663739366132376433656236313230386433333566653562 -65343137343465353032643763663766366163646530373836323239613431623961343066623038 -38656431353435303466616535363664663131633162316238323630663562623033643564653430 -36383465346462626337363139363837623834326139613932343432333937303237613766373437 -31653731363861656230393737393239393536626663366135633932666132313135643036333038 -37663465336633383139316361633065646332336633646535336464666231363334393364336632 -35636630306161646438646533633465653866316634323733333465313963666535616162653761 -38646138333665396165623832636238356333623438383536306232613931363531616237366438 -66323831313066653264613061363765623831313433306566313966396235623862323830396466 -66323231646565343966646531306539343034333536336639653164323538396537326566623464 -37646531326139336362326430376437623063333864633034643638383637656437383038663165 -37633435383064646633333764633366623036353130616364323339613032313632393233393963 -31333136303365343034646265313335623730343065646430333230616134376531656133323766 -35613064333162383063393835623830356335306638666365396261336264363433343832333564 -65323135333465346633343033643033666635306562356636323233393861363563653563303137 -39356137313239373665653063373961316436353133663634643138373232313333393561313431 -38343766316439653633646134396162376366313332383838613862383430343062353937313166 -66626265326132346438643266346565303834636665363534626131353561636133343331353431 -66373239323439623534663539336664636236303961646138313036653533373835636362643039 -38613133646436623237383832336464373864393933653630656233613433333063393135623234 -36353437646138643036623663326264663938323432336430643463353563386533393133393736 -63316137343334653837383265326631306136346430643665363163316364643735666132306235 -38333365353461623964353661663764343133376335663061626664626162333266363463383663 -66356364363138313939363635613362326536613437623264313533383132646661656432636333 -31646163643963666435633833636338663030636462633166393466373334636131656665653136 -65336336626530383764343939663262326365356162323635393834646430323832333134353538 -61343739653166393032306161393662633563346262666631336332656237653235383037623431 -36383064363631333863643236353363323563356535306531313662363766636338386163343765 -35623265303635646339323937323733353236303937643765326131306631313837646238623437 -34613936363838636239376435623763643762653630333436613333313639623537613365326264 -64323038326463623963363432333339356438326630626237653330616336343735653239303437 -34353566303631616435353464663432623431666431306631646262366530303361333066663861 -38313063343231643637663162363761383831383065623435353536643364333266356261333061 -64663734333364383238393761666332306334616266306163373762336566353836346434336163 -62626266336665353363653665623963666361393861323264616135646162386264373332396432 -62616538643437313864623537663866363737616566323463343866666161353731353465363565 -65323366373631353262356161366537313837653137326338343431613630303935306266363265 -64623861623836393733626639393732646365643039323533363938646465653863373934313166 -35633236393930376363643966393637636464383266343366623265376336633364346564656664 -35316166623532613935393761396337373031306361353539346135316265626163316662383966 -35646162356462356561656336343937333864306232663061633539363539323966623236323134 -65323238626564636537656431303237363534303861366564316166303461363836333565396362 -36343565663639376138663333376164373037396161383534616534366334613630343339363830 -35346438383331623432333062656539616136323266356164306264306562386564346534656461 -35376434646433643134306365666138333133343864633935636265353863663135303337626138 -63636236316533333563653362653761356239393735663165356334316639663131323834626663 -31343136306362376633636434373736346265303864653862626166613938353831656466643431 -63653737383930663534363164393932323434666139306663346233343633353532383534346162 -36343162353361363461656130376664643666376435376361643866393230336565613139333564 -66643331343132643361323331386161393866386465343537393464326261653063393238343138 -64386538326666356663636564653135343632353131653266323630373766633763646537633337 -34386530633861393137303736353433393361393533666665383931666135616431623231616332 -62626466323666393663363136383033393136656132376239653866376332303736303539303963 -35613236623938653363316261386164313462643438373037373234303366303735373932656461 -64383830353962333038363466623730396666323431656565383636643635613232623633303538 -65363032333737316332393763613836346630646232643165313162363861636133303832333139 -32633137343230623864356339613731326563623162383130326630343031316564303864666133 -66336635376637386531373639633438616265636235333065643439306166313763373866343566 -63326532363630326635383631643435373963613733363737326234613265383464616164333864 -64613162623130363239633361643938373239356463326461616433393237666530386634616136 -61376434373439346463333035343236303339343061613836393932623736656438616638393539 -31333461616137356662663032383334393965356531333463613433323336343732623434323962 -37636365653734366532643432353537306232313032356230613334646134643832323866663935 -64353330656436333466313832643533636632623962383135623036356136373331643434373734 -31363537323861376631646434646337636466663863323835316336626337343965363030633264 -33626432656239636337366365323837643438623930636337653534333563643462636135663532 -62353535633238616265363330666235306139343934613438396536323361623766636161373862 -37356161313236373461343864626232663033313363333632373363386263323863356332633765 -31383332323636396430333961353064653035636264343430653265316165613833653135643537 -37616234343138643265663930366363346464336234623063346132613831616632653866323636 -32376362366130306363373738303631363264353564323930363631323738383161666435396633 -32343865663238636435393961663135376430333166333864663931316337393864373330383737 -65303064313363646566303537623939396462366430646461636435363637646661316535383831 -38366335316464353933373736643834396164333831376339326665616264366437313066656661 -39656362663661613935323734333239343038356434643634303365643037663735396130333134 -34663335346162663734313465633231306564616139336665613934643934613132626265373737 -33346530316236353135623330623430336263316435623435366362626332363631393934636539 -64623465623665616263386236353838343839326633303631326135656234663735303737373263 -38323132376365373939346339336538353331346135363136346139363531336636336564376162 -64323838356233343861353839383839336535306435656138626262383364313335383763326133 -63663937656534303635643530613932653462636364626137356564376138326137363433653639 -31353831333531663334643535626634616135363036363365333536303638626535373862336431 -66633331613331623731663832336566306130363033336461393039346435393062353664333636 -35316434373839666561626633653030326333373662626337333266616539636431633931393066 -31643465323165343166316539343466333634346239336432323333656336666166323863623634 -38346162653032646139643539353164353233393438376132316665343437353037663735383939 -39343333386236616538376565313432363731666634393431636563646362386362626338356363 -31383165616264366565643234656337373236623332343531313561313231366534626263666130 -34383134393135633331356430646161343739396131626466663662383032363831643830656235 -38646539343434656163666361393639653266386637613165386234343861393231346661376632 -31303739373936653039663533623066393431316333333236653363333161633566663134636532 -37393337343764363831316362316632656466323163383536643737663531633737393162306333 -34303230323831393263666338373365633661313935646564343464643231323965383363646635 -62373732356133626261663432323339633937646335623563383565303463316365646462613831 -38616161323263643234363464316466613939396633333239393832386433393265653562383564 -65366633653163333535363638323131346533653133343130353837363738383136383038633462 -35393536303565663763393165636634613432363663303438386237663330373266386261643761 -34623663366439363035626439316461303063333433373834343764306538303738333866626233 -65626339643533366561646132303262353532373537343637613861303538303564646139383663 -32616230333232626562356231666532636434383835643838343363356433613161373263646237 -33366364313235373339363238656336623239623166393135643365333336616264623233363135 -37383764653435333138643237646264336662316665373137666566323934376263313864653861 -66383839663432363338343130653936383065303462343366393830643535336164646136613532 -30303266346631643137396361323733326538333037633365623463633432626130613434356564 -30333530653561656333646230323366366162326232353565356531323935343866383233323934 -39623766656239626135363136363137346539666538626435616533376131613566363462323833 -64363731303063373461653361386435643735613161356536623332356164326237333934653339 -63346238353830353962613365326631653538666264373764393537363136373336666366393539 -61653761663864656534316664343961393039356138393865613632643061623439653036383232 -63626666316261626466313634326639376164323739613864623762666439313935613334636661 -65616636653263326435646262663364663836346637323430353465366261643364356530393034 -66646466633734393838353664303962656636383663633732376463306363656633663434373364 -35656436616139393734616263613063333736356437633063356234653539353834393033343433 -65623030393633333531333162373034643838653864306238386463383932316538636435636436 -35326138363537396331306439306661343934353633636637363737303532653235366166373338 -64663130306630376261633030643965326332386566356332646130616564613666643336636633 -38313232353232373739393563373832356261383736336634343262646431346530393839333664 -37303664316361343332663337333761643838643839646461323434306330653838373134643630 -39626336373635656265353161663230336431643064643239306434316566353131663261386364 -39306236386230333062663233353466323862373237396237643765366363303764633264383939 -62346233623461326531323738346266306439646161633863646665653262306663636639663737 -61356165316135633932376533343838353762373031386337383764376237636239396164373031 -64636265656232363631313338356639613431313637643963303161653939653337663232633839 -65363661363639663666643539363331636131613931316131333238396565343333393764643737 -32383831326634616465633563643064343266643830626465386231303763626530393037306361 -62356234623034343462663438343363623832653563323065306337643566623335306433616438 -32653035353231306166323566356566326666393666376135363230316564356138386538353962 -65383037396338633539636131306263326432366437326338393966346535346334303532323562 -37353063636266313538353633303061303366316164383331386230633731643262653961356665 -34356462623035323034346437346533376661646130313063356139646264343864633736383936 -30353563616539333862363962343232376665626162326433303534656566323866306263303438 -65623033643065306133666261623330383665613836656631336666346361393631623964343730 -64653134643062373530326162393032363533366162373132626365363165646433643537306262 -61653933363036623036313634663563306364636533643766323438323436633732323230393138 -31396135643533313037623038663262323736626564353531626139623538643663353232373063 -64376462366163363130386238353638353764613435353162653164633933656631616137326534 -31303832356634613531653339366364393933323239393163313637663833313061316366623266 -63656464376561623366346334396439646665396463643337643038333062386262323630306465 -38373633623961623363333065303138393635393736373432626437646362336630306361626566 -65353930633033333935353833363439313664386661643365656430363162363363663039663333 -30363435303463646439323064613139313635636535396563326337333339393130333434633665 -35646465613431313236323064616533616665373737313139633539626536653633626235376663 -30353536316263383165373733383538383962393166383765663934303964363465363531653930 -63363864353065643030383337326535373130383430323462346166636565386631343737646630 -36393736366139646431636661396233656631373534333331336237366430306539653839663062 -65653235646232343865393565613164663737626361326431373763313534653862323735646565 -31393831613033306630376134656263343639626437653838313134633934356437393061383538 -38353461323933333535633164343333366463353366303363343937343938383732656437373863 -31316537346263393432353064633035376562646666653235633931333064656638316637333765 -31383635663331376230643535333032373261613437616561656262306331636164376162633836 -32353534633564343136353134313632393333353766663932343164616233373631333962353362 -65616635653262353737643833386162356533623534353733376261623231663664386339333735 -33353630623031323566623536613933353631623135346638376633356461343639373166393230 -35326639333734323531666130373230383266376265333532383732646437633431353061306138 -63373863643435643436366139383864343239646163396265303361316533303832323930666465 -38313863616362656565316332396164336235393434373762316564346130633765663936366265 -66636363393536376363633462643534373030373434393637306431663735313836323633343032 -39393966373939653635386365313365376539623965333937333133633038313662393534373731 -64383166396337333430623330613933633763363137613063316239666431616261333836316664 -36316264626338303538383261373962613536336261353565653036636662663466363332353565 -66393934613463333765393330366633386237316565323137356433333565383934613064313164 -35633261306530373264353161666666383865636332373731376339363137343139383734303061 -65366366323836396438373738393131373230333661613062373261373031626132373631383962 -34663065616135613365333062613439383364666464643230653630643561313164353939656465 -38316339613337613861323538393331363038376332383732623032343263376431353063646564 -38366566393535396535353264393832316263663966333230646533633363373231646166383639 -63613864393565376164333966383139313332363061633836616134376632356266643537393036 -65303236313862653939376366653563376334323665353138616632626363313332613734366133 -31313564363232306366396135396461363338303361646239313762303563323638643834303633 -65306537633235366339633237313737393662306433323337633235346536636637313865626633 -32326138373166346163336365633562666361336130316434316664653736633030356538373432 -39646536333839353237353939623137386164356335393330346232373763346231313037363038 -64396362363939306136373463613334353361303037393763653566313938353530616335666137 -33356163303831396639616462373561393933353861313630346434376635656566383439376630 -39336662376664633866636261643962613330306539343139623666313730393237656339316566 -63633664613135343364626138656233353862623561646561353362346562353937316461623830 -35333131646638643235643237643235303935383333393936626163383130626261613331623731 -38316464373765323931303431666337643534383035613535336537323230643532656630326361 -66343532616561646164343564353638663730636339383734323436323933373561356233616535 -64333233653238663436386131323930323838383261323439373261366664373164643734663064 -63386163353534376332643031396261313032316561623661363766336630323330336339313964 -35643861623466396532663462366161643237356164333062353565613738363965303236363663 -33366633623638663264356135316435396138636662363964313537373835363731666365633061 -31356335343764346238306530336632303630666465623565633263356165633637316466363434 -63393562616464313331663635343964666339336363383664356563313033646538653436313334 -30393966336531376436616139303339386634333761383765343664356138316632353465656432 -65353033653766373637646537336635346334333431376463356462313564633739316161373539 -39653230316234346433343563333565353664376261386239313431356337353336663061316337 -61663066306337363064646530393365636366633536306331386133303864363535336238336565 -35333039376365393335343734316630353930626238636238363137323335633739396536623636 -62353835343565643463666261323333373934623433663539636361653734393531663163616534 -61306330326165626534636233663935623164353431643633363932373662353431303363366661 -36353430396230646233313232386461333564303865373137363731356632333238646435316135 -63313734663039393765633734313335313137636637323966316162363763646539326430323731 -32343238633232373865323031376436393937333838313139653437396162623035313535353736 -32313535336237663063636265323139386237376261313434626131616336396461613032343561 -36326666373564653634363736373864613762333065343333363837653866663033323563393437 -30353337356666613066633330393338623630383964646636623933393366393238353538373762 -30393833363138356562306134313563666339343162613433303430343433366461633538336237 -64313631643431303932633434353332343431356131666661393137303635303730323864356130 -37373361343836633033386164343633396361653766306564646234373838353161356165363232 -38343538653535653935663265376262636635353535626566303737396566306263373233353638 -37653632663763306636396337306139656139623264646631383339623664353339363030373934 -64343966336262386533646664363032636533663635613734623938343235393932393730353266 -66356237653737383664396666643535313430396630663166653666636163303961636663626261 -38656632353061653437663730303731393364623732656562333436356234323464623265643765 -62646332663533393364386631653633316438313363666334383633623333613932343762373432 -35336433383536623863306365323531313534633937323839656133396436376634343231326430 -33653063633234633762313235343062656166633564636134366230633834306231383263663239 -62366137636438636636313939663666636565663336323331306138313464646332383766383236 -66306630663564396431386164666332633037303566643835623165643132336665316335396139 -31336665663666333431303431616261656161636530343634663335316537313035616533353263 -36613261633962656630616239316163653536653531383232653733366434323361656535643264 -35303536313865636139633131646334616239663338633763396564343432653265323530353538 -64396163353037323438383062343933306234643736363632613632333032633666616664613965 -65373636343534323437323766646334666239666534366534633662636464383964303564326535 -36353566396432346666383339383337336661396264663334343765636338343431653731313131 -31623066663636393537666365363033363465356561313030346165303964626266363732383463 -30393834613535633666366136336135663332303438633930383335363165306435306337346639 -66653164383237383137376363623163303234313032366665643734623463376361616336643431 -31353865663632643239396166393765633564623437356536663563623838643430373634343136 -34396133393165316332323430373231623632396265396566653838613336323465386335633130 -36373935343233353233653832376134313562353231376339633563663465343461343862626331 -39663831636130306262333266663330313537366538653537626534333662316437383731643235 -36666134633632353236666435613137393136663037626562353735636639643263623163656337 -66373039396536393031616462323936646164313835393965396130306334666231626633373662 -32333136656565356330636563383966626637363465376435626164303561376561316334663066 -63383639653163643633363039613563636132353336653862336564373631363936626133653666 -39356363633061306536323961353562393332366430313330613530646337303163626430333866 -39303831336561326133363266386462653730386231353931666233653434363839373430333732 -61636638653763646137343763383030376538383563313465323364316162376535656464613032 -38666333613137343738396163316637306331643066613631633266613637643236316136353465 -66623864326664336631323565613266613537633537396366373362303262396634643432313936 -39646138376233616330373239323830376665666131613564393364336638646639353838313762 -35656531346462373938663338313033346430626438353333303535303534303836306331333533 -37376332666662613663363934376236353432626137323462356538366231316161333039363837 -66383665636632363665653931303264636131326264616338383162306364393437323863323136 -61653562376361613038616661653661366336323361623963623531613531636334653131626237 -65376636376333303731363665623561623934326432386666646233343038646134613030353730 -61373465303932356134323533363033313637633762666465383131326130353431346630326639 -30646634373462613664626531633262643435376433303532323833626635636438306431373264 -66636466636637373064313261326161343239333765313561393033396666663434316539303066 -65306663623763663433663363656564353764313565613962396134323865386462306335643964 -38313961633534336131396433396666356236613031633765313931626230313035343935306462 -35393761393836303631353666323866613966373337393363626663326639666563663533333036 -62333331333537323164663162643661636533623366366133636165646235333730376237363031 -63653933623938626463346265336136653561643533323037613834303239393930383562313736 -35343261636231316461343733383866626535333831646436333332333862623061386131343963 -64653432383763333435303239626364323462356631323566383639643131333465663235323964 -64393034356635393734363139656634653165326432656632666539656566323439653662633236 -35353633653834656230326266363238613266306535343763393866346336666366396364316264 -65633934653364626430396338663730363137646136386337383961646330663138336266333633 -33623633313036386463393765346234313438333664303263396462326432653463303531626564 -35613063663961366364333566633763613562376164393933633434626636323964633564303234 -39353963333039373331393362643966633662653662376336333732646534313538666232623737 -39613866316635386635376465353436623864643136663638633234616265663463666337653736 -30306637303735366264323536366134313264613961393130643039303534356163653232663366 -63646164643537663466376130353132633437353965643237643861353564636439366361356266 -35323135306639346136353364613561636535616563633761646632613462313966383864303539 -62396462633465653235656662656565633736306364356361343432366263356261323137303962 -36623066353661396461313435313830616235623834643335356535616239396463333539353561 -65643939346633326631663437623530363335353165616238666530323633656334386165376462 -38663561353639313262353562323061306665326633303731306533353239393339653138383938 -35326538373061383264303134653564653263373638373032316564343131336238313034656663 -33643935373539623139316334636634383861363634623363616462636532333835663233316366 -62326433353232666431653230613065396566386366326330623662333861663361373265663962 -35343762366230636634653838393865333561626261626236393561363263326133393138626562 -35323466633862383162613739646434336163313839346534336337366563383435306166646436 -64316534316637623834623365623435346666393264313338383561613133333035306330366539 -61633034663665343037326338346339623764303961313266616464396265353264363162326364 -63383130343933353037313039653061623236646538626262333939643862316538343565623863 -36656332616537633064663765313035326634386664383532333735353562343165346561323232 -31313433316530353530316233666337323638396363336464323338323537316533376365373765 -65616139656230313966653764366531346632323463633166396564653638366461653164346536 -33363533363033646665323233303435383331363263633136313634643066343733386231373739 -31656230306238626165313238396636333136666136653261623437326235373935643431623635 -33323763323334323838633130303563316536333532353933643261383131663463383838313238 -38373030313332396136636632383131663437643434363261316566613831633838396565656439 -30653361346264316431626462303236313331666433363962643366663961366361306531366261 -30613136613133623035393638663561646236373232663734303637663830663632376466643661 -33643066383537303263326139333866666136623162306261323166623832393636363265343362 -34393362663438666161366231353738643964663536666636653235656631636461323562323266 -66353036343234633566623337626162353537343034303237383164623933373465323833353036 -30353136626562303161306463623366613265303436386533396134663563663933386563336137 -37393638353361663565303439313435633963653463306434396635333565643762353934363838 -30323834613361373730663736396534386165633362313136333631393061393338336633386432 -37303264343464323930346432626236353465376231303633623530633364643965316436366530 -37343061633238383465313337663864613963313432343265666539353039353330313937306361 -31353861363766623730343537636431373431306638636133386364343664643733396462383932 -39663433653536643737336439633338383832333261616430356633646436363234613037623534 -62366237616332333630353738643566663137636334623633653436646663636638646265613564 -34366262616565336566373462383463353635326165323262653336353737353530346563653064 -62613431343836616562326533313064376661663531346138393133393666333735306235336464 -66323834333666353032666563313935666632396364333532643261643062393438643239346330 -61393662383166343233363937343138633962633734393964646234663166373433643361346539 -39653839303264616132363261373837326636613539643962663137316636326666313261653566 -34623237656238643233373561323461353731313036326461326635653836636239303865363863 -37383938663962343530333532643038316533343132353530356239353662643737313730636535 -35353263626436336337336365363336386165666335356566336139303663646532336263386465 -30303737376130663834616562316364643862333365353630343762623364333565663236356666 -35633630326165616261393438653836386230336561383466386662333939633566323135623932 -65613632303739323261393531643063353763386338323562313130343962303034663064326538 -36336132366134346434393936346134613735633034333363393062363936373163323931353330 -62363138376138323833396233373066663339623334323261333835633062356535303639373834 -65333230366639363839393330343438316631356464633730343635326536343337666163373939 -66383235356536663663393261366361303636393563393465626364303339303533633634336339 -66393362653934653838346461323938333533633234623234643335623939366339313436613334 -64333633643935353466333938363839323265323864633731353135383637306630613732633764 -39626333353439313764356663636137373263316433616337323463373061666365313237326663 -39396234373637366434666130396438663066313664333562653166343337343665636335353438 -66393766386632353434386231646535343462303338616461366461336264613461663233646666 -38666139393764386433303831396532343262656634656234396235363439623463393033326261 -36363661616538333039366534336232636562326331623338353262366135663039313365366361 -64643931353538343864366231366239653734623134313534653663643937303037666663396432 -65623162633339616564333564666339626233326432353431303931616233636238663463383138 -39343764653332346661633832303635613333343131363263373562343737643937643234383532 -64656564343030353131353663663330626234356162663737383366633033343563626666333032 -33653831363930663363636631346664346139626264633864653862393538326632653131663038 -30653133393432353235656134313530626337373866636237666433356533356562613864343032 -66643130643538383939623765323938356334393330353161643862316237396138376336356438 -30313762373430326161656631353238376435396234396163646466326532623335656437636464 -37343262336464366434626664626332343963653565396331373838623031323030353530623164 -61613631663763343430306437613665336536393161656266623233376635366266623938653765 -35303465333830663362363532623238353431326538366339626538303762343630303739376361 -38643164336535353634663937316135396236643738383562663832613761316138376332613335 -30396364373532383930363966646462636233623431343266323565383835393330646131356663 -30303437616663346565393332343662653735353239633034653736363137306230326530616562 -37326134623331626363386639303534396266663231313232373861653430613837366638353232 -37326263326232366466316264343534663635623631393536373830313535626634666364373734 -66373731336665306465643334653638313931396466663532383234613135363433356363373963 -32646563653566623435326165366166323263363766313530393163393562353837366434626533 -31373735666539646330363430303232326631383464303437306563613463376262626463336331 -64386139353030346338316535366330333765313338646233373630356662393339313630363031 -35316235316163626334333037346436613839616165336534323861353237393336623139393537 -64646430333830336466663730363636613633326132316666353663643537376436353864616264 -32653261656330633031373662616233653838333231636262316634633038373635323333613233 -31653861653238303730656463353039666266666165383432343436356234636231336665663231 -30623331313866646239616232666437373739353230376562316332396136336431393433366463 -31393030653530613439306563616666303838363135346562633232303132646637623930386161 -63626338613564383061386637363166353566313433356662306138346362373839343937653663 -39313365333331346537663935393961373039353161343838376339623966303564623833393666 -36646264343562613435663233393935316532303364623938393232336433613530653237323666 -38313461333932623633323631663234326663346334646437613131386335316466636537326133 -36323565396436376533353661613634326534316466396331306436666537316637663761303833 -37613831363033323738633234323762323038323734333633623766333634633935343934656530 -64303338663432373264633863646164663436633134356661363764646136333764663835343037 -35383463613837366662633733356138363830636266613164666535396238663865323337636131 -33356334636238643037663962303330303730626233393333333236343562386331373762623736 -64656537343666613065326463316434303533316333316562663431316236656265383730643263 -35393065343466353364306438333962336235343537333761633730306638343239663133383466 -63383965616432396432376366653335333365326666306661653638623338356437343766353631 -31336333666630313631646339626233336664323736653234663337386632653034373362643562 -35646661303562613938366339346263323733396266376263363039376261366665333165653235 -31633235326365323538643331393831656235343932383864353337353264653966326461353263 -35353732323661386437376330646337366335373032313732303738393039366163336364653431 -63663465646134303539373036633866336166643661663330643936376136643362316137393563 -64653866333161343237333533353061616632363761356264663762643236343931633637333236 -61306265646233363635316262323339616462643937343261613137303633656635313232333031 -63386262366662393032313539656262303639343333373338303338613561313633633036343565 -37373636333963366235626165623861333231663734343231303236336661316337313738323966 -34373763353062373365336361316630633734636266633835326132356564636663363139353961 -37363137336633656639623639633037383064393539663961303730326261316266623165363766 -33323337326134623037346637363061376530393036326439303631383339666662383965316164 -35373564396262343863616231643262616639393864663735323338363232613563323732626630 -61343935663434386534333330643861303931303835343435353233626166636236323733663263 -31633134363435343638363865643936383539326264666437646562633662663763356235383338 -30653863336137613534336539366163353366633766366661633263646539656438313061333132 -37373134663064343239346337323139306432643731346435326139333138353565666439363339 -65656236313937376263633061373661653135386337323062313936343866653564636163643734 -37343131383362376364323831303966663435356133323864666131643764623238346662613632 -38613932393035353635336364613632346135363465326639343935323666343533643863343866 -30643433313566376165653839633936336265353434346336336361353634353430656530383865 -64663866323863393462346363333165633137343561393731303434396464613437313830396539 -64633434663638346661376539613561653731633262373062646238313561623933663866653462 -30663236646366383436313931663431326532646532313161376461323365656530636431353932 -62623032333064323763326636623263636363633234623637623763643931326130346161343533 -63323963383563323337616339666461336238636162396263616639633930366630623265333338 -32316465313931653038363666323931623836666631326238396362616162323063323230616233 -34376432346164353264313163313761323232633938313736313738333161386165623664373630 -38333733373562306164383963626164613531386237643138393866643533646633353436633036 -39646533383461393062653861643135656230323239343866626339346137333662386134363936 -35633064383966613764636463393763333661303362383161346233303763616138633934383435 -35316161613465363232326631386566316430333437333535333536396630626662366164333537 -38643237623062326131383665396638373037643938636565653862356663306162363064323532 -36613633666563666331333535353233373261616430336534653763653564626464306632306539 -64663061393361343364636461313130383632376534623431313035643063383934333038333639 -36366539346436383735326137666664303663633930323236613236393266623864373539633534 -66363432646332633339363730383032663334343834633630646231306337663065623537336132 -32366336646665343830383362636139363363393238636235646337623033376564663339663939 -64616430366264393665346435636434633130333666623466303932656236653332326162373361 -64616438313863343138353239333831343930636332363266393535633435643437643332663634 -35333936313665386261393161653436373935663461353434303731626166376630663931653137 -38366333663833636233643731326138333339386535623462383139306664613830326165663333 -61343361666431306163666132313935356362343365663133336665313665623039366537613263 -30323135633361333038646432633464626366613762333837393539343034616163383462303665 -64363131386266353665616535623931396337373565643434326532666431336661306163306232 -34346463316361333563633731646338643765393066366539616662316131333063366430313261 -66383634346237326630363837323036333931306432333234613763623235326164633637653763 -36396361613831333464323332353933636631333931616564643465373530366133336366313035 -62393639316430353938303432396535623435356665306563346363333765353562636639373265 -30626134386462366534303066393033616532396434656132626131613932373536303163633563 -36386332396164653962653438623939316634663431626435376166383164633130323163323130 -61363066333566303938326239343030656230336335316436653461636666393966656432306136 -30653563383563363531626630373536333265336538373762643035373835613562333137626266 -34393234323833356435666165333464313438376465666135636334346131376534633863666635 -33363938653635653534333336346538313862616333386362643965373435353630613564633136 -37363162356265333331366136653161323635353136336331333362313338316635333830316231 -31306438343064663465656463343732323263616264646530663136353630373366333139656430 -31613261356438346536316239633736613863386235613165373635383837623032316265333361 -36646437646164333136333462656538393364303466353731616530316331393166333962653732 -65633839366131303332353732383936396137393638396330356534303937343038353165613539 -64373663336163356434313366333135643839663164343866333065613435306461353438353435 -37623233306433393866316562306661636634623261633934353064653439386535626663643162 -36323164623963323335303438646332366563323935663037346333356165646264343665616235 -62343964333336616563633139396663613131623836663930663336333335643135613531613265 -38376562663566613932383536356433373865633161363134626631396464643938336139373531 -30613437666436386433373965653136653164303164643737386463636632363034356131633334 -38366362613938633939636464386162393139316136306465636564623931353430656530346665 -65353936663065346237303331336135333733383232376234303364393333313762643135316334 -37343162623762393733346336313431643164623138643135396337653938643332396133326439 -30383132663934366635616161323635373834646666376461376439346535656338323433383537 -36343163386638323364366533643939326663316439663030663161613637343938343661623734 -64633539383364613632363930373833646237366463323663386139616461636331316438616637 -32646261386637633066306363326663666161653235326338393330613533366536643136383636 -36313232663764626239323234363566346266393161376431653732613165396239326562393933 -65663664353736303434383666396334633263653135393934653266373464333964646537346132 -37623436393365323534663937333561326437613431346162643837316162313033663935623735 -39356233373137323762353733316430393136353938653432656266653839343238646565363130 -32666533353230383464353339386233333437653761646533623263346361353737653035353937 -36643736616130633233303862373065393333346232333436366436623764336565346265393661 -62303238386135613834643938343639396535353536643261343539623630303461373633396636 -30323534653065366330303466323861376135333366623435663966613636386332376263333964 -36343165653530653633643137613638343766633062386132306464346239613035343436613332 -31383839353961343130616431396634653533646633646164323465393637303838633636366564 -32623865353130333736353037373532613862353338323732333534666332623963336432633430 -38313834373065626139393434323032663562383730333439383430376539333230323265666631 -33303063386464356131666262313632613565313634616433633034616335643763363432333865 -30316430353533386638336234353932363139333865323330393834343932313637616239616562 -62646361613939626132363366633361336363653630303230326464373465626666366466653861 -36393265363636663736613332336362643035396165343830326237336531656366313438393262 -66663662653334353864663963353539303263633465363931386136356566626133626634326434 -63336134326362663135643834323339353465326336326631333864323736623961653462616665 -36303533373735663136383439363335656639646333303737366531633832666136353264656561 -62633730663262363630336262343164646431313239663337383937643935663237303363343339 -64323936306234646465663366396365646439316462623536393631353938613664383164396232 -66336337646633626530313462343561303139653739663331383930656566313039646331653630 -36616261336339396137333734303532643666376361656563653666306365383165646334323863 -66366437306165636639373335363939656133613632386264366639623932333832613235313935 -30303538323863316639633139393266383661623735623938653566396330393065306537323633 -32376535373336636134366630303563396562626364376237383162373832626334633765376436 -31313163366430313136663262383739646465383263346461396330396432316266653836386637 -39363965616336373937376362373737316635326338643530663137616162376535646464623130 -64626163363462663831313365653264326331346435653830343732396537313262623566373930 -62623036396435613337303664373564623132396262353863393165396332343333303938313138 -31313964383534366635653939323530393839366266303437383163353037626166333735656237 -39383639353635346437366464653135303832353037303738333139633966356663336664663630 -61393532346534383163633862326261616565636337653432646462386139323462376565333330 -39396431333537663839393430663362643830346466313238366330643737636635373835363134 -39373037313861356539613930356639356538393433353362366130313331383534343665346535 -64343166643465323635363363623337363033393864643537353263636165666139313630383930 -34633763356261643938646361376466646435366363656436666363653763373162626362343461 -39393033353263313237616637313364326162353836356635393262363131396635636330353636 -37373033313435663437323762313361623434333034653363323165633131363164396161663737 -30333937633633396564336434383933613162353135393931343965636135646266633164346636 -36623933376538306166636530636562373432393336343031323632343331386361353232353236 -35653535613233373161316235616463633736383561353763623630646164363935366434376266 -32393933613862386534353232613532363264363362643864646139346339363163613237623865 -36373035336138376131386631353731303234363737353363353566613166343937353531653733 -38333761316235623536396539656462663037336662336165393065386162383637663730303566 -31326439306536393665633939393563653065333831373363306631656261363130313137363462 -65373164316135363532323739303961643161653436346463666566353865653330373964333934 -34316439616138623933613465613939303965613133663833636431376235346163316566653733 -62386438316564643366373666643438383164363131636161363237353062666536303131623662 -62316139666534333938353463343935623835343165353739623166643535383132616636383239 -37393131636266326432663531323461363035636530363738656337383831336262306330383336 -64316638656261343862316633663131303465653137303735323138646635666239383861353563 -63366665386336306130633363656266396438383935666539326266623538353931313138396161 -32613234663065663036656262366566393066323830343932313563666332376163373266373761 -36373766393464363233643935386465313735366132376531356565363333646666343431386435 -36313535353237613830613665316362356666396166313738373030666131646337656334383839 -36336462323662643233646664666537663264356562326565303332656334643838653735323563 -37316338633865346163313231313531336431333533313165393230613165613634323435303362 -36393364383737663638636139303039666239626431633438386562306132323461336465383130 -65636438343964616132323732643265633736613736346366343231343835313130666633336335 -62643733626666373762356436343637376236353136626264306531313239303931636364646666 -66373835666664343166366664373436323136373836336262323331366635646432643837303233 -32333639653062616631653766393462393762366536623261346165326230326162613534613265 -39376439353635656261303734353736393835353861336636646661643666613936356438646538 -31616466633937313361353662616263393761653465303633323938393761616339333337643239 -63303133333461373630656566336166666330393161376666383633333864383961643062646235 -65313136353561373863623231353163383331363761383837666564343330373331636534656565 -38313632353031353864393063613139353166383466353036613638376635393931313031353533 -66613862643931356463333065653864616235623831643438393162643266663633363066643833 -65653232336462363164393861333234323737383330643162313532643430353461643733383963 -62323966663761346561383666376234646134326533613934643131306561316536353339633564 -64666161383563353763636631663533386437666432623236643335376239666437636138383437 -61663866373630336164306436346161386337616464396132636561383733323836353666653038 -31343564363336613161343235383532306662623439383836613030643239313234313166336132 -66356231303638353064623962636239653633306238346430336266666231303661303337333163 -38376332336230353230666539623132356435663137396364643937366336363435363939623539 -36373464613130663937613937333430653233383832626331356633626335343635396339656232 -31626336323937653132343265313463333264646139343137616163646234633563663135366437 -36363432346662646638613438633137653838376361303234346430323933366661333336636433 -38333533353232373862643531333238646435343338666233336464656234356239616566653338 -65353234306134653332303636613133636539623039396166636639333065636433646638653661 -33666234396539306566316639336439636365663330326361623136656465643838636436316330 -35393139623732353761626561656233346462633638386639626162623739323664396234306462 -39393063386336343761306532333764313930396131653361306235376538326436366639393665 -37333034323735613233373838393162343237363437306631666230336233303363336237663638 -35636561376462323338623734333361336661386538326539383761663838393666616663366261 -65633833653337643138313930366661313838646566323036376137386361336263366261653634 -66616339623562616536383638653131633662323562363236623862363637393064333862616335 -37633934366465306235346332313134656465336531653261653765366632343766323936306363 -32303130383030393462623433313465333965633364333830333831306639663134666639663664 -38363462393339376139653131623033343039343666626666666361623235633937323733376164 -38316235616531383666363233383531323232333966656531393537643266653535396137386265 -63626134393738643432343630666537373430383533333835633630383239666264666465646131 -37663732356338356465633039633937613934656165363833663638613661343362333533386563 -33636265333230356635383165366466643461333836313565393339333935303239376362346563 -32366536616164383361323535636330373765326137383330386635353136383433353832333964 -37656564303830613266656337643134323866663138363435653965323433396636343931656338 -36313531633930316632386565636234343538653538333634353937356231666433313138373038 -38373562303531343432366631643663353138303434363738376363366238376633646231646465 -35303765366161303762306337653435633666323064323330326630366131653865336333616461 -32666239363334303165313462333734353735656563303235306534373335376439373064363036 -37653065653432353061356336383565303031316330353962303063343364313463353262336463 -39636464376237623337636566353338663761356465636634343234643032616230613938633561 -62653632663932666435643335386564313262303066643635623865373134326565386562613863 -39376464313964383133323465303135666535346433386437646661366138666361373161663962 -30643131373363356363386266316365396264343064313466643534653262653032396539636236 -37396333373163313837373064373366383137663131383964393032356639376337666662643834 -64333735333332653634313962323938366264313962323663613737323161313937313135643538 -62396133616162616136643563616635363236353537303337643031386438376138323139323338 -30663562653966643462613639393439623666323338633662646666306431343733613462316436 -35623261306261386135333830326234633561343064333765653432653237626264313964376461 -31636461396436653236326333386264396230663037636332393162373739343034663465306463 -36323138323138336263346238623062313061646438383962636262383963373533366130356234 -39346666333265366462623433393264313135343430333561623530373932303338323132643562 -38663037303031656466383332653132346534383464613230326164663831646463663239323433 -30313231333135666537306134396332333933393466363163656233346364343236646538363939 -63356532306235636232396231306166303931313861346565643633656133613231353863393663 -66623035366534626330393934613163663839663561623962323236633037343239373931313162 -65303534636635633137313664636335323665313130666463666137623530626231613930336430 -32336261393330396539323863323838393439393438306564633439336666313732646239333733 -31333366393538666233383437613762316339646564393561383138326266396663366530336639 -39316137306464646130373034653561343463313064656533363834363638383832623537383034 -31313034353931313834663765306666386465656131393139663066343065366462663661633364 -34626434303835633939653539306265323136383439393663623434303264363962333732363636 -32353063653765653266643130613465333862373333303262326636303233326631666537306130 -32313237646361373864363038353239313663366135333030653631633330613335653965356133 -62383266646437666137306132633730653963336364366630643134356536336139336238363465 -30363831366639376431306264663834346632386362666561343265346334366238633065383362 -34356239643231303866656231623235316231323838656435393437313834353331633462383761 -30343239313132333739386266663330626362623236363439613732653030386434383631353930 -63643731373261363061326335613265306463663566303237363963643530643933353266393739 -39363963343339396464643136386464663464666637663233666463623563633362346130663038 -36336330313831333038636132373165616532383265626634333962333431333839626661336133 -31376564343032636634396332616432313763303331326135306531333064653239643138613566 -33636538333264363263 diff --git a/infra/ansible-docker/inventory-sample.yml b/infra/ansible-docker/inventory-sample.yml deleted file mode 100644 index 5ae98cf5..00000000 --- a/infra/ansible-docker/inventory-sample.yml +++ /dev/null @@ -1,39 +0,0 @@ -all: - vars: - ansible_user: - ansible_python_interpreter: /usr/bin/python3 - ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q ubuntu@"' - -traefik: - hosts: - traefik-1: - ansible_host: - -keycloak: - hosts: - keycloak-1: - ansible_host: - -lakekeeper: - hosts: - lakekeeper-1: - ansible_host: - -trino: - hosts: - trino-1: - ansible_host: - -elt: - hosts: - elt-1: - ansible_host: - -superset_accelerator: - hosts: - superset-accelerator-1: - ansible_host: - -superset_farm: - children: - superset_accelerator: diff --git a/infra/ansible-docker/playbooks/cloud/private_network_create.yml b/infra/ansible-docker/playbooks/cloud/private_network_create.yml deleted file mode 100644 index 776fca95..00000000 --- a/infra/ansible-docker/playbooks/cloud/private_network_create.yml +++ /dev/null @@ -1,36 +0,0 @@ ---- -- name: Read global configuration - hosts: localhost - gather_facts: false - tasks: - - ansible.builtin.include_vars: - dir: "../../group_vars/all" - -- name: Provision private network - hosts: localhost - gather_facts: False - tasks: - - name: Create private network - openstack.cloud.network: - cloud: "{{ openstack_cloud_name }}" - name: "{{ openstack_virtual_network.name }}" - external: false - admin_state_up: true - state: present - - name: Create private network subnets - openstack.cloud.subnet: - cloud: "{{ openstack_cloud_name }}" - network_name: "{{ openstack_virtual_network.name }}" - name: "{{ subnet.name }}" - cidr: "{{ subnet.cidr }}" - gateway_ip: "{{ subnet.gateway_ip }}" - state: present - loop: "{{ openstack_virtual_network.subnets }}" - loop_control: - loop_var: subnet - - name: Attach to External router - openstack.cloud.router: - cloud: "{{ openstack_cloud_name }}" - state: present - name: "{{ openstack_project_name }}-Router" - interfaces: "{{ openstack_virtual_network.subnets | map(attribute='name') }}" diff --git a/infra/ansible-docker/playbooks/cloud/vm_create.yml b/infra/ansible-docker/playbooks/cloud/vm_create.yml deleted file mode 100644 index 2a7d4607..00000000 --- a/infra/ansible-docker/playbooks/cloud/vm_create.yml +++ /dev/null @@ -1,20 +0,0 @@ -# The following variables are required on the command line: -# - openstack_cloud_name: Name of the cloud in ~/.config/clouds.yaml -# - openstack_key_name: Name of the SSH key to add allowed to the deployed VM -# - vm_config_file: Path to a yml file providing instance-specific VM variables. - -- name: Read VM configuration - hosts: localhost - gather_facts: false - tasks: - - ansible.builtin.include_vars: - dir: "../../group_vars/all" - - ansible.builtin.include_vars: - file: "{{ vm_config_file }}" - -- name: Provision VM - hosts: localhost - gather_facts: false - roles: - - role: security_group_create - - role: vm_create diff --git a/infra/ansible-docker/playbooks/datasources/group_vars/datasources.yml b/infra/ansible-docker/playbooks/datasources/group_vars/datasources.yml deleted file mode 100644 index bdd1c74c..00000000 --- a/infra/ansible-docker/playbooks/datasources/group_vars/datasources.yml +++ /dev/null @@ -1,16 +0,0 @@ ---- -datasources_influxdb_port: 8086 -datasources_influxdb_container_name: influxdb -datasources_influxdb_data_dir: "{{ cephfs_mount_mount_path }}/staging/influx/influxdbv2/data" - -openstack_server_name: datasources -openstack_flavor: l3.small -openstack_security_group_name: datasources -openstack_security_group_description: Ingress to datasources ports -openstack_security_group_rules: - - direction: ingress - ether_type: IPv4 - protocol: tcp - port_range_min: "{{ datasources_influxdb_port }}" - port_range_max: "{{ datasources_influxdb_port }}" -openstack_security_groups: [default, "{{ openstack_security_group_name }}"] diff --git a/infra/ansible-docker/playbooks/elt/deploy.yml b/infra/ansible-docker/playbooks/elt/deploy.yml deleted file mode 100644 index 6efd9cbd..00000000 --- a/infra/ansible-docker/playbooks/elt/deploy.yml +++ /dev/null @@ -1,161 +0,0 @@ ---- -- name: Create logrotate_entries variable - hosts: elt - tasks: - - set_fact: - elt_cron_log_dirs: > - {{ lakekeeper_catalog.warehouses.keys() | - map('regex_replace', '^(.*)$', elt_cron_logs_root_dir + '/\1') | - list - }} - loop: "{{ lakekeeper_catalog.warehouses.keys() }}" - - set_fact: - logrotate_entries: > - {{ logrotate_entries | default([]) + [{'name': item.0, 'path': elt_cron_logs_root_dir ~ item.1}] }} - loop: "{{ lakekeeper_catalog.warehouses.keys() | zip(elt_cron_log_dirs) | list }}" - -- name: Configure ELT instance - hosts: elt - roles: - - role: robertdebock.logrotate - become: true - tasks: - # ---------- UV & Python ---------- - - name: Check if uv exists - stat: - path: /usr/local/bin/uv - register: stat_uv - - - name: Fetch uv installer - ansible.builtin.get_url: - url: https://astral.sh/uv/0.6.6/install.sh - dest: /tmp/uv-install.sh - when: not stat_uv.stat.exists - - - name: Execute uv installer - become: true - shell: env UV_UNMANAGED_INSTALL=/usr/local/bin sh /tmp/uv-install.sh - when: not stat_uv.stat.exists - - - name: Remove uv installer script - file: - path: /tmp/uv-install.sh - state: absent - when: not stat_uv.stat.exists - - - name: Find compatible Python version - ansible.builtin.command: uv python find --quiet "{{ elt_uv_python_version }}" - ignore_errors: true - register: uv_python_find - - - name: Ensure Python is installed - ansible.builtin.command: uv python install "{{ elt_uv_python_version }}" - when: uv_python_find.rc != 0 - - # ---------- AWS S3 ---------- - - name: Ensure S3 configuration directory exists - ansible.builtin.file: - path: "{{ ansible_env['HOME'] }}/.aws" - state: directory - mode: "u=rwx,g=rx,o=rx" - - - name: Ensure S3 config file is absent - ansible.builtin.file: - path: "{{ ansible_env['HOME'] }}/.aws/config" - state: absent - - - name: Ensure AWS configuration is present - ansible.builtin.copy: - dest: "{{ ansible_env['HOME'] }}/.aws/credentials" - content: |+ - [default] - endpoint = {{ s3_endpoint }} - request_checksum_calculation = when_required - region = local-01 - aws_access_key_id = {{ s3_access_key_id }} - aws_secret_access_key = {{ s3_access_secret }} - mode: "u=rw,o=r,g=r" - - # ---------- dlt ---------- - - name: Ensure dlt configuration directory exists - ansible.builtin.file: - path: "{{ ansible_env['HOME'] }}/.dlt" - state: directory - mode: "u=rwx,g=rx,o=rx" - - - name: Generate .dlt/config.toml - # no_log: true - ansible.builtin.template: - src: dlt/config.toml.j2 - dest: "{{ ansible_env['HOME'] }}/.dlt/config.toml" - mode: "u=rwx,g=r,o=r" - - # ---------- secrets ---------- - - name: Ensure secrets directories exist - become: true - ansible.builtin.file: - path: "{{ elt_secrets_root_dir }}/{{ warehouse.name }}" - state: directory - mode: "u=rwx,g=rx,o=rx" - loop: "{{ elt_warehouses }}" - loop_control: - loop_var: warehouse - - - name: Generate secrets - become: true - no_log: true - ansible.builtin.template: - src: ./secrets/envvars.j2 - dest: "{{ elt_secrets_root_dir }}/{{ warehouse.name }}/envvars" - mode: "u=rwx,g=r,o=r" - loop: "{{ elt_warehouses }}" - loop_control: - loop_var: warehouse - - # ---------- ELT jobs ---------- - - name: Ensure cron scripts directory exists - ansible.builtin.file: - path: "{{ elt_cron_root_dir }}" - state: directory - mode: "u=rwx,g=rx,o=rx" - - - name: Create/update ELT cron script template - ansible.builtin.template: - src: ./cron/elt_task.sh.j2 - dest: "{{ elt_cron_root_dir }}/elt_task.sh" - mode: "u=rwx,g=rx,o=rx" - - - name: Ensure cron logs directory exists - ansible.builtin.file: - path: "{{ elt_cron_logs_root_dir }}/{{ warehouse.name }}" - state: directory - mode: "u=rwx,g=rx,o=rx" - loop: "{{ elt_warehouses }}" - loop_control: - loop_var: warehouse - - - ansible.builtin.include_tasks: elt_cron_tasks.yml - loop: "{{ elt_warehouses }}" - loop_control: - loop_var: warehouse - - # ---------- Iceberg maintenance jobs ---------- - - name: Create/update Iceberg maintenance script template - ansible.builtin.template: - src: ./cron/iceberg_maintenance.sh.j2 - dest: "{{ elt_cron_root_dir }}/iceberg_maintenance.sh" - mode: "u=rwx,g=rx,o=rx" - - - name: Ensure Iceberg table maintenance jobs exist - ansible.builtin.cron: - name: iceberg-maintenance - state: "{{ warehouse.maintenance.state }}" - hour: 3 - minute: 2 - job: > - NO_COLOR=1 UV_NO_PROGRESS=1 ELT_SECRETS={{ elt_secrets_root_dir }}/{{ warehouse.name }}/envvars - {{ elt_cron_root_dir }}/iceberg_maintenance.sh - > {{ elt_cron_logs_root_dir }}/{{ warehouse.name }}/iceberg_maintenance-$(date +\%Y\%m\%d_\%H\%M\%S).log 2>&1 - loop: "{{ elt_warehouses }}" - loop_control: - loop_var: warehouse diff --git a/infra/ansible-docker/playbooks/elt/group_vars/elt.yml b/infra/ansible-docker/playbooks/elt/group_vars/elt.yml deleted file mode 100644 index f7600c35..00000000 --- a/infra/ansible-docker/playbooks/elt/group_vars/elt.yml +++ /dev/null @@ -1,42 +0,0 @@ ---- -# VM config -openstack_server_name: elt -openstack_flavor: l3.xsmall -openstack_security_groups: [default] - -# Deployment config -elt_uv_python_version: "3.13" -elt_git_clone_dir: "{{ ansible_env['HOME'] }}/var/git" -elt_cron_root_dir: "{{ ansible_env['HOME'] }}/var/cron" -elt_cron_logs_root_dir: "{{ ansible_env['HOME'] }}/var/cron/logs" -elt_secrets_root_dir: /secrets/warehouses - -elt_warehouses: - - name: accelerator - github_org: ISISNeutronMuon - github_repo: analytics-data-platform - sub_dir: warehouses/accelerator - sources: - - name: statusdisplay - hour: 1 - minute: 45 - dbt_args: "--select 'models/staging/statusdisplay'" - - name: accelerator_sharepoint - hour: 2 - minute: 2 - dbt_args: "--select 'models/staging/accelerator_sharepoint'" - - name: opralogweb - hour: 2 - minute: 11 - dbt_args: "--select '+models/operations' --exclude '+models/operations/power_consumption.sql'" - - name: electricity_sharepoint - hour: "*" - minute: 5 - dbt_args: "--select '+models/operations/power_consumption.sql'" - maintenance: - state: present -elt_threads: 16 - -logrotate_frequency: weekly -logrotate_keep: 30 -logrotate_compress: true diff --git a/infra/ansible-docker/playbooks/elt/templates/cron/iceberg_maintenance.sh.j2 b/infra/ansible-docker/playbooks/elt/templates/cron/iceberg_maintenance.sh.j2 deleted file mode 100644 index f69f2e1e..00000000 --- a/infra/ansible-docker/playbooks/elt/templates/cron/iceberg_maintenance.sh.j2 +++ /dev/null @@ -1,15 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -# Constants -ELT_SECRETS=${ELT_SECRETS:?} -ELT_COMMON_GIT_URL=${ELT_COMMON_GIT_URL:-https://github.com/ISISNeutronMuon/analytics-data-platform.git} -UV_GIT_REF="git+${ELT_COMMON_GIT_URL}#subdirectory=elt-common" - -# Read secrets -set -o allexport; source $ELT_SECRETS; set +o allexport - -# Run maintenance tasks on all tables -uvx \ - --with "${UV_GIT_REF}[iceberg-maintenance]" \ - python -m elt_common.iceberg.maintenance $* diff --git a/infra/ansible-docker/playbooks/keycloak/deploy.yml b/infra/ansible-docker/playbooks/keycloak/deploy.yml deleted file mode 100644 index 2a491e51..00000000 --- a/infra/ansible-docker/playbooks/keycloak/deploy.yml +++ /dev/null @@ -1,7 +0,0 @@ ---- -- name: Configure Keycloak to act as an identity provider - hosts: keycloak - roles: - - role: cephfs_mount - - role: docker_configure - - role: keycloak diff --git a/infra/ansible-docker/playbooks/keycloak/group_vars/keycloak.yml b/infra/ansible-docker/playbooks/keycloak/group_vars/keycloak.yml deleted file mode 100644 index de6d02bd..00000000 --- a/infra/ansible-docker/playbooks/keycloak/group_vars/keycloak.yml +++ /dev/null @@ -1,27 +0,0 @@ ---- -# VM config -openstack_server_name: keycloak -openstack_flavor: l3.micro -openstack_security_group_name: keycloak -openstack_security_group_description: Ingress to keycloak ports -openstack_security_group_rules: - - direction: ingress - ether_type: IPv4 - protocol: tcp - port_range_min: "{{ keycloak_http_port }}" - port_range_max: "{{ keycloak_http_port }}" - - direction: ingress - ether_type: IPv4 - protocol: tcp - port_range_min: "{{ keycloak_http_management_port }}" - port_range_max: "{{ keycloak_http_management_port }}" -openstack_security_groups: [default, "{{ openstack_security_group_name }}"] - -# App config -docker_configure_container_network: "{{ keycloak_container_network }}" -keycloak_db_port: 5432 -keycloak_db_dbname: "{{ vault_keycloak_db_name }}" -keycloak_db_user: "{{ vault_keycloak_db_user }}" -keycloak_db_passwd: "{{ vault_keycloak_db_passwd }}" -keycloak_bootstrap_admin_passwd: "{{ vault_keycloak_bootstrap_admin_passwd }}" -keycloak_java_keystore_passwd: "{{ vault_keycloak_java_keystore_passwd }}" diff --git a/infra/ansible-docker/playbooks/lakekeeper/deploy.yml b/infra/ansible-docker/playbooks/lakekeeper/deploy.yml deleted file mode 100644 index 2fc14644..00000000 --- a/infra/ansible-docker/playbooks/lakekeeper/deploy.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- -- name: Configure Lakekeeper to act as an Iceberg catalog - hosts: lakekeeper - roles: - - role: docker_configure - - role: lakekeeper diff --git a/infra/ansible-docker/playbooks/lakekeeper/group_vars/lakekeeper.yml b/infra/ansible-docker/playbooks/lakekeeper/group_vars/lakekeeper.yml deleted file mode 100644 index 3de71880..00000000 --- a/infra/ansible-docker/playbooks/lakekeeper/group_vars/lakekeeper.yml +++ /dev/null @@ -1,26 +0,0 @@ ---- -# VM configuration -openstack_server_name: lakekeeper -openstack_flavor: l3.xsmall -openstack_security_group_name: lakekeeper -openstack_security_group_description: Ingress to catalog ports -openstack_security_group_rules: - - direction: ingress - ether_type: IPv4 - protocol: tcp - port_range_min: "{{ lakekeeper_http_port }}" - port_range_max: "{{ lakekeeper_http_port }}" -openstack_security_groups: [default, "{{ openstack_security_group_name }}"] - -# Deployment configuration -lakekeeper_container_network: lakekeeper - -lakekeeper_metadb_host: "{{ vault_db_host }}" -lakekeeper_metadb_port: "{{ vault_db_port }}" -lakekeeper_metadb_name: "{{ vault_lakekeeper_metadb_name }}" -lakekeeper_metadb_user: "{{ vault_lakekeeper_metadb_user }}" -lakekeeper_metadb_passwd: "{{ vault_lakekeeper_metadb_passwd }}" -lakekeeper_metadb_encryption_key: "{{ vault_lakekeeper_metadb_encryption_key }}" - -# pass to docker_configure -docker_configure_container_network: "{{ lakekeeper_container_network }}" diff --git a/infra/ansible-docker/playbooks/superset/deploy.yml b/infra/ansible-docker/playbooks/superset/deploy.yml deleted file mode 100644 index 0ccf00d9..00000000 --- a/infra/ansible-docker/playbooks/superset/deploy.yml +++ /dev/null @@ -1,17 +0,0 @@ ---- -# This playbook is designed assuming that there are many superset instances on the same machine. -# The inventory defines groups named "superset_" to use. We pick an arbitrary one -# to configure the prerequisites. -# TODO: Is there a way to pick up all of the unique hosts (by ip address)? -- name: Configure prerequisites - hosts: superset_accelerator - roles: - - role: geerlingguy.pip - become: true - - role: docker_configure - -- name: Configure Superset - hosts: superset_farm - serial: 1 - roles: - - role: superset diff --git a/infra/ansible-docker/playbooks/superset/group_vars/all.yml b/infra/ansible-docker/playbooks/superset/group_vars/all.yml deleted file mode 100644 index c046cad0..00000000 --- a/infra/ansible-docker/playbooks/superset/group_vars/all.yml +++ /dev/null @@ -1,10 +0,0 @@ ---- -# image -superset_container_image: ghcr.io/martyngigg/adp-superset-forked:901412e - -# common db -superset_metadb_host: "{{ vault_db_host }}" -superset_metadb_port: "{{ vault_db_port }}" -superset_metadb_name: "{{ vault_superset_metadb_name }}" -superset_metadb_user: "{{ vault_superset_metadb_user }}" -superset_metadb_passwd: "{{ vault_superset_metadb_passwd }}" diff --git a/infra/ansible-docker/playbooks/superset/group_vars/superset_accelerator.yml b/infra/ansible-docker/playbooks/superset/group_vars/superset_accelerator.yml deleted file mode 100644 index 040099da..00000000 --- a/infra/ansible-docker/playbooks/superset/group_vars/superset_accelerator.yml +++ /dev/null @@ -1,16 +0,0 @@ ---- -# db - needs to be manually in the database -superset_metadb_schema_name: "{{ vault_superset_accelerator_metadb_schema_name }}" - -# app -superset_instance_name: superset_accelerator -superset_http_port: "{{ superset_instances[superset_instance_name].http_port }}" -superset_app_root: "{{ superset_instances[superset_instance_name].app_root }}" -superset_secret_key: "{{ vault_superset_accelerator_secret_key }}" -superset_flask_debug: false -superset_env_name: production -superset_config_yaml: "{{ vault_superset_accelerator_config_yaml }}" - -superset_analytic_dbs: - - database_name: accelerator - uri: "trino://{{ trino_superset_username }}:{{ vault_trino_superset_passwd }}@{{ top_level_domain }}:{{ trino_https_port }}/accelerator" diff --git a/infra/ansible-docker/playbooks/superset/vm_vars.yml b/infra/ansible-docker/playbooks/superset/vm_vars.yml deleted file mode 100644 index 6749c00f..00000000 --- a/infra/ansible-docker/playbooks/superset/vm_vars.yml +++ /dev/null @@ -1,14 +0,0 @@ ---- -# Specialized variables for the Superset VM. -# We run multiple instances of Apache Superset on a single node to maximise resource usage. -openstack_server_name: superset_farm -openstack_security_group_name: superset -openstack_security_group_description: Ingress to superset ports -openstack_security_group_rules: - - direction: ingress - ether_type: IPv4 - protocol: tcp - port_range_min: 8088 # how can we not duplicate this... - port_range_max: 8089 -openstack_flavor: l3.xsmall -openstack_security_groups: [default, "{{ openstack_security_group_name }}"] diff --git a/infra/ansible-docker/playbooks/system/system_update.yml b/infra/ansible-docker/playbooks/system/system_update.yml deleted file mode 100644 index a82f2d72..00000000 --- a/infra/ansible-docker/playbooks/system/system_update.yml +++ /dev/null @@ -1,4 +0,0 @@ -- name: Apply system package updates and reboot nodes if required - hosts: all:!localhost - roles: - - role: packages_update diff --git a/infra/ansible-docker/playbooks/traefik/deploy.yml b/infra/ansible-docker/playbooks/traefik/deploy.yml deleted file mode 100644 index 4c20c57c..00000000 --- a/infra/ansible-docker/playbooks/traefik/deploy.yml +++ /dev/null @@ -1,10 +0,0 @@ ---- -- name: Configure traefik - hosts: traefik - roles: - - role: geerlingguy.pip - become: true - - role: docker_configure - - role: docs - become: true - - role: traefik diff --git a/infra/ansible-docker/playbooks/traefik/group_vars/traefik.yml b/infra/ansible-docker/playbooks/traefik/group_vars/traefik.yml deleted file mode 100644 index f6ff5b4b..00000000 --- a/infra/ansible-docker/playbooks/traefik/group_vars/traefik.yml +++ /dev/null @@ -1,12 +0,0 @@ -openstack_flavor: l3.nano -openstack_server_name: traefik -openstack_server_fip: "{{ openstack_reverse_proxy_fip }}" -openstack_security_group_name: traefik -openstack_security_group_description: Ingress to traefik ports -openstack_security_group_rules: - - direction: ingress - ether_type: IPv4 - protocol: tcp - port_range_min: "{{ trino_https_port }}" - port_range_max: "{{ trino_https_port }}" -openstack_security_groups: [default, "traefik", "HTTP", "HTTPS"] diff --git a/infra/ansible-docker/playbooks/trino/deploy.yml b/infra/ansible-docker/playbooks/trino/deploy.yml deleted file mode 100644 index f75b2a14..00000000 --- a/infra/ansible-docker/playbooks/trino/deploy.yml +++ /dev/null @@ -1,9 +0,0 @@ ---- -- name: Configure trino - hosts: trino - roles: - - role: cephfs_mount - - role: geerlingguy.pip - become: true - - role: docker_configure - - role: trino diff --git a/infra/ansible-docker/playbooks/trino/group_vars/trino.yml b/infra/ansible-docker/playbooks/trino/group_vars/trino.yml deleted file mode 100644 index 99da5fda..00000000 --- a/infra/ansible-docker/playbooks/trino/group_vars/trino.yml +++ /dev/null @@ -1,18 +0,0 @@ ---- -# VM configuration -openstack_server_name: trino -openstack_security_group_name: trino -openstack_security_group_description: Ingress to Trino ports -openstack_security_group_rules: - - direction: ingress - ether_type: IPv4 - protocol: tcp - port_range_min: "{{ trino_http_port }}" - port_range_max: "{{ trino_http_port }}" -openstack_flavor: l3.xsmall -openstack_security_groups: [default, "{{ openstack_security_group_name }}"] - -# Trino configuration -trino_data_path: "{{ cephfs_mount_mount_path }}/live/trino/data" -trino_jvm_xmx: 48G -trino_shared_secret: "{{ vault_trino_shared_secret }}" diff --git a/infra/ansible-docker/roles/docker_configure/defaults/main.yml b/infra/ansible-docker/roles/docker_configure/defaults/main.yml deleted file mode 100644 index a5dd0daf..00000000 --- a/infra/ansible-docker/roles/docker_configure/defaults/main.yml +++ /dev/null @@ -1,18 +0,0 @@ ---- -# geerlingguy.docker -docker_edition: "ce" -docker_packages: - - "docker-{{ docker_edition }}" - - "docker-{{ docker_edition }}-cli" - - "docker-{{ docker_edition }}-rootless-extras" -docker_obsolete_packages: - - docker - - docker.io - - docker-engine - - podman-docker - - containerd - - runc -docker_packages_state: present - -# this role -docker_configure_container_network_external: true diff --git a/infra/ansible-docker/roles/docker_configure/tasks/main.yml b/infra/ansible-docker/roles/docker_configure/tasks/main.yml deleted file mode 100644 index 1e996bbc..00000000 --- a/infra/ansible-docker/roles/docker_configure/tasks/main.yml +++ /dev/null @@ -1,16 +0,0 @@ ---- -# Install docker -- ansible.builtin.include_role: - name: geerlingguy.docker - apply: - become: true - -# Create docker network if needed -# TODO: Migrate this into each role that needs it -- name: Create container network - become: true - community.docker.docker_network: - name: "{{ docker_configure_container_network }}" - driver_options: - com.docker.network.driver.mtu: "{{ docker_daemon_options.mtu }}" - when: docker_configure_container_network_external and docker_configure_container_network is defined diff --git a/infra/ansible-docker/roles/keycloak/defaults/main.yml b/infra/ansible-docker/roles/keycloak/defaults/main.yml deleted file mode 100644 index de8f5f60..00000000 --- a/infra/ansible-docker/roles/keycloak/defaults/main.yml +++ /dev/null @@ -1,4 +0,0 @@ -keycloak_container_network: keycloak -keycloak_image: quay.io/keycloak/keycloak:26.3 -keycloak_image_optimized_name: keycloak_optimized -keycloak_working_dir: /var/keycloak diff --git a/infra/ansible-docker/roles/postgres/defaults/main.yml b/infra/ansible-docker/roles/postgres/defaults/main.yml deleted file mode 100644 index 2c86375e..00000000 --- a/infra/ansible-docker/roles/postgres/defaults/main.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- -postgres_image: ghcr.io/cloudnative-pg/postgresql:16.3-bookworm -postgres_db_port: 5432 -postgres_db_data_path: /data/postgres -postgres_db_user: postgres -postgres_docker_initdb_path: /config/postgres/docker-entrypoint-initdb.d diff --git a/infra/ansible-docker/roles/postgres/tasks/main.yml b/infra/ansible-docker/roles/postgres/tasks/main.yml deleted file mode 100644 index 1f90b43b..00000000 --- a/infra/ansible-docker/roles/postgres/tasks/main.yml +++ /dev/null @@ -1,61 +0,0 @@ ---- -- name: Ensure postgres data directory exists - become: true - ansible.builtin.file: - path: "{{ postgres_db_data_path }}" - state: directory - owner: 26 - group: 999 - mode: "u=rwx,g=rx,o=rx" - -- name: Ensure secrets directory exists - become: true - ansible.builtin.file: - path: /secrets - state: directory - mode: "u=rwx,g=rx,o=rx" - -- name: Ensure postgres secrets are available - become: true - ansible.builtin.copy: - dest: /secrets/postgres-passwd - content: "{{ postgres_db_passwd }}" - owner: root - group: root - mode: "u=r,g=r,o=r" - -- name: Ensure docker initdb directory exists - become: true - ansible.builtin.file: - path: "{{ postgres_docker_initdb_path }}" - state: directory - mode: "u=rwx,g=rx,o=rx" - -- name: Create docker initdb script - become: true - ansible.builtin.template: - src: init-catalog-dbs.sh.j2 - dest: "{{ postgres_docker_initdb_path }}/init-catalogs-dbs.sh" - mode: "u=rwx,g=rwx,o=r" - -- name: Ensure Postgresql container is running - become: true - community.docker.docker_container: - name: postgres - image: "{{ postgres_image }}" - state: started - cleanup: true - detach: true - restart_policy: unless-stopped - published_ports: - - "{{ postgres_db_port }}:5432" - env: POSTGRES_USER="{{ postgres_db_user }}" - POSTGRES_PASSWORD_FILE=/secrets/postgres-passwd - volumes: - - "{{ postgres_db_data_path }}:/var/lib/postgresql/data" - - /secrets:/secrets:ro - - "{{ postgres_docker_initdb_path }}:/docker-entrypoint-initdb.d" - networks: - - name: "{{ docker_configure_container_network | default('bridge') }}" - comparisons: - networks: strict diff --git a/infra/ansible-docker/roles/postgres/templates/init-catalog-dbs.sh.j2 b/infra/ansible-docker/roles/postgres/templates/init-catalog-dbs.sh.j2 deleted file mode 100644 index e666d9b5..00000000 --- a/infra/ansible-docker/roles/postgres/templates/init-catalog-dbs.sh.j2 +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash -# Create the initial databases. -# Use the Ansible 'postgres_db_names' variable to define a list of the -# desired names. -set -e - -psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL -{% for name in postgres_db_names %} - CREATE DATABASE {{ name }}; -{% endfor %} -EOSQL diff --git a/infra/ansible-docker/roles/security_group_create/tasks/main.yml b/infra/ansible-docker/roles/security_group_create/tasks/main.yml deleted file mode 100644 index ab20e3b2..00000000 --- a/infra/ansible-docker/roles/security_group_create/tasks/main.yml +++ /dev/null @@ -1,9 +0,0 @@ ---- -- name: Ensure security group is present in openstack (using verbose rules) - openstack.cloud.security_group: - cloud: "{{ openstack_cloud_name }}" - name: "{{ openstack_security_group_name }}" - description: "{{ openstack_security_group_description }}" - security_group_rules: "{{ openstack_security_group_rules }}" - state: present - when: openstack_security_group_rules is defined diff --git a/infra/ansible-docker/roles/superset/defaults/main.yml b/infra/ansible-docker/roles/superset/defaults/main.yml deleted file mode 100644 index 17399860..00000000 --- a/infra/ansible-docker/roles/superset/defaults/main.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- -superset_working_dir: "/var/{{ superset_instance_name }}" - -# app -superset_flask_debug: true -superset_env_name: qa diff --git a/infra/ansible-docker/roles/superset/tasks/main.yml b/infra/ansible-docker/roles/superset/tasks/main.yml deleted file mode 100644 index 7f3947b2..00000000 --- a/infra/ansible-docker/roles/superset/tasks/main.yml +++ /dev/null @@ -1,76 +0,0 @@ ---- -- name: Ensure working directory exists - become: true - ansible.builtin.file: - path: "{{ superset_working_dir }}" - state: directory - mode: "u=rwx,g=rx,o=rx" - -- name: Ensure configuration files are synchronized - become: true - ansible.posix.synchronize: - use_ssh_args: true - src: ./ - dest: "{{ superset_working_dir }}/" - archive: false - recursive: true - delete: true - -- name: Ensure templated configuration files are synchronized - become: true - ansible.builtin.template: - src: "{{ superset_template.src }}" - dest: "{{ superset_working_dir }}/{{ superset_template.dest }}" - mode: "{{ superset_template.mode|default('u=rw,g=r,o=r') }}" - loop: - - { src: docker-compose.yml.j2, dest: docker-compose.yml } - - { - src: superset-cli.sh.j2, - dest: superset-cli.sh, - mode: "u=rwx,g=rx,o=rx", - } - - { src: docker/dotenv.j2, dest: docker/.env } - - { - src: docker/docker-init.sh.j2, - dest: docker/docker-init.sh, - mode: "u=rwx,g=rx,o=r", - } - loop_control: - loop_var: superset_template - -- name: Ensure superset_config.yml exists - become: true - no_log: true - ansible.builtin.copy: - dest: "{{ superset_working_dir }}/docker/pythonpath/superset_config.yml" - content: "{{ superset_config_yaml | to_nice_yaml }}" - mode: "u=rw,g=r,o=r" - -- name: Ensure certificates directory exists - become: true - ansible.builtin.file: - path: "{{ superset_working_dir }}/certs" - state: directory - mode: "u=rwx,g=rx,o=rx" - -- name: Ensure certificates are present - become: true - no_log: true - ansible.builtin.copy: - dest: "{{ superset_working_dir }}/certs/{{ certificate.filename }}" - content: "{{ certificate.content }}" - owner: root - group: root - mode: "u=r,o=,g=" - loop: - - { filename: stfc-ca.pem, content: "{{ stfc_ca_cert }}" } - loop_control: - loop_var: certificate - -- name: Run Superset services - become: true - community.docker.docker_compose_v2: - project_src: "{{ superset_working_dir }}" - pull: always - recreate: always - state: present diff --git a/infra/ansible-docker/roles/superset/templates/docker-compose.yml.j2 b/infra/ansible-docker/roles/superset/templates/docker-compose.yml.j2 deleted file mode 100644 index ce1e3526..00000000 --- a/infra/ansible-docker/roles/superset/templates/docker-compose.yml.j2 +++ /dev/null @@ -1,99 +0,0 @@ -x-superset-image: &superset-image {{ superset_container_image }} -x-superset-user: &superset-user root -x-superset-depends-on: &superset-depends-on - redis: - condition: service_started -x-superset-env: &superset-env - - path: docker/.env # default - required: true -x-superset-volumes: &superset-volumes - - {{ superset_working_dir }}/docker/docker-init.sh:/app/docker/docker-init.sh:ro - - {{ superset_working_dir }}/docker/pythonpath:/app/docker/pythonpath:ro - - superset_home:/app/superset_home - - {{ superset_working_dir }}/certs:/certs:ro -x-networks: &superset-networks - - {{ superset_instance_name }} - - -services: - redis: - container_name: {{ superset_instance_name }}_cache - env_file: *superset-env - image: redis:7 - restart: unless-stopped - volumes: - - redis:/data - networks: *superset-networks - - superset: - container_name: {{ superset_instance_name }}_app - env_file: *superset-env - image: *superset-image - command: [ "/app/docker/docker-bootstrap.sh", "app-gunicorn" ] - user: *superset-user - restart: unless-stopped - ports: - - {{ superset_http_port }}:8088 - depends_on: - <<: *superset-depends-on - superset-init: - condition: service_completed_successfully - volumes: *superset-volumes - networks: *superset-networks - - superset-init: - container_name: {{ superset_instance_name }}_init - env_file: *superset-env - image: *superset-image - command: [ "/app/docker/docker-init.sh" ] - depends_on: *superset-depends-on - user: *superset-user - volumes: *superset-volumes - healthcheck: - disable: true - networks: *superset-networks - - superset-worker: - container_name: {{ superset_instance_name }}_worker - env_file: *superset-env - image: *superset-image - command: [ "/app/docker/docker-bootstrap.sh", "worker" ] - restart: unless-stopped - depends_on: - <<: *superset-depends-on - superset: - condition: service_healthy - user: *superset-user - volumes: *superset-volumes - networks: *superset-networks - healthcheck: - test: - [ - "CMD-SHELL", - "celery -A superset.tasks.celery_app:app inspect ping -d celery@$$HOSTNAME" - ] - - superset-worker-beat: - container_name: {{ superset_instance_name }}_worker_beat - env_file: *superset-env - image: *superset-image - command: [ "/app/docker/docker-bootstrap.sh", "beat" ] - restart: unless-stopped - depends_on: - <<: *superset-depends-on - superset: - condition: service_healthy - user: *superset-user - volumes: *superset-volumes - networks: *superset-networks - healthcheck: - disable: true - -volumes: - redis: - superset_home: - -networks: - {{ superset_instance_name }}: - driver_opts: - com.docker.network.driver.mtu: "{{ docker_daemon_options.mtu }}" diff --git a/infra/ansible-docker/roles/superset/templates/docker/dotenv.j2 b/infra/ansible-docker/roles/superset/templates/docker/dotenv.j2 deleted file mode 100644 index ab2684e6..00000000 --- a/infra/ansible-docker/roles/superset/templates/docker/dotenv.j2 +++ /dev/null @@ -1,29 +0,0 @@ -# Metadata DB -SUPERSET_DB_HOST={{ superset_metadb_host }} -SUPERSET_DB_USER={{ superset_metadb_user }} -SUPERSET_DB_PASSWORD={{ superset_metadb_passwd }} -SUPERSET_DB_NAME={{ superset_metadb_name }} -SUPERSET_DB_SCHEMA_NAME={{ superset_metadb_schema_name }} -SUPERSET_DB_PORT={{ superset_metadb_port }} -SUPERSET_DB_DIALECT=postgresql - -# Caching layer (match service name and port in compose file) -REDIS_HOST=redis -REDIS_PORT=6379 - -# Environment/development settings -PYTHONPATH=/app/docker/pythonpath -SCARF_ANALYTICS=false -FLASK_DEBUG={{ superset_flask_debug }} -SUPERSET_LOG_LEVEL={{ "DEBUG" if superset_flask_debug else "INFO" }} -SUPERSET_ENV={{ superset_env_name }} -# Use fixed container port. -SUPERSET_PORT=8088 -SUPERSET_APP_ROOT={{ superset_app_root }} - -# Make sure you set this to a unique secure random value on production -SUPERSET_SECRET_KEY={{ superset_secret_key }} - -# Auth -LDAP_SERVER={{ stfc_ldap_server }} -LDAP_CACERTFILE=/certs/stfc-ca.pem diff --git a/infra/ansible-docker/roles/system_configure/defaults/main.yml b/infra/ansible-docker/roles/system_configure/defaults/main.yml deleted file mode 100644 index 5a4c6539..00000000 --- a/infra/ansible-docker/roles/system_configure/defaults/main.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -system_configure_timezone: "Europe/London" -system_configure_package_update: true diff --git a/infra/ansible-docker/roles/system_configure/tasks/main.yml b/infra/ansible-docker/roles/system_configure/tasks/main.yml deleted file mode 100644 index 4f5d72b7..00000000 --- a/infra/ansible-docker/roles/system_configure/tasks/main.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- -- name: Ensure APT key is available - ansible.builtin.include_tasks: ../../../tasks/ensure_openstack_apt_keys.yml - -- name: Ensure timezones are available - become: true - ansible.builtin.apt: - update_cache: true - pkg: - - tzdata - state: present - -- name: Set timezone - become: true - community.general.timezone: - name: "{{ system_configure_timezone }}" - -- name: Ensure VM is configured - include_role: - name: packages_update diff --git a/infra/ansible-docker/roles/traefik/tasks/main.yml b/infra/ansible-docker/roles/traefik/tasks/main.yml deleted file mode 100644 index d5187699..00000000 --- a/infra/ansible-docker/roles/traefik/tasks/main.yml +++ /dev/null @@ -1,109 +0,0 @@ ---- -- name: Ensure Traefik directories exist - become: true - ansible.builtin.file: - path: "{{ traefik_dir.path }}" - state: directory - mode: "u=rwx,g=rx,o=rx" - owner: "root" - group: "root" - loop: - - { - path: "{{ traefik_config_root }}/{{ traefik_etc_dynamic_path_relative }}", - mode: "u=rwx,g=rx,o=rx", - } - - { - path: "{{ traefik_config_root }}/logs/traefik", - mode: "u=rwx,g=rx,o=rx", - } - - { path: "{{ traefik_certs_root }}", mode: "u=rwx,g=,o=" } - loop_control: - loop_var: traefik_dir - -- name: Ensure TLS certifcates are present - become: true - no_log: true - ansible.builtin.copy: - dest: "{{ traefik_tls.filename }}" - content: "{{ traefik_tls.content }}" - owner: root - group: root - mode: "u=r,o=,g=" - loop: - - { - filename: "{{ traefik_certs_root }}/cert__analytics_isis_cclrc_ac_uk.pem", - content: "{{ vault_tls_cert__analytics_isis_cclrc_ac_uk }}", - } - - { - filename: "{{ traefik_certs_root }}/key__analytics_isis_cclrc_ac_uk.pem", - content: "{{ vault_tls_key__analytics_isis_cclrc_ac_uk }}", - } - loop_control: - loop_var: traefik_tls - -- name: Regenerate Traefik static configuration - become: true - ansible.builtin.template: - src: "templates/{{ traefik_etc_path_relative }}/traefik.yml.j2" - dest: "{{ traefik_config_root }}/{{ traefik_etc_path_relative }}/traefik.yml" - mode: "u=rw,g=r,o=r" - -- name: Ensure Traefik container is running - become: true - community.docker.docker_container: - name: traefik - image: "{{ traefik_image }}" - state: started - cleanup: true - detach: true - restart: true - recreate: true - restart_policy: unless-stopped - network_mode: host - mounts: - - type: bind - source: "{{ traefik_certs_root }}/cert__analytics_isis_cclrc_ac_uk.pem" - target: /certs/cert__analytics_isis_cclrc_ac_uk.pem - read_only: true - - type: bind - source: "{{ traefik_certs_root }}/key__analytics_isis_cclrc_ac_uk.pem" - target: /certs/key__analytics_isis_cclrc_ac_uk.pem - read_only: true - volumes: - - "{{ traefik_config_root }}/{{ traefik_etc_path_relative }}:/etc/traefik" - - "{{ traefik_certs_root }}:/certs" - - "{{ traefik_config_root }}/logs/traefik:/var/log/traefik" - - /var/run/docker.sock:/var/run/docker.sock:ro - - /etc/localtime:/etc/localtime:ro - -- name: Create temporary build directory for dynamic configurations - ansible.builtin.tempfile: - state: directory - register: traefik_dynamic_confs_tempdir - -- name: Regenerate new Traefik dynamic configuration (staging location) - become: true - ansible.builtin.template: - src: "{{ traefik_template }}" - dest: "{{ traefik_dynamic_confs_tempdir.path }}/{{ traefik_template | basename | replace('.j2', '') }}" - mode: "u=rw,g=r,o=r" - with_fileglob: "templates/{{ traefik_etc_dynamic_path_relative }}/*.j2" - loop_control: - loop_var: traefik_template - -- name: Synchronize new dynamic configuration - become: true - ansible.posix.synchronize: - use_ssh_args: true - src: "{{ traefik_dynamic_confs_tempdir.path }}/" - dest: "{{ traefik_config_root }}/{{ traefik_etc_dynamic_path_relative }}/" - archive: false - checksum: true - recursive: true - delete: true - delegate_to: "{{ inventory_hostname }}" - -- name: Remove temporary build directory for dynamic configurations - ansible.builtin.file: - path: "{{ traefik_dynamic_confs_tempdir.path }}" - state: absent diff --git a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/lakekeeper.yml.j2 b/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/lakekeeper.yml.j2 deleted file mode 100644 index 6efc8800..00000000 --- a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/lakekeeper.yml.j2 +++ /dev/null @@ -1,35 +0,0 @@ -{%- set base_paths = {lakekeeper_base_path: hostvars[groups['lakekeeper'][0]]['ansible_host']} %} -{% if 'lakekeeper_preview' in groups %} -{%- set _ = base_paths.update({lakekeeper_base_path + '-preview': hostvars[groups['lakekeeper_preview'][0]]['ansible_host']}) -%} -{% endif %} - -http: - routers: - {% for base_path, server_name in base_paths.items() %} - {%- set santized_name = base_path.replace('/', '_') -%} - {{ santized_name }}: - entryPoints: websecure - rule: Host(`{{ top_level_domain }}`) && PathPrefix(`{{ base_path }}`) - service: {{ santized_name }} - middlewares: - - {{ santized_name }}-strip-prefix - tls: {} - {% endfor %} - - middlewares: - {% for base_path, server_name in base_paths.items() %} - {%- set santized_name = base_path.replace('/', '_') -%} - {{ santized_name }}-strip-prefix: - stripPrefix: - prefixes: - - "{{ base_path }}" - {% endfor %} - - services: - {% for base_path, ip_addr in base_paths.items() %} - {%- set santized_name = base_path.replace('/', '_') -%} - {{ santized_name }}: - loadBalancer: - servers: - - url: "http://{{ ip_addr }}:{{ lakekeeper_http_port }}" - {% endfor %} diff --git a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/tls.yml.j2 b/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/tls.yml.j2 deleted file mode 100644 index 64c7d1e5..00000000 --- a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/tls.yml.j2 +++ /dev/null @@ -1,4 +0,0 @@ -tls: - certificates: - - certFile: /certs/cert__analytics_isis_cclrc_ac_uk.pem - keyFile: /certs/key__analytics_isis_cclrc_ac_uk.pem diff --git a/infra/ansible-docker/roles/trino/files/etc/rules.json b/infra/ansible-docker/roles/trino/files/etc/rules.json deleted file mode 100644 index ff06e006..00000000 --- a/infra/ansible-docker/roles/trino/files/etc/rules.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "catalogs": [ - { - "user": "ingestion", - "catalog": "(isis|accelerator|management)", - "allow": "all" - }, - { - "user": "superset", - "catalog": "(isis|accelerator|management)", - "allow": "read-only" - } - ] -} diff --git a/infra/ansible-docker/roles/trino/templates/etc/catalog/isis.properties.j2 b/infra/ansible-docker/roles/trino/templates/etc/catalog/isis.properties.j2 deleted file mode 100644 index 572c81b3..00000000 --- a/infra/ansible-docker/roles/trino/templates/etc/catalog/isis.properties.j2 +++ /dev/null @@ -1,18 +0,0 @@ -connector.name=iceberg -iceberg.catalog.type=rest -iceberg.rest-catalog.uri=https://analytics.isis.cclrc.ac.uk/iceberg/catalog -iceberg.rest-catalog.warehouse=isis -iceberg.rest-catalog.vended-credentials-enabled=false -iceberg.rest-catalog.security=OAUTH2 -iceberg.rest-catalog.oauth2.server-uri={{ keycloak_token_endpoint_url_isis }} -iceberg.rest-catalog.oauth2.credential={{ vault_trino_catalog_client_id }}:{{ vault_trino_catalog_client_secret }} -iceberg.rest-catalog.oauth2.scope=lakekeeper offline_access - -# Our S3 implementation does not have STS enabled so we need to provide -# S3 credentials too -fs.native-s3.enabled=true -s3.endpoint={{ s3_endpoint }} -s3.region={{ s3_region }} -s3.path-style-access=true -s3.aws-access-key={{ s3_access_key_id }} -s3.aws-secret-key={{ s3_access_secret }} diff --git a/infra/ansible-docker/roles/vm_create/tasks/main.yml b/infra/ansible-docker/roles/vm_create/tasks/main.yml deleted file mode 100644 index 1f7f29a4..00000000 --- a/infra/ansible-docker/roles/vm_create/tasks/main.yml +++ /dev/null @@ -1,50 +0,0 @@ ---- -- name: Ensure cloud server is present - openstack.cloud.server: - name: "{{ openstack_server_name }}{{ openstack_server_name_suffix | default('') }}" - cloud: "{{ openstack_cloud_name }}" - flavor: "{{ openstack_flavor }}" - image: "{{ openstack_image }}" - key_name: "{{ openstack_key_name }}" - network: "{{ openstack_virtual_network.name | default('Internal') }}" - auto_floating_ip: false - security_groups: "{{ openstack_security_groups | default(['default']) }}" - timeout: 200 - state: present - sdk_log_level: "{{ openstack_sdk_log_level | default('INFO') }}" - register: vm_info - -- name: Associate floating IP to instance - openstack.cloud.floating_ip: - cloud: "{{ openstack_cloud_name }}" - server: "{{ openstack_server_name }}" - network: "{{ openstack_virtual_network.name }}" - floating_ip_address: "{{ openstack_server_fip }}" - when: openstack_server_fip is defined - -- name: Set fact for IP address of VM (Internal network) - ansible.builtin.set_fact: - vm_ip_address: "{{ vm_info.server.addresses[openstack_virtual_network.name | default('Internal')][0].addr }}" - cacheable: false - when: openstack_server_fip is not defined - -- name: Set fact for IP address of VM (Private network) - ansible.builtin.set_fact: - vm_ip_address: "{{ openstack_server_fip }}" - cacheable: false - when: openstack_server_fip is defined - -- name: Display IP address of VM - ansible.builtin.debug: - msg: "{{ openstack_server_name }}: {{ vm_ip_address }}" - -- name: Wait for connection - ansible.builtin.wait_for_connection: - timeout: 300 - delegate_to: "{{ vm_ip_address }}" - -- name: Ensuring VM is configured - include_role: - name: system_configure - apply: - delegate_to: "{{ vm_ip_address }}" diff --git a/infra/ansible-docker/scripts/platform-system-update.sh b/infra/ansible-docker/scripts/platform-system-update.sh deleted file mode 100755 index 1ff0148b..00000000 --- a/infra/ansible-docker/scripts/platform-system-update.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/bash -# Apply system updates in a specified order to the platform nodes -INVENTORY=inventory.yml - -function fail_with_message() { - echo "$@" - exit 1 -} - -function system_update() { - local selection=$1 - ansible-playbook -i $INVENTORY -l "$selection" playbooks/system/system_update.yml -} - -test -f $INVENTORY || fail_with_message "$INVENTORY must exist in current directory ($PWD)" - -# Do updates in a top-down order -system_update elt -system_update superset_farm -system_update trino -system_update lakekeeper -system_update keycloak -system_update traefik diff --git a/infra/ansible-docker/tasks/ensure_openstack_apt_keys.yml b/infra/ansible-docker/tasks/ensure_openstack_apt_keys.yml deleted file mode 100644 index e2acf644..00000000 --- a/infra/ansible-docker/tasks/ensure_openstack_apt_keys.yml +++ /dev/null @@ -1,17 +0,0 @@ ---- -- name: Check if a keys exist - ansible.builtin.stat: - path: "/etc/apt/trusted.gpg.d/{{ item.filename }}" - loop: "{{ openstack_apt_keys }}" - register: file_stats - -- name: Fetch missing apt GPG keys - become: true - ansible.builtin.get_url: - url: "{{ file_stat.item.url }}" - dest: "/etc/apt/trusted.gpg.d/{{ file_stat.item.filename }}" - mode: 644 - when: not file_stat.stat.exists - loop: "{{ file_stats.results }}" - loop_control: - loop_var: file_stat diff --git a/infra/ansible-docker/tasks/ensure_openstack_bindings.yml b/infra/ansible-docker/tasks/ensure_openstack_bindings.yml deleted file mode 100644 index 762c7f84..00000000 --- a/infra/ansible-docker/tasks/ensure_openstack_bindings.yml +++ /dev/null @@ -1,6 +0,0 @@ -- name: Ensure Python openstack bindings are present - ansible.builtin.pip: - name: - - "python-openstackclient=={{ openstackclient_version }}" - - "python-octaviaclient=={{ octaviaclient_version }}" - state: present diff --git a/infra/ansible-docker/.ansible-lint b/infra/ansible/.ansible-lint similarity index 100% rename from infra/ansible-docker/.ansible-lint rename to infra/ansible/.ansible-lint diff --git a/infra/ansible-docker/.ansible-lint-ignore b/infra/ansible/.ansible-lint-ignore similarity index 100% rename from infra/ansible-docker/.ansible-lint-ignore rename to infra/ansible/.ansible-lint-ignore diff --git a/infra/ansible-docker/.gitignore b/infra/ansible/.gitignore similarity index 100% rename from infra/ansible-docker/.gitignore rename to infra/ansible/.gitignore diff --git a/infra/ansible-docker/ansible-galaxy-requirements.yaml b/infra/ansible/ansible-galaxy-requirements.yaml similarity index 100% rename from infra/ansible-docker/ansible-galaxy-requirements.yaml rename to infra/ansible/ansible-galaxy-requirements.yaml diff --git a/infra/ansible-docker/ansible.cfg b/infra/ansible/ansible.cfg similarity index 100% rename from infra/ansible-docker/ansible.cfg rename to infra/ansible/ansible.cfg diff --git a/infra/ansible/group_vars/all/all.yml b/infra/ansible/group_vars/all/all.yml new file mode 100644 index 00000000..cbf92446 --- /dev/null +++ b/infra/ansible/group_vars/all/all.yml @@ -0,0 +1,93 @@ +--- +base_additional_apt_keys: + - url: https://dl.igtf.net/distribution/igtf/current/GPG-KEY-EUGridPMA-RPM-4 + filename: eugridpma-rpm-4.asc +base_timezone: "Europe/London" + +cephfs_mount_export_path: "{{ vault_cephfs_export_path }}" +cephfs_mount_mount_path: "/mnt/lakehouse_data" +cephfs_mount_access_name: "{{ vault_cephfs_access_name }}" +cephfs_mount_access_secret: "{{ vault_cephfs_access_secret }}" + +docker_daemon_options: + mtu: 1450 + registry-mirrors: ["https://dockerhub.stfc.ac.uk"] + +docker_users: + - "{{ vault_openstack_vm_user }}" + +logrotate_frequency: weekly +logrotate_keep: 30 +logrotate_compress: true + +keycloak_base_path: /authn +keycloak_http_port: 8080 +keycloak_http_management_port: 9000 +keycloak_url: "https://{{ top_level_domain }}{{ keycloak_base_path }}" +keycloak_realm: + name: "isis-analytics-data-platform" + display_name: "ISIS Analytics Data Platform" +keycloak_realm_url: "{{ keycloak_url }}/realms/{{ keycloak_realm.name }}" + +lakekeeper_base_path: /iceberg +lakekeeper_http_port: 8181 +lakekeeper_project_name: "ISIS Analytics Data Platform" +lakekeeper_catalog: + uri: "https://{{ top_level_domain }}{{ lakekeeper_base_path }}/catalog" + warehouses: + accelerator: + storage: + bucket_name: isis-dataarchitecture-prototypes-warehouse-accelerator + key_prefix: "" + +pip_install_packages: + - name: docker + extra_args: --break-system-packages + +stfc_ldap_server: "{{ vault_stfc_ldap_server }}" +stfc_ca_cert: "{{ vault_stfc_ca_cert }}" + +# trino +trino_http_port: 8060 +trino_https_port: 8443 + +# Superset +superset_farm_url_prefix: /workspace +superset_farm_instances: + accelerator: + app_root: "{{ superset_farm_url_prefix }}/accelerator" + env_name: production + flask_debug: false + http_port: 8088 + secret_key: "{{ vault_superset_accelerator_secret_key }}" + users_yaml: "{{ vault_superset_accelerator_users_yaml }}" + analytic_dbs: + - name: accelerator + db_url: "trino://superset:{{ vault_trino_users['superset'].password }}@{{ top_level_domain }}:{{ trino_https_port }}/accelerator" + +# elt +elt_warehouses: + - name: accelerator + github_org: ISISNeutronMuon + github_repo: analytics-data-platform + sub_dir: warehouses/accelerator + sources: + - name: statusdisplay + hour: 1 + minute: 45 + dbt_args: "--select 'models/staging/statusdisplay'" + - name: accelerator_sharepoint + hour: 2 + minute: 2 + dbt_args: "--select 'models/staging/accelerator_sharepoint'" + - name: opralogweb + hour: 2 + minute: 11 + dbt_args: "--select '+models/operations' --exclude '+models/operations/power_consumption.sql'" + - name: electricity_sharepoint + hour: "*" + minute: 5 + dbt_args: "--select '+models/operations/power_consumption.sql'" + maintenance: + state: present +elt_threads: 16 diff --git a/infra/ansible/group_vars/all/vault.yml b/infra/ansible/group_vars/all/vault.yml new file mode 100644 index 00000000..6da161fb --- /dev/null +++ b/infra/ansible/group_vars/all/vault.yml @@ -0,0 +1,150 @@ +$ANSIBLE_VAULT;1.1;AES256 +65643366333061633330663138646265373135643133613962323865353330636162393466646238 +3066333033326338393830383736316638316464303162320a343032396539636661346437346466 +65363466306632333636653231633064646435383933306133633166656230613830653362366266 +6430623666323338350a306233333738613031626430613062623931393335326638343466376263 +31336634353037363234386234633762363631343839643338666336616364343436633034626465 +30656362326166363863636462383365643139396338376632313366623863346638613834643763 +66623534353838346333356432396333343131333330363562356532663265333537363636623231 +65303636326531643962396163663137363362346465616430373666306163383364363032303239 +61373063646630613462306166323736626330376266366136643661663839613039656234393831 +66393533363237303631616363396234303064356136653935643335393937363537353737393762 +34613330613263346537316336636639323738646564393664366635353631353363316363303763 +36333036373533313139353361343863613934653261333934643735323662653764343463316535 +30666139326638336263396461383430303961663030396363623138383764303133316634396262 +37356235306265666637633336313231303638373031353766303062343265646339346334643466 +35643163376334336238363335613935666533333139616333353662333532616563343336396534 +64623661346430363163666331383339346334323530323538633932653638303162663836656264 +66323538646133633736336131353237316365323134343035336238306333616232323963373131 +36373036346430326165646338343537393931363665393537623536343336353261376232326563 +33333238306164646534636338393261363763643436643834613737313034396437666638316331 +62663336383464346236383635666663383763316331656535643266376233306536633435343363 +33313334393131396462663335653431373538633465333836636135663135313261623664646433 +39383465383632326337303833373438303237323531343166323736376438376562663136346335 +32383030613164626462313762643435666530373930633635393039646439346137323561393734 +61396433646435393235303835663533356363346162376232323761346234633638353837376635 +66333764646662343437333266393262356434633565666234653734633231333866353338363766 +66653834366130383665393761343261616531653865636538303233336238653032346266316137 +37646463303031303662363338373636613566376537623264373233373165613065333264656564 +32646465386534393631633865393537653566353863393339373663346239623063373339323163 +33303237386231386639633839663238363366643766346561333162376232373336363836303938 +30316535383165653336616366663362613734396239616538623063343935306239626433653936 +35616536666230396530306437386262623863633665356163343130643136353866373962303237 +31613766393139393066636430303831346236636130306461653231356239363636383862633065 +38386532623031363135626464373766656632366531633739343238343939396363653337653131 +61343463313461313839616134333666353930383163383238653633373231643832616239323439 +38623164626365326530356632613861363037303435643766613536336333623238366332356139 +38313561313134666435633333643834653032396433396138323930336166393034323363316232 +65353230613731333963386361636432373530623463376163306338386334323964343436653232 +36646565666530303562376337313465616334323034396132346162643034386438343631626263 +39346239306633363135396363303961383662326366656363303333623765623430353136616664 +37623830663737323431323261366433633264393566626137656561383436323535356631323061 +34343234373531313066353665623861366664646263386131643432313831653863616366636663 +61613132326530343237323032633938656133663330626266666139623231306233373131326139 +62356365303564353033313330323432376361613262653635356434633837386264633663343431 +31633464343037663130633038633334623634313263316330653637663034313664663833623930 +36333435656362303030346366366263323030613334353764633962306264356634653264653333 +32636563313639663763366532356437336364363237636139646665333265616364363433323630 +31343564653332373535323730656535646464303737356435396339653638643332626639626138 +39633161386463666430303939623466366534383034373161626430623832613838313032303638 +37346264306537613339633435346164303766393732313537613238313664633330653563323736 +37613531653035333338653863623234386538353261343337313839333833396162666136613532 +63303739323531373233626264373561373735663261626161663531373763333938646630373831 +31373865666462303462386134646638373364356163653365303661336261613737393833623863 +62623364643432613465383535316664646132636536613662363331306539396163373531313330 +66333465336430666531376264346131653964343530633639396366316235386365383464303661 +63303330646331666137663230616634666464663739346635626666356239363832663634386262 +33393331383166333431383336333361316461363239653634643235313365353961316638303663 +31653431666337363266636237643039323664616631366562313439663832323333303734373263 +32313738666533373666616333653061323135323336336163343436663431353766633461306331 +35313639373665623733343061613537383563376162663737623139343536376364333431393239 +64626639383366373135343435333166303634616530386464666138333163333566316333383733 +31623539313564393763313466306130646334363937646333656263386465383131653137636134 +30366566313538643035663731663063306362346366303039653937356132313931393438383263 +35613166613934633264326430383062343535643135633237666166323261613065313664646262 +64343161643530393137633732326166363539353332333033316537363766613634316662326632 +37653235383463643862376435356161363230396564623038653033393633636563323433613833 +64393734393834616662356538343766393761393130333661316130363932613166643262313136 +32613835326466366133326332333434633438666636613534373031353835653333636362336466 +31633531346663343561613332323137373434316262616165363262313833386164393061303533 +38376332646161656232366564323032303735313530386264633062333462313539323438353039 +35363330316235613062623065373935306263313439323263373139326536326234303833343839 +38633462633434383438613938663331346236303461393662653535623862653337643932393138 +39323361386230376630353661323764383931323664383430623864663363346635313533666361 +39343831373033643430626165653766663038363862316133653366366461626131323732333366 +31306634646636383536303737376337356436363737363434363262623564386564366534373736 +61343736656632646530373039643663646464356530653834376438623138366164373036306661 +32316335653038396539643365396637653862303864613966313264303833303130303032323237 +61363735323164333531346435386232633763313430613866306366303133633834616562613663 +61353066393138306230326662353062666131616164623733363934626464313134356430343866 +34393132363364376136626561393939643266663534313232336539336161366132363434343730 +38336562383630666338366462333665323839366361363330613037653361316338376665636636 +63303336356532393462323339646665323538663562343130633164633932386138353333626365 +36623131313566623531643662656434643235383935663434343230323763373038373162303166 +64366638336566333463353334616433343164653634663135303232376632353939336336656131 +63366332663562336233623838663138666137663861626564623633313934656438343764326535 +64343366303365646232323432356635643161306466633736313663383438633733646636346265 +61613165356362323766633537303737383133396636336163616563613734646639613261626162 +34613737326661643565663433666638386635353966336134616335656234666530396135633564 +32353164643134376561303737656233633066643564356462616231663463333039306434306465 +38626464343066343962356432306335396636616334373065313163376630636363366530303763 +65363265383265653132386631646263386137373736333462313839333932643931343964663431 +38376563353330303238613636613931363533353765393236303164626637373032323733643533 +32343765333466376531376662613265396334383838656261323334613438363233656262323964 +35663430373334396436613363376163623261623430356463643666353365333839613461383530 +32386236373664346435646132303431386662386132633639383837386336623466613165306235 +65646636323266653361343065333336356566313233366631613463363434396433306265323734 +34656261376662613239613237376231623136643161313737326435653132623333633931313033 +32323166643338306632326330356238346531633035643237333937303033336236393962313734 +35343235663735383138646164666363393037313465356232353338363637343932313762336131 +34353034376665623061663665346136333764366337663836373932323965623032376130346161 +32333762643836323061336134393766623737313433613064316334633535303966313539643632 +36646131663234616433356332373866646563313437383131646139616361386435386532333433 +64303036343862656533396361666164356235653362346663333533633137356333343064393261 +34376538373331626436396332363633393439303965633165393139623132643930306461353836 +65613261383062343265633663616239396235363338643265303137653035623437633865616463 +32356334393862343165346666653439363161366634613537623030323362373733333035393735 +38653035366165636535313334316635643731313430613163313332336366363266653630643639 +32666365646335363430356361646664633062323732633765376232643737656665313737383834 +36656234636566356636663866623835363637663336653863383664363936323537623733656466 +35303861626234323263376130306339396234653537366661336334326565373332626666306366 +30373831656438333731633362303262636538656336653139613166653261363331313839343562 +64633565383961306237383938616331633033666431653635633561313232616438626565613961 +37363238356433666265636162623033666563363632326239373439363063386334386465363364 +30306430666366623133663562616162393164383439626635353830313030633061303033373462 +30363135663661613935316436633965626134646532633936313366653630373431363665393737 +62656530336331383331303139643739343936623138333330386633323938646333356566386261 +65333465613363653030666337663831663137366361653661633835633163376261343834366538 +62333539356135653766346633383061636235623233306366623430313039316635653836366132 +32653364386161623162323536353665333231633030323762393864646634646264663965653834 +61373966623436653238656663383931396632633137396632356461656233646230643362376263 +37376637376330353238306561366165643364326131666162393861653232303664633338626164 +33623137326365646363346538383533343238333061366430343332653137353533666334346139 +39313662323232353830393536343166346333353762376235343135353435623336393162636438 +32366238623730376239346266306131346561396437646531363461636638633831386563356533 +65356339636235653061333534343237663331303362633532353833333535636232356230393139 +64656533333931323966353637393163393838343136376561353336333866376238646238396261 +37623033616631653866363563656638653339353436323539316535653931313166303462386434 +66393431656534343038663239656564613861346666663935393437353637626635663132373636 +31363963303139376139643130623931346666373736636633353964613563613364303037613234 +34313035656164323263353835626230663565346337326465346563303530623134326364363036 +38366538613039303764306361303731396666393335643932346631633762333734323739383636 +66386138626535336132383638616239396232343837613133383837303465393933353331303862 +35366336323736643763346335393432633766613231303932323835386434356632666138386363 +31356336383532356365636333303165613037633535313733323832316333653431653634666630 +33396435383436636430336161386234646163343863386236666630646532613430376164373434 +38336330626233323835313865373134323331393430643635653433636364623830616137366230 +36393633656562623930313165393538326336393639333133386461306236356530376162613732 +38633363373965656434346333366364316333353139396233326236306366636664653939643335 +62613164313439336464643830323432393836353363373831616532363664636334383433653363 +61616166333737373936373065323737663836313261316634343130316132373865613266656463 +33386538346263333364386339613137313237373765623734343161333464336232633063396562 +64613832323432343866393433313563623239316161653831633338303232663563316235643561 +37383062336664333837646162613038396537323839373465306635336331376138303364656133 +64343665626464396538386630373537376437363434306565386432303865313534326461386663 +38626564303130643837343336623832366361653739363235303161356331313963393238636136 +34663030656333306636366235333531623635633065663531336636323166323361663035363361 +38313635366431393837646262636463343763343037643231616663613265313465343764336132 +38343165636263323735653065396562363366666330313135623939393331643161326437306337 +63623030646561626266626331313364363235313530303563366664343961363733613138373131 +33383039306261623334366365653936626530633539383537373739316138356364 diff --git a/infra/ansible/group_vars/keycloak.yml b/infra/ansible/group_vars/keycloak.yml new file mode 100644 index 00000000..8921ff47 --- /dev/null +++ b/infra/ansible/group_vars/keycloak.yml @@ -0,0 +1,71 @@ +--- +# Eveything is assigned to the realm defined by keycloak_realm.name in all/all.yml +keycloak_client_scopes: + - name: lakekeeper + protocol: "openid-connect" + protocol_mappers: + - name: "lakekeeper audience mapper" + protocol: "openid-connect" + protocolMapper: "oidc-audience-mapper" + config: + "id.token.claim": false + "introspection.token.claim": true + "access.token.claim": true + "included.custom.audience": "lakekeeper" + +keycloak_clients: + # confidential clients + - client_id: ansible + name: "Ansible client" + protocol: "openid-connect" + public_client: false + secret: "{{ vault_keycloak_client_secrets['ansible'] }}" + standard_flow_enabled: false + implicit_flow_enabled: false + direct_access_grants_enabled: false + service_accounts_enabled: true + authorization_services_enabled: false + optional_client_scopes_additional: ["lakekeeper"] + attributes: + access.token.lifespan: 300 + - client_id: trino + name: "Trino client" + protocol: "openid-connect" + public_client: false + secret: "{{ vault_keycloak_client_secrets['trino'] }}" + standard_flow_enabled: false + implicit_flow_enabled: false + direct_access_grants_enabled: false + service_accounts_enabled: true + authorization_services_enabled: false + optional_client_scopes_additional: ["lakekeeper"] + attributes: + access.token.lifespan: 600 + - client_id: elt + name: "ELT services client" + protocol: "openid-connect" + public_client: false + secret: "{{ vault_keycloak_client_secrets['elt'] }}" + standard_flow_enabled: false + implicit_flow_enabled: false + direct_access_grants_enabled: false + service_accounts_enabled: true + authorization_services_enabled: false + optional_client_scopes_additional: ["lakekeeper"] + attributes: + access.token.lifespan: 600 + # public clients + - client_id: lakekeeper-ui + name: "Lakekeeper catalog UI" + protocol: "openid-connect" + public_client: true + redirect_uris: + - "https://{{ top_level_domain }}/*" + standard_flow_enabled: true + implicit_flow_enabled: false + direct_access_grants_enabled: false + service_accounts_enabled: false + authorization_services_enabled: false + optional_client_scopes_additional: ["lakekeeper"] + attributes: + access.token.lifespan: 7200 diff --git a/infra/ansible/group_vars/superset_farm.yml b/infra/ansible/group_vars/superset_farm.yml new file mode 100644 index 00000000..a22e971a --- /dev/null +++ b/infra/ansible/group_vars/superset_farm.yml @@ -0,0 +1,2 @@ +superset_farm_redis_host: localhost +superset_farm_redis_port: 6379 diff --git a/infra/ansible/group_vars/trino.yml b/infra/ansible/group_vars/trino.yml new file mode 100644 index 00000000..9dc68bc2 --- /dev/null +++ b/infra/ansible/group_vars/trino.yml @@ -0,0 +1,2 @@ +--- +trino_users: "{{ vault_trino_users }}" diff --git a/infra/ansible/inventories/dev/group_vars/all/all.yml b/infra/ansible/inventories/dev/group_vars/all/all.yml new file mode 100644 index 00000000..dc060c4f --- /dev/null +++ b/infra/ansible/inventories/dev/group_vars/all/all.yml @@ -0,0 +1,11 @@ +--- +top_level_domain: dev-analytics.isis.cclrc.ac.uk + +minio_api_port: 9000 +minio_console_port: 9001 +minio_console_base_path: /minio-ui + +s3_endpoint: "https://{{ top_level_domain }}:{{ minio_api_port }}" +s3_region: minio # not used but can be required by apis +s3_access_key_id: "{{ vault_minio_root_user }}" +s3_access_secret: "{{ vault_minio_root_passwd }}" diff --git a/infra/ansible/inventories/dev/group_vars/all/vault.yml b/infra/ansible/inventories/dev/group_vars/all/vault.yml new file mode 100644 index 00000000..ea8d297e --- /dev/null +++ b/infra/ansible/inventories/dev/group_vars/all/vault.yml @@ -0,0 +1,653 @@ +$ANSIBLE_VAULT;1.1;AES256 +36366134663631386165313134373838366339656632613063636233643363653533363563356566 +3261386466373364373835303033396233373435326663620a626564356235376138653330373633 +65653733636339366137653663363236313562623939633330323135373139373661396264353263 +3132323836643365330a386236343530363337613235356462636663653964373633323130333634 +31363535363864323464303063373836656330313366393466613334656661633732333336386437 +33383039393561616462393762326131633963656332336166613533313961316435613261346461 +31393565666263323236613339303136396161626238336466306237313937306438343734363964 +62613165353933633666663837333963656236366136323163626366393939343761396266663363 +39653965663330343365333831366664353663636666323966373164343434326538363263383239 +39313063316564346635393064663330396465626531363262643936343966393832333530366535 +35623661363234336334336261633466643230386138363434336538333363393333373161356237 +66343730343836666166373638663132653462633536653235616565393935353231313335373165 +35636563623231343361346536313633336366623136373634633235663265376639396233316631 +33326166663062316134656465663636616166343337383964326534666539666661306462333366 +63303739373733396339663935396634393732333936343266306530343436303231656464333066 +36356131643065613438303839373430316430643363623861396235303538636164653166643938 +36333536623136616232373830636263653133633335393835353939376531313762623765636636 +30366636656634353631323431393863383538383834326566313663323163323733366166653962 +63323866396662383339346235616239626465616462636532613931326232343632633030333163 +37343463613165623935366266323235346534346636386435393233343839313262656236396631 +64663661666363626436393562316231373337326531376238656366303732303965343636333964 +33623364313035656136396539623766373032626238393234663436356362373163313863633738 +64303466623063306466363032373331346638623534633936323561346630336465386637343563 +62623731613538666334613630303262656430343065356661323738666136373231343464383961 +36366166396336623764303634306564363336663635393739316638663565386264376635386330 +64393231613033616162363132363966376134363264303839326164383434303436306431336531 +61616662343365663463363264323766366463343430396432643039613239373634343861373864 +62343830376134396131396636633031393737313033343162316232303533396166373830633133 +32623432393239346662346261623539393934306533356465393663653035323337646336356164 +38323131313831616633356333646630366161363530313134613534383536363432656364373166 +65633130666638613662346337366534613038613430666434376536616531373834306535333566 +65313763656632303138323937663730333736646631333866663736326138346230643031303139 +32306561326437626663313639346562626236633633623864613932656334303232313032383038 +63663036626430666130363136346231333634666635303065356331386465646239303033386638 +30386263353832343234666262386666643962313639386661356130313334393039663864396438 +32653662636131646566346361366535373638306262373066373036636365393436386134313863 +64663733626266373465643835633062326437303232333133356130353634636435306366623561 +62323331323262396361323937663062613065333764626264313337313532323838383966396466 +63333538333937396662336264636264353263336563646235663639393630653365643866613334 +39353564343561626236613938316538313039303064353437353234333537653961353662343330 +31343161333639663837396431643831626632346239656637353639653038356138323031663932 +62663666353866313631336565316262333830326264323437303763316265666561623237376233 +37636439313264653762656537373961666236613836373538376565663137356233396435366634 +35323035633661383838303864356466626537313963636432326133353663343366646534356333 +32666533333931633437363763383730613366316633316632326561643361666530653531623730 +62323861346361666362616435303939346230306566653261623839333732333538333563333639 +66386561336632343833323830663063316138616430663262396137383563373264313834353462 +39616639373935383761346465613162306534643435663532636634316361636237373435313039 +31653261316564323262373862663237313331653330396461303064376665663862323736313734 +30313736636566653535373335623531343466656332613363306337333037613937623037356666 +61643933356363386238336632383135363534306232626331666137656663326662623737366531 +38346336353531623331623133626131376431633330316562373939383064366136653334393165 +62383931393535396463376232616338376533623962336330393034353161316434376232383763 +61333830353932303430633730303862366466633836323837313564383665363533633261333730 +39653338633332363436313166396564323834363135386231366231336462653861386337323762 +37353566386130613136316566316565333330303462333965653464343766613939363166346238 +31616434666236666230613763333337646363643163336664653933633265393637393136326233 +35616339653636303865313237396562613061353331626665323835653164626433633863396263 +30626439333037333332306638663962386334623735316130353838333966653266636364363661 +36306366646334363939393063343935653437323835356437393266373433613766636532386266 +63626365336338616333383665373162316162313962376637666630633166336634343935653965 +31393463366465313131303162393730333239333534346165333335313630383461623339306366 +36343835633562313735336161363735346461356133393863363762653462366136343230633037 +37326432306634356132373830353862653430656137323433333333643536373233326266636465 +64353034383730383238303834333465633637303465356639323137386161386330656562353031 +64343262326161643434363666326230633038613764386666333338613262623462663239646537 +34626661646531343436303636393761346436386531613437326131333234656539306334363761 +31663031323764316666653232396537376632626566343438653936656538343964613561393063 +37323664636533666561376362666534373430636535643435336436393733376136383737316135 +65336231333430363033633738393461626266316165303435646138653236343066316638386437 +38383035393239663866643930646162656139633766393966663630636661646463643666373736 +65326663353835663336336661623130636533613066653130383938663161396566353138333736 +31616532363830393230343239366664383439646338323131333366666461303563386336353431 +37396531333863643739393135333461646161623031306138646631663730333562383661343762 +61313537653531356139333533636662383935366138613065633733626536663437363531373835 +35373039396238626232663839623863303232623537346566333338613030656339656237613535 +38623431613434626139356363313931333232636363373330323138663537643438383165386534 +35396336623365306133306164396430303738376136366462306338313438626339303863616630 +38393966613864366331663564623032373761356264363931313833326232663461313936333937 +30653934396631343937626633323161623835343133646566313237613862303537666631656461 +36306362663061343263313939666365303036346336353638396437386431356530626531623234 +66653838646332386336306237326537326262633065343231393565353935313734626330366237 +33353932643661636635393933386139383830333666333361666232393535363964303636353561 +39396236366230616165346366323433363637316237383139643939313231383066393637633736 +65303961616136306630376231313230373131633361313466393638383937396465626539373130 +65396538323030336563346338326566393130383732643566303563386636643361383962393363 +39613062353164613835313631353065396533633862333963356339393938613035306231303537 +33326665383661656636343236616238396439663664336263333632393839313063376639306131 +36626231353038646564626566393863396333336234373739393461383330396431353036366535 +32383539373039336535623937316638623334653864653734323034393834663730366332623237 +35346433323866343139343664656635633734646233636661383036333761626561376638366235 +36343531346361333439333430363433383035386133353639343864306266616430623530323963 +33636237393562303437376233613838326662613763346662623732353038656237616561646131 +33653633303635336562666362313662343231376466316165393339666638333633353866316565 +62363264663035653365616461636363313765626637323934346135643932366130613765373762 +34363535646332316531313935656461666364643138356230396233643832373531633165616430 +36373732303633376361653966623532373761653039373634613830343166323430376234353466 +33353537363564336235643230343533326237653766656662396139336536643966656138623233 +65343737393433393934353730333661353337616531316161353038663237346439646266363734 +39656661383666313739363634363530396263393364343661636339656332336563353134643966 +30303831306263633638393333643334343162343364396533303165656463653339613163333937 +30396466613636373736303335343131396666336338303233616637623537633338333633656336 +61646534333864636338633736363739383034323366643463316163616330363137333930313665 +35626138653035643461343435633437626362383731623862303231623663643833656133663734 +35366566306530336333393931653339336435343130343165373035656366306137386338633665 +36373063383865393338323762626534333164636462366462306464353365363065343462343661 +39663537326465306433663639343534653434633835383331353766393638356539326132316439 +63363236373664643931303534363930336562396336666430353666383662376238643739326266 +32633932373464623130363262313166393065653338393763323235626336353461616462623162 +36303765363135366361343538336231356138333039643136653534396633333130643436383430 +36616364643834386333386231383736396532626132636233353732323236306130636661623565 +36376333313738343730633261616539623661633337663362636334333564326335356232643662 +31316562383536633862343532343434353134373032343065303832363630353234643838386537 +66386239656532363131613661383864373165623937633837386432333762383261653431333932 +63383033626339316364393931383861383439353964363166303937323537373036376539343430 +36633865396665383565303962356435313138636430343561633535616639393038643564656535 +65393063646562623166333564376330663330626661363330303464353134333638306665346533 +33616231636465346466383130613363313334306632666131623635663434343834623736626632 +63323466346362633566656334623233396531613162663831393937343136373737336562303838 +36663635653233656264636433393763646130666133396131373762626330363461336162326465 +61346333363861376331656333343130373366623731356438616331383366633631316462373366 +66663032656630323862623962373433653763663265663430633030386662643331316135323233 +35613362646666613239626264663038363535626534623065363638646130633036663330333562 +37373761323062393736623537386133613436623263363238343433333039343032663139336132 +62393865383366363866626166313635663266316266636533373630663733386366303962346539 +66346364333130396338363530643665643237303635356131623762383864613437623431663838 +39383236663832323337353065386466303666316538353761623463353261343663353764316537 +37626638343764363735393634666237333464373065633066623232393231393262663630386566 +36666363623230323865386234633730623233653435623062363839383164363635376131373162 +34313036346561313162306239636132613633656232333232343733313265643238613232333135 +66393035313338313834333033336463633430353166663539343535386336333364373433333961 +63616664636437643532663262353736656331313233623862383733623937306139666136653963 +30366561313265386434633138326633623262313965666338633736303137636162636665303766 +65393031363738613665663336636435656233393538353932373139393335666632333632626533 +62643738376133386239333663353639653064363534353036326165396630653864383965323165 +35303461316334396434393439653039653236383361376436616538663130613862363531386366 +62396563343230373635393766386437636163313433343864323965643738353336643932366161 +35616264373661346238313436643061613863653765333435363331333561336563303363376135 +39613764613632353064353061326331323465343439303764653339356536303635373530333734 +35353464306438616665326435333436373634653337396663653837656338666666633465653232 +61643761666630363031663961663133386532376165396561353534613434343265396339343235 +66323332633463386337613133623939396264376233386265303638353762636361323066386636 +32396437643962653930663164396639373164353065323261313239366664663264396332313864 +33646430316438653736626635313336656263643666616639653138383232383035373635306338 +64363664393638306337653838653539323264363966313962353537353633636362313165316235 +38623232373739376363623861396563336238393832383833343434313338383834633132323164 +64363631326334326466643062323935313635373837313065336464653036326637633766316530 +35643666613165336436393165636562656231656134336234326635336166313439623561396461 +39363462666234323464653134353131333232623637383063656439336666313966373338396566 +66303233313131386365653966626562636262303762613366356132346438643037616137646233 +62373462396365366630616133653038656466343435313430376363633964353931663038313930 +61373766376432323363626530373030313662363230336266393133366330373938323763323165 +32323162363838633565643162623330613764656562613038346537613366653661386261666634 +38663539396130633336333139646464663365336334656464353130323736356132633039373034 +37303930356238336366343432623761393561663038326365393232386332643530383039313564 +32383962376532346236366164373933323634376138646436383165346639613338623935393437 +62383464333961623837303433353336623564623436343061363532353832613732646535333762 +32633138323936346261373431336366346365316165623861663336326634343131343533653562 +62383763333530313336343862383831616134383834383666353037393138343235613064373533 +32396134633330323232353562323830316432363966326361343933393961306464663732323361 +65313038373933643161623633306465363037326466383732643739363431363961316636643964 +39623433313065373662373936366437303366633432643632376535623763396133333439633833 +34663666666236643861316239663035393336626231383832323330383132633139356436313166 +39663364366139666537623432623837656662323163363838376333643461383932386461633066 +35636161316130653236333765313132656137303333393631666332613664376439326537333436 +34303137633962623432306531323234363737376431626365643633616631343062313765613661 +35656237613330393939386533366535393335366264353764376262346230313930646537353561 +37333230663035333265393334393634333939386332643335623638363064646361613966353864 +63386333383639363766623466613734303666336264626364613563373634613362306463646239 +66313264643365383462653965303739356630313863653537343132336335393162623632356466 +33363337626564313535613435643661326463366435653638356364316366613536333730376536 +30353632396535653935303331303061386562376133306333323139623233376632373663663164 +35633465383138343430393364326337396565663039656535373335306164356339323034623433 +37343762366466323235326161643633313132346638623962343239343366613039343766323939 +36383638326638343737653737333432326164663532326536373265366364613632646131343938 +34333764323162346135306532383733633130313932373062373634373165363462326166343635 +65323166623530343331343633653565366561633838646136616539323139363937383730333864 +61653336626365653338633433373038633039313530373431343039633939323836386631316236 +66346431373361396436613030616236653165643462626234666266613265643763393232623262 +66633464656138353038373931313362333237393638666537373363326561656232323161653162 +30326136626433323162343234356461633464653836316566633237396266383539656361363166 +38393166333438393235363963336666626163393230323937373638383831653236313164336436 +33646339666637393762326532656265666262323438633536633665306539666135646332656132 +66356533656331623638626234616531656662393530336332303134646436643238356433393032 +63376161383465396132383533386334363365646135656433343564643564306337313466626530 +63646138383438316232636232363735393334623430653535373730383334316139373035653065 +64633365376337373863326231643464616339666665326636666665663762313164646666633037 +64363363616430613930633765326432346336383961336539303635363262666262343631393963 +30353437373333393434633338383564336331653862363662383838386565343939393933356637 +62386561323433396463393662316135656336323432636163666436373332326362323064393165 +36613061313030396339333538653134366633306233346538383439313537333234323130623061 +39373066316632653832303936663632613737353765366264643633633737313861636361333837 +64316636653937323730346533316534383863306532346364663636663065636535373133343261 +38366464356635303465386633613036313766633030666263613331386562343163333936313964 +64306561616334353139373835373537396334646661376339613963336230383838313637396131 +33366163616532366265366262396339336235396465313661626565663633376131323161636365 +62616633373235356330643964636566303235643462343138613431393031376362656666376132 +35643137383763386230373465663061383939613836353662363365303635636335306639643766 +37306162663432633336646639623762613933353933626532666564343464316232326336343235 +62386438626466656137393639623833353365633561393938313864333165643265383665623762 +34633463333863356139343431623362636334626533303463333765616633313536326632366461 +37386261633836326637333739306265383532346465303066646133336532666163633431666635 +62393330343839316664646232393934633666613066613837616233356362646635643365346130 +37343261346136626663396335363737326331306633343264663532306666376361383937346237 +39646534346536336430383733633439303034313538633562373162616666396231376261623234 +66383831646531333661386638383139373966326436656335343738306566376533336463373632 +36623730646133396237316336323835393461633364383231353261656561366331653935663737 +38306637666335613362363431663535353031363738326532383032323465386131333731343237 +64373935323866346530376534373263306438656635303766393830623839633839393236636461 +37323565663163623230373831336332343135393061383935316262663439373239336137613666 +35663336653430333765383264623930343431646361636561306332356533316333376266613566 +30303837663137313136666462393063613761363334336235306434633338653331383030663136 +34353865623034663135386235376666333639313235323637643931316131643134373033353365 +35326339343430653138343961336539353630313632393930356264346633653839663933393330 +37613436303766666564343033656337363231396132623464306531666562313232633837666534 +37323663653866626534613865366533343232353264353830623762333738616533313264343531 +31373661306337363531366532353639643962316663386633373638343431636561656532353833 +65323238643534666535363734336661643030386364383766383663323234346332383934623532 +33343661303232653263343835626632336235643234353366663337353230646563663166656463 +35653465323963353738316534653032613861346332626633363534323163353530666436326363 +66326463623139623735323631626335303833376333346233343032616636326534343139623239 +34323734313039393538373236336231363938353566303361383731666438643666613966383062 +37653764333931376134306130386165373865633539306135303139393435306436353330356531 +35333834356466316337663139396633393135393966623865653235663864306230383061313330 +65393161306130626261306463333636623630343661643831383761323264636332303530303137 +66356438623366313761323533343532323665323663663738663066663063636665313638653066 +66393761383066393762653661393931393532363861373564633733663864306461383561373439 +39333066666537386433343962646235623632356632633466356230393337306230353831303965 +39626635323631636162343334333935326362336131646638306533343065336565633531653338 +36656131656439383134636330366161396366326130616431313762313963323935643039353737 +36326133396337353563623939356637636336336238653336626432353037346630656363366434 +64383338366537346332386438363535326332613638386336343036616461343832626233643662 +31383338626330613163626463313065366231376232353461353537376437393066306638306461 +65303530636239623466363935316438633035313231303135623738306430323231386464643931 +64663766356636623761383166656237613035643761353237323430393662396131633537613435 +31636364343639393561396230303035623636653164303131646332316532366136633432643366 +38323737313630346364336662363431633937353532356330306435643830313735636237643462 +32373961643066653330616131343561653162393233323765656430326235393465626239613663 +33316637316633613731323636386432316436623966613264313661636262656235386661326433 +38333336623262653665616633353431613835343137383366316338366430306363376563393564 +31383461613430393530666637303062366536353933636237323662666433346161616262393363 +66333166623264363034336366326165616363323535343765613939313263323434663634616333 +65346235623165306461663537323630616435356538386566353937646131326231366166346262 +37646430393434393263633232353138643538393338653934383939656661613663363865326463 +66386634613835663332643031323338346135346534376131666565313435396661363238666334 +35363535383863326532366666613061356366323331353632303630373338393066356566653234 +62323935313032323263613763393765353835393861666664646538393265313331303133306135 +33643331653436653135316266666131653762306165663561383164323035633365323864373638 +37373237383635316632396636653662386363633761636665373335326132363532306335343364 +62366630326332653062616132346434313233656133623664646538373361633437346164323261 +34323839316263653264333863383937316430613961303565326462333337633039316163363534 +30663030663533373537653266396339353537663232386237623936353666363063633061363062 +32656632656162616633613363356334633336366465313734623430303734373139336238376130 +65386466323939633866633466303363636362326133333634373638633836626233383937363732 +31326465323365323333303632353033313166626338316663616434373965373134623230646562 +37353436383066633439393239316364653861333466653637393935323737353838663733386135 +63343133616131343238633136656466303639613665353936343063643937363137386533303635 +66366637663639363464336136666235343135653464376430383738333231663264336530306330 +31313830373462643865623463646634613164396666303736363362383634643063313934623638 +66646132633737356465366536653866623831326362646336333338333930616636376132626131 +61333361616233393238336462653961323365373164343133366637313139346266323738653565 +33633931303531343030346330363639393437313763356630313562346636316462383535663461 +65353433626361633861353134366434633238313563396461313865313537356164303662366532 +31333832333634663237633231363632356665383030656363316533303933653562643535373966 +37303339623863386439363361623365623964653033636135353931356233393731643765343362 +62366361633732616333346636323239386634383030376534376437343033653039393735663333 +31666630393035346239303031343338323862663936343430373238323436623131626561663231 +32343932383365613531353338646337653738633630386630626562343164326237396438396531 +61396233373732383766616534616163653830653334323432616561393034393561623466356438 +35636562653338363461343639363430633730616438663230643865623262353939346466376163 +37623165663635633430333533366363373331383164313835633232363134326665663532393333 +34643539343730356261343837376261383339396266373961353433653638336465353764366338 +30343138653431323766663932616536666130623937326138643663383532613061653966313865 +66363531343466326465356663363866343039616161313233386661636532346538363263633738 +66306432316332653430626464643233613064656436323435616437633639653761376138313233 +31396634313332626266643534373631643839373863326536386361633963383865393337663064 +30343565383033316131623239663636393633383530303031323233613064646231613337313334 +35366166393164653539356539363131303838383334353233333336363762396639303237636330 +66353866383661663163653432396638323366666634663465336137336331373832386434323437 +34363137353363646534383936303737343136303035633135333232653261323837636364336433 +65353830303562373663366463383865623733343964333733653762613535636361353132323339 +30313363393962626631306261303062636232303334633265306332303438653133306166653739 +61646232366265396666383834323036363964383666623639373764656630313963313830326638 +39313963376262396636356332383137323537343363646533633739653037373338333336656165 +66353734653437303539633066303964393939326161393939653136356463393039653165363064 +63313861643966316135316362646233623964653762356465643439393963633033303130343039 +35636363623035303837333535356434656237396331393165643031346665393736376466323538 +38386136356233653236343136333538343961383936346162656231326266323132653366623765 +36316434636134386565383739303439333239636139393038383234633632353232366261373961 +37356264323236383166636161353334663964393232376136343139616635623663326233343563 +30666166343538306464626365313830373736363766653039356334393136633634656330313137 +30626339343538336566373464386230383663316365323137313263353138643864343331383034 +31663732636466306537376664656131366437366638316561326432306564316164643164363932 +34663761636239323538313630623938346638313532656661653131323231363931326132633833 +36613634666432373662306538336337316565653733333162333864303833343332353131366236 +65663963333033363462393432386336643365376661633536653865643933313938643663323437 +37363761646337316339336136636637363863613632373064396330646162373166643565613636 +62636233353261623262636237333764386166663732393231313664616237353733343830373234 +66633235373164353034623431613634356562646664303439633963313634613461623932613166 +39633330323962633337316463393938326332643665366239353961663036666339643664323931 +38613465633836346332633731613236623061653138353334303062326664633266373031613663 +38333161343261366632306562346437636132333162346538356562316163346262363334313565 +39346461303132656235656462373631623563356163366136393134393037303039656466323533 +65336162396161373534643537343937386365356439636139333638366561303664663132653039 +61386562323138323338383337633063386435343364633863396234396566663764336264613232 +66666463326362343466306239396135306334636439636133636637346336383238646234613033 +37383565626665396166326439303730653862353631363063636464323433363036383738303231 +31623230386138646639333935613930333139396633626339386561303238643361626265313231 +37323839333066323662393164366436323339633661313765623432306539623835316437336335 +61386334646336653431623433656632353665646538343463626530633336376134393164323463 +34343031393835326338643663326265636338336431323964383664363865373562643464373463 +34663139623562366364616435613838323730353134326366396237643939663661653332316362 +37306434343464646462346234346466616238323236313137373663326564333465663035303530 +65636631383964343230333465373633663166393335303433373463336230336462616632376635 +66356563663634613038303739656162643564643335343331306266303635353130653263303937 +37636662656466633236666163373266376665346336313364306262636238646631626131353431 +39376563333235663135303065646163323035373033633937626262323664353637326165636632 +39306537623334396434653732373964333565373762336434663763636138333562303666636536 +66383736636137373938313062396262363865666234363462363662396461393433353766626666 +34313765663938666237353331303034306264656364336639616334396562366133323865383930 +63353466643634316633373066333435636631643539623633363334323132653834383364663933 +38623264306463653161383339306430636664303864626364636335323562346463336536343237 +64303965313266303636666132393938356432323936396234356236306330653035656439323364 +61653639626631313734646633366136616334353563363066616432633635626432623930333666 +63313130336231643637366638376462636166633632326535393638323133363733373261393638 +65323261643738303338353963336537653338386638333237323035333464393663653632393662 +66623534633965623731633337383162313535336537386633323662613134346166633734343832 +39303465333535323565356432396435383732636331666632313266616133343466363531336466 +33306132366535373331343263373262373163353566343534363838306466376232343134343461 +38343139623738653463653762646339303930663761343062636138363061323634383732393238 +62333338616131303739386538653262623832303735306333376131326636373961663032303166 +34346665393265393761396635633366366537323233373932636433636365656238633565643062 +36643333313739343237633332376238393461383265653633326637643863353233613330383733 +31663065353664366237303933346466323932363564353165303763666530623163643562373533 +61353165633535383838356262366334373861306165363132363666393235633266646662316531 +65643263323134346464623236623837656262663332663362643466666632666339376531643861 +32326462376335333665383661383636303639393866633233383833373238646539336135633262 +37666162326532376231316136336165626163303835343035353531643437393963323061343738 +30376164636666636639303863346431383832366263373631656166663735386231643437636338 +38313261316132396136323231303930633831376534656432316132643962636537383065393838 +64333832653333323466353661666636653337663837373965303830636466643563323337663233 +66306263316263326433656462393932343166613763613933383661343539643962616333336633 +66353639623737623639353431306561633065393837326664303538346464643861323839383261 +34383837613565346138303130323938643735303531363164346230313338316665366535633732 +64313039373931663632363936656535643337343063386363313963366134306563376665313765 +33326338356433303034343735373936643638363963643666626662633665643639616137353037 +65343034643737353762653231356137396230636532656330653435393066346533393330373766 +38376137313465633338376162353434666630316566663166646166353933323535333465356165 +65656338393262393764663236353439303632653364353935383036353966383937303439646638 +31346263646361653863346564666263346166323033346666313839653036346364323037353034 +39316634343130323236363139616633666130613736393562656239396430343066633932633630 +36343135303161633731393934303462313338643930306132396666343039336231303836646633 +30343930333364636537623835323836626139393437336332343331336238316435313765306436 +38623135656362376366616633396661663563646363393336613462343531653830333734323062 +61623237613036313262313263656233626161613534623963303733386131306333643263646135 +63303263623563363162643539323663363938336535376663653333366438343031313737633636 +64363963653436353064383233653362343939346235666136376638333639373036376161666561 +38303337333262616532353734353632663030656334353664333764643332366364653837386439 +36376636346438383835373132643164623030386563653266633966313631313262633633333439 +63393831303439636330636630306230366136376637363664646262653934613832343434303936 +32333630306339303064366333326463623735616638626131303734613533663832343834623063 +64393934616433383365356561326132336337353236636463653338633538366232616264333730 +33313236636364343762623864306138356231666365646432373434323531326637633236666461 +33353830366534636664616166366465616530303432313139646235363534383863646632633064 +35656332643734653066613839323630343036663336386337656131303865306133636466386439 +34613461383563333930303534643830373966346162363432613765656261336637613632663238 +30386464356330343135653832383264326236343635333864656365626332336535316632353863 +39666533643938316331383937616662313162663561373637333439393665383661363538346563 +31663230323865383736626539663135383532616231373532636565366562626635316633376462 +66313933376236373965386138656632626161666633646135386431653437393835376536383037 +65373631366234663136323138663835306333333034623230656434623562636666376366613865 +39373661663164306665663436383138643963663634633237613266363232656263336635653766 +33363633373365356433653062613436333431666161656365303833623961393339373635333739 +62373865333833396164343262396362613863326133303931663865343738363161373963363262 +33616639643633333865633639616234383431656362363736383266353633383564636436643932 +63613532303762376164373738396336323237383264653862383530323266393937386434383936 +61386164393535316331343533636164313532396534326638336161353165303364666437396531 +38663630633632346264313464313066323562336339643236313961316463393664653561366166 +39333631313163313165656262353864373162356530306561363366313937666530363062636136 +63333961383366653835323432323961343033393130303632376165363530626335343061626531 +37326231353662613163343565333137323035323232613130653833643634363863653335383539 +34363538656230366461353938623966643937666161333436376537376363343961323563333536 +61666564666232306463356232656331316165376265303565646139346561353039363566313264 +37326336343836646563363236636637653266353761326161323630613163393264393762373161 +63643433396633643337373930643664663236353565643335333766653065386639396162346534 +65303738633937383665343337643834653165396264626634376238353162666536393031653532 +66333234666634613338653237653534656239313233373432656230393635643038303665356561 +65373566656562396234643131386463623134343536366266333033626262636535376138343261 +31316136636236633633333833373561366539373539643066636361616431363562373331323934 +65383031623135626633383662666131346265313631323133373036313732623830313562343236 +39373536316630313164383561333961633036323166333639323632316263643933326635393638 +66376231393238373166623964623063383938393630336466353439656630376565326632353632 +32376333653136393463313637396333333762313631353462663339666537623166663638653365 +34323866646561373232656134306330633866643337376539353961616231626334613137393462 +61616562646130393539646531366231633138323463333539356131393964343936613366656632 +32636266366431373862613336363266643738633739343166396665613236633632653339343731 +64343061653531346133633237373264383934363931353835333665366435366639313365656433 +39346337303538633866366339613662653364303365303963363331316164363162313633323537 +32316239303734653362633030306630643161646662353934386566616233626231373038326538 +35383835353065366232623665306562383336323133363332363161313063343563393366646462 +33346130663061633636313765653832323535393262323562326531336332343961376261636638 +61616630393831636465376539316336343866613438653633623134366562366162396163323637 +39393463346363313565303436313231663663363536653930653366626139373661623539633365 +30636337633965346436306134343933313733306533626563653536623463346330343432313433 +64313662613331323737316262363435646266386333343736373439636164356638346631323262 +38376164336562376539323164373866393535663136633264623631653161393331343163643038 +33326665306563616435323330393366376639373831363663333539343236303737303735653134 +64613562336438396263343466323964646635663832613136323132313230646463323237613631 +65346430663038343737333736636133666336363761396665666435376333616463343663323061 +61323664343564376533643531393762396462323537386366383965333035333166376363643632 +61643135316134326431303265336435333662656334616233626332323237613663363137306139 +63656330393062333539383230653935306638343763626331633037623037366461383066386563 +61653631313730323834303663623630666134316266363630303230323830323231633664356434 +39323830353637653161653736663236333965333466316433383439313135633237306433393833 +34353064336363653338383965393333626337346462663630373030343535373333326233616236 +61343562363837343065313831356162366637636334343632386539313866306264353639646531 +30616436333439376333643665376134373566333036613064643738383632363038303631363162 +65396235623866376431363564326634343562373034303363613162623730336533643339643838 +31393739663130366638356437343639316361313166313933346466623137613866303838316637 +63613932646263636366396362336637616530336131383932353863373335613133613865393735 +65353134376165356336323133393830613636383966356165306436313037346364393738363130 +38343961393030366564363562373231643063303638613862383966363535316134666134653162 +32666364616664336534346134346435306230366434653564643365363864613366336130343764 +30616361313762343866373363386665396663366639336163623636306331353134663034616134 +35613030663331646263666438323033336637383166643464656664626164333766356564356231 +32313164373539656632336364383238323063366438646464386361376565666264623562343637 +37393835356230323965656162626132373130373362366465393962333362373536356235623464 +36353236383936616637623163386232333865313631376634633536306439613435326565373733 +36313438653034366338363730393734393862363131613137653564363033643161663632306230 +31373264376565646633313466343036623331303732373531653764353033393733666665393964 +31386432623763623361353135353866643632616530383038323061386535613166336463663438 +38383933323162353336636239376361663561323235643531323562396163363561326364613736 +36633531356231643033386237633133663837326563326464303835386364613537656631333963 +38623633303164353230373238326363303963356164633432653238326163346132386536353237 +63616432656538383562666436613263623431343264623535646461383562663737623332386636 +32316434316234623635383962353362626231623261656361336235383639663439616434373462 +37393138393566353830303039633036333062333930376230366530623565623239633536393536 +36343439653638323163383032303064356532363735653238633736633537663033653264303933 +37636163376534636631633435613561363664626331663536396634323837343164383833633637 +31663736306232346336303536646634666565333361383432333239653636653863623136373430 +34353436356632363638333638656464313934343362383631393864386137326137636632383963 +63373638363966616366393766306464613833343666303530333132653033366532663366316236 +65666533383362633435623433333266393139303661613839646232616461346663343334383331 +39623738656630326530653532643262353937306336303731383430383034643431353239646438 +32643961373034363166663130626432396235613134383037323935316164333434383063353334 +39666131376562333435386336653064633061383130663263323865323966633364316366343238 +37313532356533363161653266616535616233343563333835663666663139323733663834386535 +33393838303733336433393563353261363632656339353763376232633464616338383561633736 +31353839623365616533383931646635666338363266633136316363313938653065663234636439 +36393932623639323765363335656262356334306561306136393366366535393634363264643762 +31623332363265363861343463376263383732386130376131633062353337326635336561313435 +62316466323562376136653066616534396261623833353637363538303735373539393539386662 +62376265663036346163313464366464303337653239376232663339326332306539386536303865 +34353334353763623964636138333035306134316236366132633137343462613839303066623738 +35643762643032616635373938613863663363383130633265363765626432633565656431363564 +66646336373863333561316235383163393939323134396164333734366237636532616161393939 +65626666346431656363663432656331343334306530393730333263316464343661383163633362 +62303034313866613831383366376130313263303666363262346430636364343566633037313765 +34656534313739366565353364346437353561643564396161613137316365666336626231373235 +35643033616533373966633861393332653631333139393161333761376164393830336163613238 +34323161646262653363323139643136343934633139623430633332666637636566393566326434 +63623234303634353439643538323439333038393764653463356138373939323830343831373334 +35306561623530323436353531383738386331396439303166356632343830366536333435323735 +37323738323733376233316561613336636133396131373465363031353363333761633766313934 +61643037393966363832323936303633376433363265393862643362313630313331313833663538 +36323034346331336338376461303332303738353733656436613939313837626139356132316430 +61663266393631633537323737623137313431393166393433383933393864383734663537346634 +35623065626234316333653863346138666436336663653137383631613063613366353132356539 +37346165396265656133656666633565653134623333323363653666373731353931663466633664 +38376565333439653831363563306462333534663037316231393231363938376239666638333363 +37363362373638386332613633646266623735663536356130353836646532316632323433353032 +39366537653636653236353834363734333632666630616662616436363961346637333838353534 +32336164323634633039316361633866643536646264303436373431356165393835393963326261 +32333166373632656135306534323035353834376234633163363432353665666632343161356634 +36323562646531636430363338613461646262633038383862656665306465393834613265613439 +33343061396361346430323138666566326361373963663737626334346233316432323766353135 +38306265356635373466366337663037333839656461646563363565316530333135633865393033 +33323337626662336365346339643663646263313938393130636439653461663662636632313366 +61393138336337616632383735343963616631333631623036646632326431663335646238313132 +39333730393231343633653963643965363033326235346362356439323930343937363936653638 +62656338616132396363626537646136363139616138356166363839336638353331383836323736 +66363562303966343231376461313131666665646434306166636538616166303031343765306134 +31386136353034663664656663663137653734313830303538613130333864623339653235656239 +30386332336463666365356462333564623137356538373463333235366632336138393836366537 +61386433363964636236376333636338343066383834336463326630313539393638393634303639 +32386633333135646337636264353638653533336362656632366464316335306436653537616431 +33613233376564323633303237336635363761366335326539643565353539366331386434663531 +35353765623462633232343632313238363036616565303138363232656537363137663335343464 +36373735353530643235623333336335623036393362633332366566383734346566396232383566 +62306463383964633538393166393239333334343937633132623566383337623961653432653166 +30623532353230363039393533643431626135313935396461343834313263363862336631306661 +33326235353935336530393237346165326465643961646166326638653134313439343932336137 +37326435343239353466646462656535353161393062663031306134306436383631633666346465 +34363261666261666166326138633563313263326631653934633062656163353633306330366433 +63626430393630323966643732386466616133303337383835333035373364383464393962326638 +32366561386566346463366164323761333738363131656263333434386438363639356365323163 +61373035313935373934386162656236633634343563623035623263393665623164313339306262 +36346335386330316639393061653962626464323535336264396565613530306332323839303461 +35663962653634646331336566393837383539393830333733633739613263616261386462626531 +39653838353064336561663263386237373161626336376539386136393932666364326536346632 +39386230633737623836353061623435343861316536313962373535326366616264303333363235 +32353536633633336635306161333665343138376237396662623239336535613430373437373339 +34323637303166313835303130313539373139353537366664366165323961363736386533356237 +66333033613431343466653066323562646131363831326436613965666365393539363761633836 +33396238633533633661373531313339663363636433646562653535353137376430326562353736 +39663535656662623231316530313337396564643234356334386532626336323934356463313536 +33383965313563383161393232356535383030646133613334356132323034373634363964646637 +33393934323163313564306661346431663336376636326237316139343336353331303934323965 +35346165636631313236313633306338623064633963396430343634306636303962383931643536 +65373263636431616462353432316230386633356664623165323461666532366637656466336461 +37366263653430323237303236356235653538356539383134636538636435663235353265333336 +65616166623661353938393561303531373964616264646662393430663937313266333535383731 +30343366323263356531366532373562626434376139326564643762326438383537393565613634 +64633730343838326334343034613635383439336232316632643737333930353833383836663665 +63623934363333313633373132636563306361396631376235396635613066383766333538636564 +64323637656330613762646436636261623939666331383436383962636662623634386432346138 +32313961366132396438643032623164633162373965613037323363396363663462326161633438 +62646237313038663964653231366139613163643264303839353838393161616235326661623833 +33383562643538366432343366333939353466643766306139353338396639313234356661373661 +32643863636464343864393064363430653463303465383365643563363539323237643162313736 +34313363313033613766663930356633656263373666333864353235373166663830616233383137 +34303962666466356637653134323437633535363530356666376231303931626365313233343766 +32303335366635356538643036306336306630323736303665633565366334653430333830653636 +39306437333163633438383434666531363161363237393137356661636334633132623165663334 +61656662376463393830393064663963613235613031663234383833313630336131303934393061 +64623332613535316333303233356536353135363262303566303633343239336562616565306235 +36343835616265396462653566393437336634613535643230653530623330316531326139356264 +65316236323338333730326134303865343336613036313832303035353430663032656562373832 +36636432346436633937326439393765646333653036323637633765656139303636396630396462 +30326236626333366432383330636161393766636235623964346565653662396166623131353031 +32386632636566383765313631386134326165333839323230343162316136383631316436373566 +61626136626631393831383561393537383130393636306262633039656630366462663236393262 +33353964613539336335303832663633396664643161623366626632306364623433336437306436 +62663864323231653733656131386235656466626136613032636664373332306331653933386239 +35356135643266653935613763373266626332366565613465323831353035636162613739616666 +39363733616263623237386233373839623063353161326534336264643332346631663066386163 +38636164663563333961643834643335376632623334633765303564383335656439333766373733 +32353061613866363631636538303431393762336534313531356537353533643934323639363533 +31663231613535376235376437366638303535303762323730333231316637336262303032643166 +36356561386161333736356134666231353837656632653761353630633365633531336338323936 +63386665376562363765353138353366646532623936376365306366643332323764383538313566 +66353964633966376639646332613734323836353761613530626464303533353663343834653965 +61323332616432663330343835636163333937333235366161363630363733346631303563656431 +37653636623866333538393937313936623334643662343461626463383764613330373461623238 +63356261386536316561376363343363646633376464333733383531613330643762366439336430 +30613261323631323239646437346334313339353938613966326265663734373133373566636535 +34366334383932373038316232353431393930353562343537323066663534303061656233663166 +64333835393533653630633639626265666135636635313339316635356663323131643631333231 +33623264336635653931373731666239633536386234363733376266323530356561396166656234 +32663363396132623366346364623263356236323164333366303664613266396438643334373264 +63613131346562333431323831663132666262386264393439373765383162323261376136653930 +33393034636430643930333761363731653832393866646438303435336138316164653633616436 +65383031366632386264316635306535336234323036366262623131383735366365646536376230 +34373062313139633461666464346339643931363234666236353435313736363164656462646234 +33346434306330316634666331363536666661393862393166303839326436336231623262656361 +37336437383633656234373435613435663030393866373563643737663030306333303261366331 +63303464323064306232393633313932303239383339313730623862633365643532316162623039 +66616337323737383532303039303235626630326434393833323264353761653331393135303561 +32633131613038366564306236363230643233613337613231303735636165316630663438303538 +35346563386636616231636237343962383234633836633430663437343439373864326532616464 +38643035653164373031623435313333386134623336326236313765323439366430356335383539 +65333963656538313737366332306439383832626536613532343661633463656365373937316632 +65373362396237626137623366323230376366343632623338373264363433666638356335363131 +32383163666665623931353131636435666432353534363662373264366663353061376234633164 +30653939353236333866636239383164363763376463363933653432363831323139356537666336 +32653265393064323664306639393433653561636363636365656235373936343836613934363961 +35393132643038626364346266376539326665346432386232623764363036313732343938383536 +62306362643834663032663261366639633466633937643234393461663630386363313566363930 +65346264663831343536666534376631613234336337323832383766383762376535653030303630 +38643130363639666464623363626333646430353035396662366130613332363233333832613232 +64313631646339616263373933373839633966636364616164373134653833643537373630376561 +39383534663331643166653037373661646435336132663236373733363834316633323063643961 +34363861316334353939393731623233303833623032623030613432323938633938366161323737 +37336566356630633965366337623432376562306439393436396534393731316138383664616337 +38623035323934323863373238656263626430643963343863383931613736313939653465643062 +63303338356633666166636536313834303761636134393530333738363532343833613934353565 +62366465363762656265636536373266666430633033643436313462613639323633346534396461 +32306461663831356339313933633331346365333733323862613935343065323933633064326433 +35636333366330313030313838613938383663353166613034383337396238303034636237313938 +32303632393134616532636530326464393465653065363036613932643739323233623739646538 +30346531303238306636663232393830626533336432366335373936623538663734343531363136 +36376132653538373330653535643334663262303735353537343538333532373332663763353531 +35333661313637393833643565376664386562323135643333633234386234663231323461353334 +65653333653639373165633336656531353237666537373263393631343365366138643966376339 +63616238656530363331373937643139643161616661663863333666636561633961653731333733 +63336462353832376435323336396566366635656230646438373363346166323566373762646430 +32333164373336353566383631666665633733323530383636623962636664393730386262376132 +39666161336333623033373761393239303363366135653236373961376336353131633365376633 +30396532383135353637666365363230616439663464383035356664656237363565346338333063 +37303138363834353164383665663730623562353735653663393234616339616434653166666130 +63636336646636323438323235353030663633326265353862653766356263316339616332376134 +62633430303936313162306538643266373464303336393033336338646330386666626663306134 +65633361663062313430306161656564303831666533306532663439663061656434653064366561 +37393363623436616139353537666561336439643366666631666463666166353764356136313863 +36393034376463623763306632306566393132363330653938353034306564656665303463323930 +32333262323834633931306538356163363166613737393931336236656439623832656433366433 +36303661626137653332336432653535633531316534633630326366306330666537653864313065 +30393534356437656331653537323134646538643034393666323663636563373834366132643631 +37336537353137343337303835353534376530626338393637626636323032383865393331306265 +38386261303630663536396331353731343761353833643235666565633634303065323466376264 +64393939373431613530623336336631643637373139653235613139643334626663653136316439 +37353236366461343461616266623664336637396334326431653838306264306137303865613761 +63643430643934656230626234356535393138353931356336633138663764396463366432323531 +38396163616239373165356535636639333134323463656335323031366263346236383935373234 +62353965383137316537313562393730336463613030373563346566623065613961393465333234 +37396634613736373534383966383430616335333461623361613162333266643962613962393634 +39633235303430656465306264363936643363646235666161373434376136666531343535313961 +66353838643463613732383339633662396562363564336533633563623763643237363739383230 +31393263636539373934383433366461363034303631356238643236643363363430363638373235 +65363537363239646230633363396634616137613264373637363432666561326466306564663532 +37356266303766363135333134623236313966396534656337313038663635343639613263653836 +35343434306363303662636335376138623761313263343137383734396430343966316537393030 +30313031393466653635393036643334653763366139343862313033373461343839613462373032 +30663937336137623061643033343466653637316232643634663836313337353033323465316361 +33623431653662366261376130376338353566373839346363623061646634306233323930633563 +32326563366261396138376132653561346232353863623436656137663965353030643764386261 +37346432333861386630366532363033356135643264353464386665336534386239303963313734 +33383734656262366134663139386161373030613333633133353835353231376263396161386531 +35346538643637303761653632613563376439336633643464666339313161626136336131313931 +34313666383863623836303434366638656139613035376234656561313438633136383666663865 +36613266623039623265626164623438343831333131376164653066643066653962663931353539 +65376666626666633136323763653765313532643035643935613935353934666166376233383863 +61656130323930326130396338353738646539633737373430633330383833373939616633303831 +62376437363335396430373435326430396530646366633764646266636165313835343536663933 +65666637666261326164386661383737316337336363306536373062336161383061306562303562 +63633838656636613536346461623562346163646135646532663634663931626662316237356432 +63346535633937363636633230386634336261663164623663376235343463336135313230333836 +33646135313131363431306235333961373631643862323563373739633733333539323339616566 +35356364643035356435303831616665313435313965313763303835663536623439393365356562 +39396333363065633762333130366663623838643965366265633662333231393235396462333163 +37396139313830306533363936343838386336393333386630303261663036313636336465656263 +61653037646365636434326538386435646334303035353465396438646164353561353734656638 +33666163353830303130643466633838313838336665363234353666633764313766313661393736 +39353738363666333730633564666366663033313538383165366237313135373933343430323465 +31333264386633333463663963353964303265663834656636333133666262656335393732643430 +62633936303563663933646338306132663463613630353536346434303639373165343434383435 +66306365316636383832333663643139666663383638323139613632393134356666633132393734 +35636434323532323666666531313336646137623863373236646138643866626563333038666261 +38613433663364306161663432366665643638376333663633303964653565343537323764343435 +35623435663835656232306465303530303936646332626630626565623362333565666236396162 +37356565383732663830653637643931353436656431316265326531363639643437306239303062 +63366663303461616564623731333533643839303265363931326463353336313961343830366264 +65343239323732646339323339366265376566653739343262663934623062633063366433313038 +39386563653030636437383364313332663333326661393033353665343965656431666265323139 +61336330616265623266616466373732386663316134366632653561343864396265346231356337 +64646434636438326362303763366165623237613936663465376562396639366532623562623864 +64383934666130613235306363323637653032373236306666353165633665633931333831323238 +36626434383331633634323131313633303531353165613836666430373035663838326161333166 +61313033386633366132646137323336323366616466366539613062623936313231316165616130 +61363738313863383961646339626663646137313463633730326632663766616439396130356461 +65333834613639343334373134326264313563373061343961333734333362633131666338636532 +32303137386230616235 diff --git a/infra/ansible/inventories/dev/group_vars/datastore.yml b/infra/ansible/inventories/dev/group_vars/datastore.yml new file mode 100644 index 00000000..c9a93d12 --- /dev/null +++ b/infra/ansible/inventories/dev/group_vars/datastore.yml @@ -0,0 +1,10 @@ +--- +postgres_db_user: "{{ vault_postgres_db_user }}" +postgres_db_passwd: "{{ vault_postgres_db_passwd }}" +postgres_db_names: + - keycloak + - lakekeeper + - superset_farm + +minio_root_user: "{{ vault_minio_root_user }}" +minio_root_password: "{{ vault_minio_root_passwd }}" diff --git a/infra/ansible/inventories/dev/group_vars/keycloak.yml b/infra/ansible/inventories/dev/group_vars/keycloak.yml new file mode 100644 index 00000000..dc492d3c --- /dev/null +++ b/infra/ansible/inventories/dev/group_vars/keycloak.yml @@ -0,0 +1,8 @@ +--- +keycloak_db_dbname: keycloak +keycloak_db_host: "{{ groups['datastore'][0] }}" +keycloak_db_port: 5432 +keycloak_db_user: "{{ vault_postgres_db_user }}" +keycloak_db_passwd: "{{ vault_postgres_db_passwd }}" +keycloak_bootstrap_admin_user: "{{ vault_keycloak_bootstrap_admin_user }}" +keycloak_bootstrap_admin_passwd: "{{ vault_keycloak_bootstrap_admin_passwd }}" diff --git a/infra/ansible/inventories/dev/group_vars/lakekeeper.yml b/infra/ansible/inventories/dev/group_vars/lakekeeper.yml new file mode 100644 index 00000000..90aaebf7 --- /dev/null +++ b/infra/ansible/inventories/dev/group_vars/lakekeeper.yml @@ -0,0 +1,7 @@ +--- +lakekeeper_metadb_host: "{{ groups['datastore'][0] }}" +lakekeeper_metadb_name: lakekeeper +lakekeeper_metadb_user: "{{ vault_postgres_db_user }}" +lakekeeper_metadb_passwd: "{{ vault_postgres_db_passwd }}" +lakekeeper_metadb_encryption_key: "{{ vault_lakekeeper_metadb_encryption_key }}" +lakekeeper_bootstrap_credentials: "ansible:{{ vault_keycloak_client_secrets['ansible'] }}" diff --git a/infra/ansible/inventories/dev/group_vars/superset_farm.yml b/infra/ansible/inventories/dev/group_vars/superset_farm.yml new file mode 100644 index 00000000..f92c86cf --- /dev/null +++ b/infra/ansible/inventories/dev/group_vars/superset_farm.yml @@ -0,0 +1,6 @@ +--- +superset_farm_metadb_host: "{{ groups['datastore'][0] }}" +superset_farm_metadb_name: superset_farm +superset_farm_metadb_port: 5432 +superset_farm_metadb_user: "{{ vault_postgres_db_user }}" +superset_farm_metadb_passwd: "{{ vault_postgres_db_passwd }}" diff --git a/infra/ansible-docker/group_vars/all/s3.yml b/infra/ansible/inventories/qa/group_vars/all/all.yml similarity index 81% rename from infra/ansible-docker/group_vars/all/s3.yml rename to infra/ansible/inventories/qa/group_vars/all/all.yml index adb1587a..c8386634 100644 --- a/infra/ansible-docker/group_vars/all/s3.yml +++ b/infra/ansible/inventories/qa/group_vars/all/all.yml @@ -1,4 +1,6 @@ --- +top_level_domain: analytics.isis.cclrc.ac.uk + s3_endpoint: https://s3.echo.stfc.ac.uk s3_region: echo-01 # not used but can be required by apis s3_access_key_id: "{{ vault_echo_access_id }}" diff --git a/infra/ansible/inventories/qa/group_vars/all/vault.yml b/infra/ansible/inventories/qa/group_vars/all/vault.yml new file mode 100644 index 00000000..8a15ae1e --- /dev/null +++ b/infra/ansible/inventories/qa/group_vars/all/vault.yml @@ -0,0 +1,718 @@ +$ANSIBLE_VAULT;1.1;AES256 +39356562306463383636633232663133363663623131303830336332616338303064613032383333 +3531303030626563643136346336353533393165386637390a306130383636323166316364623538 +36616561356564333266353132306562323564303233356562383162363961393162336633633936 +3761393736353666620a326437646563306262383830393033353964343364303664383663393139 +37393665646366666563363663366332646561303530366464373738626330353530383338663634 +65313866303838623662353831343038663439353439333863316437666564643662653132393665 +32626431616631613838306237383966636661646130353932626466336435373265663862613836 +39343135333532653165616531653933316161373563356232643461326664616465623739643836 +30653165643438303239393135616364343934323334326263666466343062646536333331363639 +39353663333038333838633963653035303766626565316631613637633933356530383938393736 +32613937653136623464663539356437313033376636303135393762383437636339353164656665 +36363461653532316438306338346336303266383062393863666330663937326361613234303131 +31303530623266643464306239656565333864663962353432386265353534393862343834343962 +62623931656363363962313263353632313030333465373438616163356437346532356231626361 +63373161353635356231366433393439663239363137313633616638666235643862353933306566 +66636631313433633939626161386239386165653364643265313835643961386532336431313961 +31393134346132393138383136633631666666383461303933316532613134636633303335313865 +31316533303936313665373566363437386331356466666632623338303464666532633131393563 +39643136343636353232356364663233636563363563366165393038356136303831633335313035 +36343836356636323535373261666534623966623439393261663762653361373836656566663539 +31363433646364316435393739313136643331316464383531386331666361313134336634336232 +34666362613166393064656631326133666161383239623037386639623231616434663730346463 +64353466636339643936333663643831336337313633306166316164373537653564383530393735 +39623136383433656166306539666163373862396335333035626535366464626635646335636633 +34336237303439363935643133306466333338396438636334323530323334613864313765373061 +34393364646334663865396161303432303138313962376438313962376133393931393232363862 +37636138356635386133666363336136383863633437383030633166383136343231383063656664 +63396432303236366463663838303939373537646435613031366530306565333335346634626339 +37623334666631313138343666646564393839323538333734323964306261303263303865633765 +63346633656336383861393236373031666661663835303439333932636262623833343538653766 +39356238626137623731643630623866313435383666363662653531613337363464646461356232 +34313936663137376365343664366666353936303735666366636330633434643030613761353730 +32616532343031613531643335333130313138313663303537326366343631343831343131663634 +38373330366162316636396438333933303962653562623766366233633039643239316634306336 +35633964616661633764343938363838663133306636313431363635643732343532363234326334 +63356361356131653964356639323035303339393530636639623233333263386333636561343865 +62643466366561313939346537616438303562353462333366326239333732343865353366663464 +34393431656638626437343365316232643164386432393038303538313935353937626464383838 +39643532353065396265663636393232323365613535306464356330356463633436383037333139 +65626563333661623031323335396130373436383837303661653636633330323038613766616431 +31653232653462313132623533376466663537363038303034616332373066663562303535383631 +63316566346566333538386363373464333163653166616232393766656534353763633366366234 +34366363366334346162653866616363393837306531616162383536386135356132336530633664 +38396161316565373738303632613162356332643334346137343866356431383339316235333234 +36653533376130626662343263646631383865663866636630303731386230633962343632373138 +35623466353739623530613962616339613361396537326339663733353162656363383365383235 +61356338336332633533663062316534623663356433663136333832646537643661623337386662 +64353964363436613631646562366633333064633064353730333862383834386133613365393931 +34636265316435333734383762663838323238303165333239376565313763353932366437343236 +66616438623930396338646432303931316465613961393363636235396663346431316466393966 +66376261333635656266623462336239663937343662393366353136313932623764613863353033 +33643364346131323132663338313566666464343733303331616132653238633039303932323234 +65313432303132376130316131303833386166343330323237666361653064653361336237303862 +61656638373934623865666336363862316563666331626436323766396435666634613739303735 +34323338316133323738313035633764396232653439343265373364646162646161613637383235 +65346364393337613461346435363733346330336161343432633561663631663531373733346435 +64326465616461376337613233656330613934393139303665353231613533396436393138663038 +30666533313738653634613765333331633932313234383064316638633361303933363339343862 +30646332633030663561633034663163643931623463633231393064663133336631646237396337 +32353261623934306437396562316664386238643135346239656333646262363433356339383738 +38663433376365633333636664356266323135333965356166633336373838383639346261343538 +66386664613235326261353837396264636462353034393930346639643336626536316431353035 +39316162653838353836333934313632343737653366393730613162383563323531393137343238 +39643036663539366265353732633930343537633135633265623236643062373238633233383930 +62383136396335363732663238326566366630343032613535653437643236396635373334356564 +62386466343661643839333561363466396362663438633238393033363632393732626437323537 +66643837393766343230653339336430373438613939393132313030323837313764653234306166 +30306663653533303766323835383333626331353061366333653831626262323363343935363166 +36616561306138353734626362376437393835623934376337323033326666646239326133323433 +66633532333136373366353663373432373464323966636162386364656531633034626434373339 +65646130663839643030373563626366323262316532623136383133303335376230366161626163 +66326438663030353630356339396433366137653964656336623937303739663534393266383436 +63363339613666313933643935346162643231653933353030316439346133633733356632333136 +33363564666164316365363136313763336566306232383362306332373234623539313861646131 +64643439393436633937306435663037363366386332393434626564306266303861313738613938 +66313433316565393666393262393435653565333532386631323230613330623037363230373332 +38386239333662333239376330383139313433303934623033343463643465353031623530613262 +36623365623735393034386363343366663631366439643263346637366638323762323163393430 +61653134656538636434623936393936333233313931396463323532636231666162323766323338 +38633939663738633933613136623266626130623364303566656530386335316163343232353039 +35366537613163326534323663646432613838353166346565346262376236626134383465616631 +32343837363632366131613433323637623465393132323662353764653431326463623234353438 +37363730393062303234383363636639616432363631356636346164363538373838303663356632 +61303730396362313439636338323937333361356663366535303934373730376132393361303234 +30633833353234643665353937373038393434633635316362313636633066333238386662333466 +32623633653236396564383865303532636565386463626265653730656262643264303533643761 +31666339666133393238303131303964306630623338376632343236626435623939343933666431 +31363264383065333730346337313939303266666135383565396665313037353565323337373161 +30373539623861313136326364313131663463626136303064656563646333316337336537353232 +37376162383162633232653262376435313464326631333663303138316234306663323037613636 +34373837653434356364333033643164303262383334636332353037303530353161623034326334 +32393338643031666337383530316563633966666132326137383437333861623836343865646536 +61386331323539346666396136616438623235633531323162623830323539323936646335373266 +38623239633765373231613430663338376131616562663866656532323238393865366338353237 +66636566373962316665616137666235623061333430343161323531316462373966613031353432 +64383833316166363662616439393161333832373161363964633730306563373261653336336264 +39656663366463623038646437326638363234623435363836663539363661383433323736613737 +34663739313364323838633333343965393838616337383934396239336164303837333963326335 +32643163623630663731386630663936306637633036303832383064353130623032376562326234 +32383430306439633533636630653133303563323734666163393639623136643363633539353731 +38653866626163313262613866303533343861646534363865626439646562636134663036616563 +36636532366331306637393534303738646264333762626235386137666232366462313262353331 +62326665336562393936396230666435643866313237356537376432393063623866613762366661 +36663537396535326365386433613561646438306230343035383262303462396264316666623162 +36303432323232306431373938306462333564306666383661383630386264383934343836356331 +31643361393937393063646532323530373663306361376237656565353739373963333937613037 +33653131393330333033373063653462366637326131303734636438613162303238356165613563 +31306138303733643265353636646564333361316531643133316466656466666365653664643563 +30663336326138333133383365333065343030653937636466366239346132386462393330633165 +64383034353036313434396262613961386161306134613334396239633636386564613338386363 +35663466663131316237396238336438383431373964653235616335646665633830336630616464 +63663534613361656161633439623330333234623635373037386230363338373633313030613737 +61343738383162323734373036623663623138333435666538633163623863336330653562373431 +66346632333435376566343431626562393563393065366434306638616137373437373834646637 +33306534393332316662636663313261383135636135323262386661303733663736626436616532 +38306137623861653937353935333937636563353437306434326136623165343333636261376231 +38383431313935343939363563636562383937646466623336623830643531303663343633666639 +39656166633038353135616262336261613261303663343434666534633130396166626138653431 +35613335626466393135373464383834316662626337653563316132653866613139613565383961 +62323661356363303535663430666263666665393032366266353731643165623666373639666233 +31633063663235346232333265353033373735393730363337333138663963636436633035633863 +31373032343363316333626131616666333666333764363066653866663936346539663862376661 +35343435313461346434656162626538643763386339383332336133306334393764313864393566 +63323739356163626661666136323663376365623264653831313763653331323365633835656166 +33313535376134356337306166356436336562393664643863386536653935663235616534343565 +38333431613935383766333462663836366164333063636537333164343539333064633433326662 +35643531353361383533363330643764326638323537613037653132386137373439383661663535 +61306134313066623062613938626136633932393335376239353063633436633532633839343932 +63343332326536376639333336343164373066386466363238626337363731373332646337353236 +63626238326531386464663633653366343666623332633165343061666232613561663635356636 +34373632616132653565646131616132626362303739336337313565393035356463323638313935 +39366632373365626333666238383836366265623161326432373864633434626637353861636233 +35663266643334316433653333366434306337633361303937363939346339383132376439663465 +66363431376332346433646662343435653536356438383936666234383163383432303934393662 +65346165303239643735663665623137633636346263643736613865663432656632636464326265 +31383034333766613463306134626365646233366437333365353838303964666564646131393265 +63343666623963626630653631376132663838646434663730386637316564313535636439306336 +30646364303866363264616230653533326333646434666532366165626466363631336231646632 +35353164616662646637613739626439616265623830353562333830356136653062303432653263 +34336464303838633939333338363633653264353336376632653034643438326663633262373138 +61363038396637653166393862343635373335333566393837303135333530613233646664343134 +62313763666364363866346564323065656132306632363431383364623064323631306133636564 +33326433366231313233313362613536363836323835353866393639333230353437303362663633 +39353133366539396239303437643431653834323337623634323962633766376661386331396463 +30396461613431316565653231653432393438313335326637633661343839616262316562353866 +65323064636366353433343264313663363339393130646564646463613033623536383539636563 +33633134653331323064643031616635313163626537626230346166386461623132636238643931 +31313665623930663935646163376638623539396536316465323666633638346666323632363131 +61656535383834656431616432633265333961383764616634373161313831643964303363643937 +39303633376438386136343561363234336539316133333734666231643635326364393761613863 +34333233636235633934376661646137336263353534363865386236643964343365663332663463 +66646134653461386164663539353433343233343331653539356230393732313833646163633766 +66363439383266393638366565373232383437333639336336346139396431343466356338633237 +38653936393362646161323239663535363638326561636263613031383065613361393034646338 +34633234323231303235666464373661636632336232313265623261323437666163616463616636 +39626531656631666532303434366132333264396138613733363931353139363436313062623231 +33373536663366366330616530303764366230633734306539323235623735313063656638333361 +33396331373064633138663038346435636436393138656561316264373562616433346664366665 +35306532363033663539633062643832343532623163373638656235393865383331663131326138 +62643166666161666263396365376237373137626233633666313236303232343666393830393131 +31383664653065653836656134313837656435396632363864333761623334346636616265393737 +32353735623061343438393638323262643431333161353364383938336164623732333732316365 +62613762393832656562626662373366363931633633376665313830386230616263393861383531 +34663561643730643463373863363932613930613662333139633834393333663162393539666539 +38346437373239313039646534666262616437353534646263353237656131306434366437663162 +39613638373937623063343336383064386633356365366638396530633261313264383139336634 +63363361373766396464663263396464633663643337306137343761643766356165626537326530 +39333633346465663734663230323933643432343066323537396362303138643336363866653066 +35316336376566633431363438306530633037663836383831313133616635643837646161376334 +34303763613237666634363934623332356431323432363237656539643734346366383731616435 +31353536656562353732643831366235336133616561353434343864383333633534313432353361 +33363433643233393731376232616338313861623637616635393939376664343365643562633732 +64326636373463616530636664623265316466313237363934306436393536616335366332656236 +36643234383435373766363235313466336363343366383733623866323836316230353234663935 +37393163623863613834626331373231353466343134306237633962626665326264656661343930 +63313162343461303363653039323734656432353333323662663365313232346162303039336461 +35316637383966326436346138386161626533633839646232336363653137386463316139316361 +34333165363037323830643731343539376633366163333063643039303166346336313765323566 +66646561353432356562343139333266643163313934383738333933366131323537373236363639 +66666235653063613365383937346564363761646464643432633165386161393064316437383631 +34356536396539623766383134343536393736636138613631326630353233333432313433326536 +34393365363162313264636533383664623466653230616466656264653331336131313330343932 +34633936623730313039636233333033663364303765316461613265333337633433336332643363 +30623830393134363138663237363061353636333639666634303566346231643732633338356365 +64623161343766386265326536376233343364363665323661393536363136353939656536366233 +66306234666264613865643937303533313731333538613066636662333036373732623863333762 +31343933343336363831393232393263613039663661386338353462333362313730336431343734 +63353465363435346166343861616435303435666633616532303164623939376432376461333836 +32663538396634373334613936636230336134633664653237383466633937316262356565636633 +63363231663864656634313336666231633736633338303738343832383036326338643337313166 +35643662663538353562653230363733373061333332396337663631343638333265636538636161 +39376164316561313537336436316362656538303838373939633062393931633366383533303164 +39613436343462616362633335366533313062356432326464376162323535623735353465373935 +37313764323735313838393731626265656533653033363761326630356663316631653264643130 +34643930646634626265663733393861333363336666326433373336646565663533356432626566 +62323637376261393339343761653436356632353561323139336637386464656331393335323635 +31366330636334393231346238366564646233656639346433393765663362613865346634663830 +64643835393566633137353061323333313362363263323132663639303933383162323637303262 +34366232653139656262386336646531376435393530663964343632613031373466316265353337 +39656564653263386462333534343639333566323361343931376133373362313237633833313833 +33666631343937303863323132386231366433396533333030636631613337643965376131333730 +34373531393133353237343662373932633634356562383365386238373533663239353833653463 +30343866316562353433393331383432353364333634666563653761626530663434616462373361 +30393433663937646533383738383335353532353162353561343163386238353532356339323332 +64306563316237383335323530356439663932646134343661633339356265393262363730353436 +38353234383439313230616537326634353564396463343465393766346437643834636264383462 +32333366313361626633366530353761333537353339396434363438383034666630373361386163 +63363764386461333036333563666339376134313638333864323061323637626265646363386636 +61386330613661636434356664636164623633396137353733333162353366616662396435363736 +65623461626237326635633534343361346633663530393762316133383062323433303165386631 +36626634663161313764636231363435653231646634623038656166636136333965343164653764 +63383465333731643062393330643963376434386332393166373266633833663866663236633163 +62613239663331343536336633396336626338373638613464363331343762353938613438633738 +31633362336631343033653662323666323965373164343763666465383439383162326634376130 +64333730383139616661666665373562666535303664653264383965316465633531313339646634 +32643262653530633634666166333930636364376438306533356665346431656336623238663662 +62636262396165323137353932353737346439613437613166623938643863613937653563316562 +32316336343564356534313431373366303633353163346238323738346337326630366263643338 +34326430666332306265386632666230343935653531666665663432313738346661643436636638 +34646434633035363932343764303135346536383636323566333163613133303934333237626339 +66383939343162623239393930346233626466663464383734363734626434323765313036353836 +30366466613863393663643833663332326164613536616635316131383531386462313931643266 +65326263666130616437373237323833333934623666346531656131633565383266653931316630 +39313033333334623035636163663531613837393236366336646536383939633632353838313333 +35383161353235393531323961356163666630373837663330336361643562343562386237306665 +36623230313063636661313062633431383130323464616466616264316236366333333662366131 +30373739653066326338353438366535316337646365613364613334666465376462373063653966 +63346534393131323337373932343233666161356435326433656638656663313038633561346437 +65306565316462353637356233636339613131633734666464663266643033356538653935306539 +36636230643633626662363436653231323266303663373861643930363261353236363534313764 +36613536656561373065376136383133343635633538373637643034376238393833653161353961 +62646564616436323939343261643538313764356565613133353464636330336533643331396536 +33353132363239623830613962623434333364373233646333333033393738306337653061373337 +33343937333863323937653163333337346630313934623533333538373331376631313437343034 +36306666653665383630653137653835653535616530616166616466383763336162353734316234 +34333630396639373636376131346435616666343238393037306632663435646666323164326633 +34303661343033373365646231343937343232336238663763666562396363323563623736636435 +38306461636238656164376437356237643165326266373366663563623234663961656330613534 +32613235363535356639386163303635306535373433666666353962306632373534643234613663 +32386630313663636132383235633336303134633431313839656336303734386163353838353939 +62353031613631663338636237366135656130306162656634613833623263633034383730393833 +66643937333230356635343731633833323030373832323731643635346138376261626564623966 +31356133376165333865303737363636613863363662373066643363343537386235623035666435 +38333539326164623165383030653162383336663463366530383332326366656166643833656438 +34633430643163366439306464313036653234373265386533303238353466653564313039363535 +31643831636366656137613136613138303861393334663939303665623034303437666534306337 +63303963393961666531396434643739373934653430636539313261333131303539643962643662 +39396566643539633139323363393534353133613865356436396433653639363565396635346164 +62653537373335316638336138313830353464366532373763396632326463383830323138623834 +32626232386638386333626566633534613935336638343464626563396335373965393738383363 +34326532396265346337643561663163363234626563666634326133326562343234646439646661 +35333533316532326230313236373137666137346233633935623130663833623037663034306530 +31343334646261343333363134316333303435613034646661313866303930653435646238616430 +62643537386431323266376365363834316631386233356135353136303037303434656137636632 +61646164336231313464666438633933323635643636313332336437623032633064386130303332 +33653135616664316133623430346139636637393061343266666532393964643563373034366666 +38663366656336376638333261363037663135623438393464396331663130643236303732323763 +30333839346431303263346239613733313632626436383434336563383038366461653866616637 +32316366643062633865653738323032636165353166643639306430393965333332303236396335 +32333935336235313961343662376532616430626164346363316530663065393032633030386563 +63313564353263663066323539653463373761633662366332353262393433393437613563346164 +33646232623566376534316238653031643962613866383764306333393536616534336333636437 +34636466363132323764623830653330346264313938386231383937303432363232393563313862 +34326339343333656262333333343030636339653137623863333963396165396437303637376239 +32373633336535346332646661346631383437366237373030353964616238663961323235363865 +39636139313238653735363865396138363335623530386535333735313433633034373862623935 +36616566633566653563643261383463623061396133353363303033333731306662383862636638 +36663336383334616662653937313934356333333365306364613335373234656364373438666532 +34613936306534386535356232666335626366313034666233363530613465653830383165366530 +62333830653565366162353431666662643163373063333236666238363833656630623031326265 +36363830646266353562303532373935646539613433366137356131656363396633623261626134 +63653930393934316534623464616466313139626436356438633931373732666538303138626434 +37323463666364613337376663303466366561643535396662663063343436646136356662663431 +37363733333234366331386561363737343662373261633766613162366338343639313262623030 +62613431636335386264663138623662643865623432343137616439323335396534623165373062 +34333336636362643063613136333532653662633863633635333037653231323332396230366132 +65663639666331633933613837323264376630613730646636626435346265373761326433343838 +38643364623236326461363235336662383236313163653736313161353961313734303339366535 +30626437346631646238396561313738623938313263393964393838663961343963633034326138 +36336361643733663234353762313331346333613731636462316232376464626139653862633539 +35373061363361633831363162306636303566336465656464386133623966626137373161633030 +64663361306537663431623533343739353865383335316462636565626462333035363862353066 +39623534653166366133613563343933326237663563303938393336386436326661663838313766 +34616237633230373832383133326235633066383366663164393932336332393066333561333133 +63326165666630316439336666393866646666303231623763623064303265373163386138656266 +38333665303664363862373337623737663937356235393765636230636235363465666131316461 +61303133393534333932363038396138323035633166316637613936623664646261393730663435 +30623333613161343334393237623132633433616132636565383339303961303962626264663135 +62353231373463663465323232643062356365643030656430663131323539333331326238663564 +34323237646637653338393762336165343661323936396234376430346363353632366664343761 +65616337396262643035643562346639383562386538363231346663353061323333313464313138 +61366630623163303566383562646231623661336130663939623435653131663233363139663561 +35383531663666346533326465633066393062613331653434376138633261613232613133643363 +62623064363635636330313732613664363635386235363333386131333866633365313638303834 +37333638356165633363653137343533636564623238366332666139333031316438656334663335 +65333430363632643636373862313861303763383463383461643865326535376339643936653136 +61633733323938623665623062353933323131343034363765393633623335313364653965656465 +38636165623336393033336336396164663032613365383232653537653863363538656164383630 +36313037346331383766316535383930333738396261343231323535613962333630376361306161 +35343730383438643430336433303064666132316235396262633430646462613438306462666537 +33363664613165373264363136393538303736343930376364623532343665383761616536643731 +35343337666138333461393838343539383962366563323235333139646463333231393263373833 +37303466636335646630383563326365663261386433323332656436376361653665333061616438 +33323131306235316264666562633139646465656135343036373364383036353337356664373935 +39353666663461626334376532343433623062653161646430363330343835303232613032386133 +32306661653161396238653364613938396463616530343337356431353563376132396539666362 +35623864353531376636616637363330623362353634346465346663346461323832633165343239 +64313362643635643837316431353961346165653934643065643738646436343061653837336637 +66316535623832346336636531353233323831633237663764386636356161376266616464343038 +32336533373366663931626436383836313561313365636336303962653336653565373463366434 +32323564343163373337643965333564383432653063656234313237636264653261383963376631 +37363631613738316631636666303038643331303931343530663565313733623530323235663335 +61626664653331653661626464316533643566333837303836626235343133343661346238343130 +30623266356332623739393030646564653434303239333939366432383037623965646333643963 +62343135636465663065373363656238643631303137663965386531306136636535653365313535 +39303965303135616336643035376433386330373736646663353539633235383434636130343065 +61393733623333363362396365646539336435323063383734303466363837313230336233333662 +37383564353766363261656434323566663537363335393265353464336264373764306135643332 +36653333396435376463316137396564656265376239313532623936613839333861366534623865 +32653961366561383037663131336564333534343037656264633864376338636331333239643762 +37373565303635626138346233666465376632386333623435633632313163646638663461353066 +30343332653261633765383463366137303736363565363135316231346364373266396533303830 +30383162656136626431313434306536313439653162306530336334363164303962333166616530 +66343537386432626465343630626634396338663262333330333432383063613333376433663662 +35643732656638643438346438623332613564373630643161393261643961383538643461653932 +35616164373166376266306565393332653963373339313036366466333235313930643830326135 +31653063643431343439383361363761336263616562623862643333623662386535393030383864 +61633163653963303462616430633737313539323736623864656437333761623733393439633135 +63653664373836396366626137363937333666333464656339636434633032396637303739643932 +36613336346464366237633661393630343365656237333033376333333534643361383630303066 +32393563396165363361643262366462613938346231363438366163346239356332393066623831 +65343334376239303264633365663565643863353537356164393333346536366562636164646166 +39343733366235343935643266386238623038326561613838656438316237663535656637373634 +30643466646232313636653136336438356135323163636161356234326564306239633263626436 +65643365653266653862646136653464343731633638636337663562643939313732636639646232 +32393563343364636131323432333463643237653337613961303135306238386564316335353362 +34323535313436393161303639366165346339616234396163303166393933383731373764343266 +35383335366263653862333462363137626161313266616664313531623037613965656637343262 +37653361646239323137373334633338356230303632316435363631313639396239656565633637 +32303434393961366466303135333765356364326232663232363936343034393663626462336664 +37376237663932323962343263626462346237623564346233666330376234303334363133623435 +64656533376162373239353766613136393662396165363763616463393035346363336361646336 +66616662633539356662396336316331636163653061633136326436376666626461646631363861 +39353537326334396162386439643865633534653163616231336664343062653034323638393161 +64303165376536376662336237313435356436653531346431346336383731626334626665323130 +62653431663839393236613066326562353566353561643838386662303634653239356164666433 +37623563653465363030666362646632643965613764663663383061333336336261356164333666 +63386430303763393265663433333763366266316135646331623961613431663032346662613366 +62653531646563646338313761656436303865613339373735316536623538616433366133616361 +62623239343763376530353234356139373739376434333864636238306535656361303162653438 +63376162373734636263306666613166383035366663636432366338323763373636333535613530 +36373631323961643138613232353233323838393364353035373030666231386239316262393935 +34356663656266316666396230663761633762616461363138356165653363613436663935636138 +30333439666630636565303335393334656533353662353430316334376365623533353166656163 +31623039623731376165623865313534383237663137353739313735643936376265336531623239 +65366639313265653433393964363832376261616538383838366664373130393331633335633633 +33333737626431336362323632336563616461353434383638316461333537376431373864336136 +33663963316435646636306630323831666437316161333934656633646632663131656532333263 +30663162353462306230323431633532643264343361386636363435356264633634346235333236 +33306338393230633263643932623830613139366531623864643639393863333064653037393631 +61656164373864313366363530663831393862643834303965333434626666363432343833643966 +61656139393433653365653231653763353934343637313261613736363565323665666335373634 +36613962623236623236636661333539323338626334653532383333646239666662336665313766 +38366461666634653734363034396163643632643436643031303764616537653937373466393965 +34316430313665383265326439303666383339316231303933303266393635333030633730623432 +61323962366137353231363634346138373032623137383864646530366133363435333163326662 +34353632636332613432396165623530366363623838613536626664616435313231636663336633 +65666338353732643830663034663236333635396337613263633466613136396638626338363631 +62303466343238633964366364343861303038393661336164323631613664643463333566653531 +66636137336535373434393935636333353030303966373562343739333835326162323163386366 +32343839353732636464326239323839646632366435323237626334613931366266636562313436 +35633163643131613136646235613262623065376134396562356561396162313437323031393861 +61646263316330373832636439623766663736653964373034333839366330316338633837393234 +64303038626363383130653032643530323838636230616534386262626537303039323466613564 +38366139323731636231666661373335323664653334323538636563396466393338346535323162 +33353435356264306538326361393534346332333032316566326334343238666435623030623431 +35333734616566323962303965646432613761333534303330363239306232356331656362626130 +33643366633739373737626236336662343631343465306338396532333965383865356565323835 +35613838613766346438313964393432303263613937313138633036636333383433356234366432 +62663932346139613133636335356331316562303733376334623330343438633936366235343930 +61616231313434383061353137363864656663653966376432343865386435646239356464633739 +61366236666330333563393365313534633465333630633364613736393530643133343831346331 +38366434656539326565393835333964343639323930623535353334353032316536303134333865 +32396536623831616166333232346464386330336638616164643739303231666232393263666663 +35643236393834353738646130353863303838383364306161303039343030333365393439383939 +32633035613532623262393530373038633832613665313165663932306562613938333437303164 +37623233386636303064653034363437303934623435396332643933653036623033666465623166 +34323136383038323234343033383537636361643234656363623439336337643736666565353932 +61633934633539353362373330643663323662666263323066623564663938343036363066663964 +66343562336135323038333363366236386666633030663139663566336632633861326565616663 +62363161353234653864646437613533373162353638396137323639666133356538333730303533 +64376563336266393035353662323566336239356134313538313730336635613832643930363337 +64613636363838376166386662626136616135323963306437343533323838383038343136396339 +32373466653032303066626463323137353939653236613739303035363738306637666535643438 +33373763373133313131646363383161373838663465373631653662393035336461653336653938 +61616239366161303062376262383661353331663763306635313865653838383236633532653464 +32663233656239356163656165326136373036383838386565313263633265373065396339623961 +65386233306333653637383138313331316665643139663666326333633138633331663131356331 +36333836393665303763386163363563346464613066313730363332363161313231306334333330 +32323137313639323165376464306432616330646332353161663164336661373261646139613330 +62643232663238343961666339386632326131363039633963636638323331653833323262626136 +36366431383431346538346264353731373765303664373766633036333733386363376361396531 +37666664343132666564396438343932663033393532626134306230343030326238613936363230 +33643664643066316534316566653732346566666133353730633738373539653466633664363439 +31636630323961356266373530616432353836393661346437633964343763663239303231346337 +64383331656431353730353630346665383061313462646365373463626464306236336263363862 +65613533633634643137653934666666373833633465323562306536313238336237653539393932 +33386636646235353132643933623163323232356431343962306361343432636433653836636663 +33336434643431306535306139303333656533646437613031396564383036383263376239643532 +64373335373565333935323262633662613064643361636462623831616263666561383661616164 +62613061343431373539343633636632326338633131643638623631313462303364613437353830 +64656430666530373738386663343335303361303333636661663334656330336462663131346263 +35333435393639356230313764333437663135633834336534626665306565366266646437613336 +37356461636532333162376337323166663430636634353430376537326663633636323761353865 +65363134326631303862666264363434326435396539613766333639663733383933636630623563 +65663032666437386338613532643064346262353530346531613936323932626636656661393931 +31616539336166656238393162353065383463343734373732613863353563643731356532356631 +35373034383166653637376131396335633434336230396130343165633137393537643861303437 +64623866313435363864393736666331613339373330323437383836656234306431623566353136 +37663439303331353131313635303366633232373135336161343064393237623836613566353630 +37633339653235303231663034343966316365653230666562393464356630366365306236373235 +38323638366635333030656337623837643861383734376339343864643434613335366136633861 +37646237346339656539666631373466623632653764313637386238346236633865623938336166 +61666438333638636466666232366435363732323866643364633732373331306661393431336663 +36623839666135363230346336613065306365663866623132356165633431373539306534646531 +38373061383462646630323536653839383537613363393839376435386131653038376666656563 +33663133643638393265653435353436366364386632303162356630616465613333306565663661 +62343664356631613066396130366665353963316332343736633332376665373932313466356437 +39333439383036393861373366663334626131313230326134333264633737333666326330313936 +33376238613565313136393966316663316361303033383838313534663534313132336538613163 +65636339616234626431326364623231656231343031333936376364386638623233303836356130 +31383463613864613634336536643539363962626435363736393534336633313431306338316330 +37626235636564306632376634303638636231323830353134363031386462396335363762353166 +33623234626531343630356666306338393863633665663537646539316334626263626665306565 +32363431643261386561346533353834366535653139346139373733396538306264663264613666 +33386164613866373632643332366338393134306363303762363834306666653966616331623032 +65306633333564363766343165653764356633313834616633306134373933336162623935393732 +31303661373434653638363162373730323765353666643132653732323831323633356337646136 +34306537666261646238393436353839643534616536373537646564646262333764383566653230 +64613362346237393336613133343831306138393638353537303936363062343433306361653934 +65373363363634306166616338343337303533323262376533316664306133383663633030636238 +62656230626234666466663963353039626163346366613166633836613766313766383030383634 +30356533313536646631313531633436396331393463336461623037303439336162666264333931 +31343336306233346565333763623330393538373563343131343934643033626564343030633837 +37656432643832343662343836346631333766303665326165643937396430313661633261346535 +33623639633032643930363561613531383964336339356463366631343964636365393564666265 +31356232633666393839393766663535383062656338633731353035383566396532323335386661 +61373861333636393432633237376533633865623139663135666565613734653064616436333864 +37306631633934306266336562326164623733653834616434393939333266336666613133396130 +35323835646232666536396130303135653636326332336330363365336139656363643633326132 +35326139393332323366386163633137366634346262633531383732376433343132653739626664 +37663231333362643135353435313830333937616331303032323231303766323166633262303932 +32393736353131666332633139636664663837336639633262393232393936353838653862373764 +66613039613835323736373133356535396561393461656361346434626566643936653133666661 +61653437336537643034646131613437646163356461623963626531303433383762303534303964 +39316262353135353630306539376139366364306535343534326262643163323332616664646666 +33346136333664656233643031373833303836306162666161656134383636323063303363393961 +33316234653933323434343362616536323835303630666230323265363634623963356661646434 +63356633653730353933353234613034626464616235303135363962303663653138616434316237 +31333565613938336532663235646532663039373363323964313962373663353239303938363061 +38316133633239316137343237623939633463376438323236323535303532653762303261633337 +63323431393364333635626265323966326435653733323031353538356564396237633834306338 +39366564646332666263653364623132623034643862363439306665646337313262663236616133 +62323162613537313463653866626631376432636661323337396236306564393034323465643839 +38353766396564653335646365653362303463326531646465666262643866356433326262343461 +39383635633834313561363366376230346666656465383462313762666632646536373338613732 +31646532323337336530356164616264363464363733663536623861666665336363313639663334 +39363734643035393764666330393861643663633564643735343661336233383237396162646365 +61656539313961623539613961663263323062666532643062383930353439313336366532363935 +33363137383661373436313736316465623834613832623032303036633731303138336138383164 +66393965353136626461373538356465326264653732633830636261626162626434303264313532 +31623739383831396462646238316137366563386466326566346562343634333739363031376632 +65346662353039323264313535316235663839326334643535646430396666333238623433373335 +30323463326562313761633631383639636434653939613266613238373532326638353433626138 +33633234623431316638326661303031393734316166363339356430303330326162633036353664 +64383463393532373237343636313536393965396164313962383434306465306466306136303864 +32336633653933633633326539326365323161333064336231643838343030343561396231353461 +64333534656234343765356436656163323665353331666563313063623534323037326562633730 +39383538663666373665353234393038336165376233643832666536393332343139376330663364 +34343832656235636230666135386431613337393635373066393031623761303365363461613335 +63633532303565386166393238336334663039653035373364623938376265613834323635343061 +61636533313862613334623833613834356662303862313232346338353637373330356534663161 +31383035303962346638646230616166313239613866636234346537303535343934613336343961 +64653964396562373132396662646430396165346536316136623733666631343132636233663530 +35333762656537623032363965616539626164343339313832323266356330626230623533313338 +66316161386332666433656135613634326261333966613239366637646131653565373162353664 +35373735373431323339613163313635343663666665396262353535616230336437653566333938 +34376534623631373630396664623134393161376664303332636635613435656331333566613530 +30626566376365663835333066343933376237363833653939353336653434316431303434623736 +63383932636132663636353763643938656662386237646530356164343131616465623538633763 +36333466623562663261353966623033623932636338313965343930623463666139393334666535 +30666630393965653232613838663263383062313366626135623333666132396666363766326666 +33623739313463656632363234376166313333393964623365653935353631636533643136326563 +61623339653935376434386432386437663061333262393364353530373630666531366431393536 +38653733616463353032353962343135373232616635333966613030613932333865666334656539 +31333333623431366432633334616630613462326638623532396336303236646438343761353237 +63326431663733653834656364626163313336386539386330653530646634336339313463656662 +31363031306535633234366535373431306333366163316665646662663265386361366536613538 +36383633643931633038346632663761333235323132663563326230653262373465376463633537 +64643863323035313865666537326432343435663536666234623462363935396266346235303536 +65653136643266356364396162616264303861653763333461643839646466316663636534633332 +34393437373334613636646230336634353636666263363636303833663835666134393038326261 +30613032363432353333313731353863613332643563363830333637646437363434666665396431 +61333231383063663266396436353331353332333136646163336237306566333164663439333834 +65656237303132643639303661303664376464656639333164316338383531333231333434363862 +37613337653362633931636162633637316232333635323839636165396466373432386332316361 +65363332616161613335363862376266626166393233386230326566373866393335313735633262 +33393738633137666635363861316639336161636165363161313432653037633762626666336236 +37633766336533336363383534626365333936316330613763643966636436356561643438663435 +63313938636439666163366333636432613539326233653630626535363666333239666265323337 +36366633373266333162306432343034326331316165313033303639333861646466323364633165 +32346565343436623631636132366166613235343739303761386631613465393938336432356233 +63313164373037636464396138663636363237323331623463316638633335656334323936663339 +38373335353462306537343236646135363464633031616334383164313931323466373865303433 +65636533613533656337633861363531326165306137633963666230353739633033373039386530 +32326561313166623736326432656462343762323134343964343533323038333038306634663237 +65373038316636363434303633626434376230353130336466303735353863643162346662663839 +66613532636234336361323037623231386566323136396562316661333537633362393062613962 +66653938656536643061636363393737663166623865313339303339373064643832346163353665 +62343465343335393931303161373032666266636335653832333831633265383635653764663662 +38393364626332303762653964363337366134613530633730353032663234323864643432353432 +35653932323332316264613230353637353435663663613862616563353133386363356463373933 +65656261313837363339656331363231376630316162326130376536373262643864363164376364 +39386130366531333161333266313237376538396334386337386332386463386531393930323065 +39366138346662663831326234363237656165393163303162396236653035323436346565353938 +64383162356531626535663065623763636265353236386464613337363937636637343263366561 +35313038313166623762653164663565643862623530623732306130626437623931376364666431 +63363130663962323834313730396261623138363837353837346337333064666332363639356166 +63326439376133316439316635346165663933383739313633356334623035396663336263613337 +39303233346335616233633235363531653364336336666164656133623063346336366465323461 +61326230333934363461323265666430613833663238623735646432626563356630633132663037 +33383230363135613439303862316264616636353630646163333030613837376235636433373030 +36366665313364363165633562616437373931366366633936636561616237626662653731636639 +36653738383333636562313131336335353632653633653436343731346532306537643364356438 +62656332373062356339363032643730303936663432663537323732356565393865663961333134 +65646233666538326635373563393838663462336437366364303765303764336535356534313333 +62643961643266613766363336383933333537323666346638323065306338333433643339393833 +66323035313930396231643632383636326233356137653134633864326366343666613838646662 +32343961366661623232633930346235353033303464316561646264653737336330353737353837 +61306639643433396264616132326230656132306530666637396432346465343762393461383161 +34373466386661333962363866376335303331376561333037313262633638656362343861303630 +37393864323534616339613639353630333363656438303437656664366266363037643361316530 +39653336383062333061396565323965353531396532323064396234316336363338353237356565 +36636136336632336664396234353933663932356363393234633735653935313763306634616433 +61393436343035626263356438396338623131336564633431383739376133613933636134363839 +63343736653434633330613161353462636238643934396438633737366461356133333232653534 +35653765326663373133646136643234353330366636613638613066323438393561643738653336 +39623561336566313863333438366438396230653830376230343465356664373862303433633665 +35343935666631393637633961356461666432393866643336323239323262356630636438363535 +39316662376165323437613365666566326339616561613162303839633864363163306136396333 +34393066383166316530633934383039356437626336643364343239383236376661653838366632 +37613933356464383031376662383435633836643665356566363364396165656332656134393662 +37303339656633333538303736393165663131393030326332636339313235623132313461343939 +63303265613335656633303536346166643235623039666562616135653834343237393833326361 +31646631393336333764316434356665363831366538363063303537636437636336376335663032 +32363938333763356662363433356465646365376562643737663538656565393236363032663461 +34366464353334323731323162666463626639643964666535366439653534343332656630316263 +62333663336439326533396233303035343637303439386634303931396532633335313863356337 +61346633313966623031303530613032326462326634653331323561303237663061663735623466 +62383835303163323762313231643865373430316338316230666232323437326565343334633933 +36323331633964376139326135336366636131646438363061376635613038316138343164376163 +39646335366466323333396530396363333330363462396238653735353230646134353330393938 +31316133313434323931316661366239646239653463393762303532383338623431363661353132 +38636335383331363037616139383935313835616461333065376133643965393733393539393365 +62373862663439633133653636616534323437373563666163666539616263393635353437613434 +37393137323662336163616130326634613433623964373061373265313166353461306365373363 +66376238343934623837313761613164653735373038323462353335643833333130396132613335 +37366366653261353161356637353036653562336161356333383761636534373432633065626561 +63656238663165333334373233363630333039393661666337313135616164396261346437343435 +34653463333331656434326535363561306461393634616461633130303264366361666233643130 +32386633303839313330653961646537663934316631626666303964353035353632613239353863 +37306537376463353937353164396562396437353237306231306531636332626630383939643262 +34303634343965336664633936643933333265653232643136313637616362306636393131336432 +33303438336531323861636433643136343061393632616165393465316165356665633438336338 +38363537353432363361396332646564323861323334616636323331323031306233623031333832 +65333338356135633866313031656431616637633133383061633462333138366665623364313165 +38373536323062313134353432636662316234346232366639393332663432373265666264623138 +32373762353437326338636235336237663230663534376162306363343632383139663266386232 +62626338386366666265303038373664393630306265303838313561376637303964623030316539 +37633464663939626437663633653531623334373631633163646239666235346132396239376238 +39346165393934353462353937646664313137336439306531336537323536626438333531386631 +62373866633664363561343636653538636466363265353966643138366238303335376439613063 +32333536653861383161316466383762376239393835353063313636626563643361363936653631 +39613131386334326439356538383337346462303964316462383535386635306462353731343237 +63623934346534656566313162336435336134663130663238396361623865383365363239623338 +33383534316165306161646563326337643062663434306333376133633138353636313634343730 +66346637633265616537373366326166333265336464613638373634386134626235303937666634 +33373461373262656162653639376633346265306631653638623730363764623137633833336631 +30656466626538376165633039636465376137653861396138616336653736373362383931323934 +31363738383361333432343738643735313065633430326661623434326462366631363164333930 +39306534356435376262616335633032383361303864323933636266376632386434653533333261 +61643632316533386266623731656334376631343165353032366237646630383033396339663666 +64616337326135643763613736393237383935363931303239393538393530313435333734393031 +64366639393239376166646133303932653566323331313765373730393834333933666133396137 +34616266626336313362663464316439353062376663363130643466613837656236346366386430 +65303564653961323235353364313963316230383135316331386532303332373933333264303265 +33616562626638333430316535346661393632306130643366656332386466663834353830383964 +32373561633265633332353464383564643636396531623066613631653130343830346432376461 +37336637613434626239633538396336666365616437623363363662373831366139626134616530 +33613665306230323535646365306164656436326466643337663431656261376366616434373565 +31383063333166323165316131313831623137653663343637306464663335666634623439356139 +64396439633233613039373239343034356430646330633663616635306332663030383230386334 +63376561613265353264346463636634333764356435653930363236663537373939613533376135 +39383030386164336234396434646636663837303266323631643966383562633832316632363363 +34316561653564616434396531336364376564386538303135653237303735653264353062613337 +35353665316633336165353139303138626261343237363734303662303037363033356531366433 +62616435326664633734326463383464383061383266633432386633326337356431326634366366 +66653465313532323761323935316630346438316561643133363639313565653639623366363630 +31333636633030363663316632646338626230616334383537313431313834336536393031623337 +32343964376564343234396237623433323436346239326133373162633661336238306438313161 +38386661303663393633656164613931333666636166643034646138353539333536326565323035 +32356331313931643635323466633365353332306433623530383031666565373831393136303334 +34346665316666326330323335653336376266343337343035663437626438663138383637313938 +64353265393030303433626538323933343661643666356362376662373765316635303164306132 +34303836626639346138646138383931323330623032326163396530376433313738353364346164 +33616332333830646166643931623831386165313463663130343039373831326162316130623666 +33613264316363626563623266386638653065643464313739626664366533303666616664353735 +34333566616262313730333935643637323634663038643937303264366137383231613531313833 +36666362656530623635346361326131343630653730626464393531306339343334653631323664 +65353464623234663363336432343863366237326336313861396466333030613131346462613633 +35383035336336623664366262363765666564386135396434383563333636616566653164353966 +64633537393364313939623933313765323433373936323137376634626261643331643861373161 +36363562333063383638616563656336306631343463333965306461323034643931306432343264 +32303062373063323761653136653530323732393463626339393034303766636161336666616634 +61656138316164366466316333643835326132313864663732316437663865303962656137336261 +39623334666363316237313232316433623566356564636135353736633736656461633934306538 +66643138323138356231336631356538626364373765353263623034353564386563363866663062 +62633138386531623434313464353262393762646365623761663438623833333063616361386166 +64636561343162323966346165313639386436326266353963636437363362343635666161613163 +65366565373264366165353333666264623035326533363935626439626334663835396561356264 +33303066323166393132656335346266386432396663663937346432373536303034623163653962 +62386339316236333365643464356132653734343132323366383131353963626533363462383061 +36636631363234303535326436613631646263626135303363356235616231663863646466646435 +38393039376237613433363461663634393236653232643833363666626663623639643138346661 +38613763626664653066633764356134643961636135373236653639396134333432633038653138 +64653966306231323836353433663964383436373233366233343764346434356532383464373066 +62613661666133383337313461386531313265646136313531343637663231336432663434626338 +38636563333834363364653362623061393032336531326438343830623862396336383662363366 +61633461303031326261323933303834636665303061373033376635343937326465353230656664 +39393362636161353339653635303532383365663737383937363761363765333738343133393535 +32653365393031626562633265363736316238396436323131643365343936356438656438306639 +34316438636239613363376230396662336534393464353264373036376230303762376439623336 +34643761613366653866356535376230336335663461383562623561393232383661393839343833 +30623962613336366336626336323537633861316230323965376130316238303662353465303632 +30356439386265323331366662383833636662353036303633646134333934623236376235653865 +31663138353638663038376363646231636537306261316561326165323937656362653831646539 +61386165623430643462363438613764383139613733353338636236303464313131313063306637 +38623237656136646333643461623563303632333963306539363063666564316261623536373961 +61356532643033303863386262393764653139643933643064376462363765373039613862636631 +34663632373238636632383236663131633033323662626530323166646139353436656632313338 +38333337313764383235626265656237616630633139366633663333613866313030656366313231 +62633165613333383335633236383864313731333631663338626339356465663763313836666165 +39393763643638333038313063303434383665376161383665656236373665333864626338643263 +61643931356430363162643265396666316538333430303933316537626130396262366662306263 +64663732363365643133353535366638333263343065366332356230623639616434393031383930 +37393861396633666335336432343762313331306136663162383165646166323833323262623839 +31323036623338396636396461303838333137393531306639653832653935663534663265316230 +64636138393464636336373665326230653039393433326232396462623434613335386665306565 +33376361386339663536653237306361666130666266366539313166323439323030383065613331 +34396633623838653033623132303163383864623963633034623364663062373233323461656237 +32313863643033633435376365633465646263333137636332626531623930363033396437353965 +62303536623764303338376533623936636631336638623332653761373164666630386331313032 +64323561363961366135383564333663653263656638303461633036306364653432646166396430 +34393639343130623533663435616165663132303064643763313133343037313836366231353834 +64633834343866396236663935613536393937616235633936633432616231373161303934353536 +66313434323064646137626639353739333836633833666630366530383438663330306565316661 +66636566333130343566336634636531646338376463323339303138326637343734613734336366 +62376639623734643133363534366462646333353863613137623738316438336234313238626236 +63353335643964636362383239383664396139386632313434633766626237353762353039303363 +36316362383964303637326166393266383266343132373763653631343031643661313064646536 +39346261323362366463303633356135336130343437383031373066373931323734613338653739 +39323230346238663262616564363936646163386538323232613036333934323661343961633936 +36623335643733666164663837366362333435326165633835636161663435393235666563633263 +37366565393263386538643534666630313031393838363366633736386162353464336438653862 +38343036376432633462333764393461353665313739633261396634323662323638366563386433 +63643737643463393532663837333034626539313737653132363438306137353432623236626537 +66323432346239326438313131396135383066393236316363646237373535346263653165346361 +30633163663838356237616637616630373539303764303531623836356131663536643734343430 +65353833333538363863383364333661323164303963613762653466366365633038363761636630 +37366464623930633861643639353162636334373533663439313037323737303031333435343635 +34636432393562313530373065636639383035303539626364646461376461613162623165643231 +33343439373834376164363530356165613037653339643861316266373535356635383030653463 +34623937313864613434653835326336623662643030326162353830356238643430613730313839 +66336232316536343733396533396432333839383334366234313130333733363134346434346365 +63313634313632346262616336353662373238336637373439373833623536396535343431306338 +37343032356439643061363234666263653238336563396334616135326534336333366631616265 +31646232363034613930633965653335633631343330303539663435363939336434386137343138 +62656665343864356561663731643937623665303733353333633634323635323463376266626134 +38623738646663336566316631643039313534313163396363303661326539336366303236383231 +62393361623866343635383665313733323632633866323661376464656366636530373032643466 +64363462653432376332656637316361393139633834303962336564383137326466356630313534 +34666133353237326162633231633162353236313933383864343464373538313964663333303736 +39663930363561393864636438393637323062343236626163613937383662633562353037353864 +32656231643664366130653637333964653962363039393339356465356263396134613731323531 +61333731666666393339613164636366396262653662316165376161326437613537373937353237 +63333234383434393437346236346133666330383962356135386265363233333163636130663466 +65353230313630646232343362643434636166356537393632303133643561626262373435343030 +62343837343263386230643334376432653035343066343131313836633532336432343139303335 +62396238663936616639373564373533353132323439393330326362373466313636373262323935 +64633463333966633031643465636237356638333739393437323235336136653135623366396231 +30616464653663353733666135633763623666333366303336393530653136343430326563306265 +34626366613833376562353365626562343133303166373365653231393863393837373461613333 +66353963373235373135353963616363323830613430623164353466653832663532343261383939 +30616661383764346139666130306439353065393231383065643932666161393438626338633566 +62636331333063636565303464616537393264333165626330643864636236306136663466643638 +39663966333362626564376264633432336330386637323463626132613237653433626139373030 +38393134306264326333326332363862353733356264666361303462303631663631383865623431 +36626137666139366164643337323836346436396332636230663265393339313264656362663263 +38333166313334323065643834646432626664623265643532316339636265623734363937383731 +64623435333936346536316161353531313063643733303635363031326566383733636636333561 +32323538646463346662613832636136653664333531363233353964656563306664303862383466 +30613738626636656363323665623261353635613066623037303036383230393837626335353133 +39613163343265306565343939386433383361336539316339373966616130643462333535656332 +65376536356230386531383939663364373334363938363062626631356536396635346534356334 +65396531383533613365376564346331383835646464313735373562366661633231303233636638 +31363230336533383435663335383565333632343463383038643735613863396337656230653930 +3438 diff --git a/infra/ansible/inventories/qa/group_vars/keycloak.yml b/infra/ansible/inventories/qa/group_vars/keycloak.yml new file mode 100644 index 00000000..9d9817c6 --- /dev/null +++ b/infra/ansible/inventories/qa/group_vars/keycloak.yml @@ -0,0 +1,8 @@ +--- +keycloak_db_host: "{{ vault_db_host }}" +keycloak_db_port: 5432 +keycloak_db_dbname: "{{ vault_keycloak_db_name }}" +keycloak_db_user: "{{ vault_keycloak_db_user }}" +keycloak_db_passwd: "{{ vault_keycloak_db_passwd }}" +keycloak_bootstrap_admin_user: "{{ vault_keycloak_bootstrap_admin_user }}" +keycloak_bootstrap_admin_passwd: "{{ vault_keycloak_bootstrap_admin_passwd }}" diff --git a/infra/ansible/inventories/qa/group_vars/lakekeeper.yml b/infra/ansible/inventories/qa/group_vars/lakekeeper.yml new file mode 100644 index 00000000..1c37993e --- /dev/null +++ b/infra/ansible/inventories/qa/group_vars/lakekeeper.yml @@ -0,0 +1,7 @@ +--- +lakekeeper_metadb_host: "{{ vault_db_host }}" +lakekeeper_metadb_name: "{{ vault_lakekeeper_metadb_name }}" +lakekeeper_metadb_user: "{{ vault_lakekeeper_metadb_user }}" +lakekeeper_metadb_passwd: "{{ vault_lakekeeper_metadb_passwd }}" +lakekeeper_metadb_encryption_key: "{{ vault_lakekeeper_metadb_encryption_key }}" +lakekeeper_bootstrap_credentials: "ansible:{{ vault_keycloak_client_secrets['ansible'] }}" diff --git a/infra/ansible/inventories/qa/group_vars/superset_farm.yml b/infra/ansible/inventories/qa/group_vars/superset_farm.yml new file mode 100644 index 00000000..94210917 --- /dev/null +++ b/infra/ansible/inventories/qa/group_vars/superset_farm.yml @@ -0,0 +1,6 @@ +--- +superset_farm_metadb_host: "{{ vault_db_host }}" +superset_farm_metadb_name: "{{ vault_superset_farm_metadb_name }}" +superset_farm_metadb_port: 5432 +superset_farm_metadb_user: "{{ vault_superset_farm_metadb_user }}" +superset_farm_metadb_passwd: "{{ vault_superset_farm_metadb_passwd }}" diff --git a/infra/ansible/playbooks/datasources/group_vars/datasources.yml b/infra/ansible/playbooks/datasources/group_vars/datasources.yml new file mode 100644 index 00000000..c9ffd066 --- /dev/null +++ b/infra/ansible/playbooks/datasources/group_vars/datasources.yml @@ -0,0 +1,4 @@ +--- +datasources_influxdb_port: 8086 +datasources_influxdb_container_name: influxdb +datasources_influxdb_data_dir: "{{ cephfs_mount_mount_path }}/staging/influx/influxdbv2/data" diff --git a/infra/ansible-docker/playbooks/datasources/influxdb.yml b/infra/ansible/playbooks/datasources/influxdb.yml similarity index 100% rename from infra/ansible-docker/playbooks/datasources/influxdb.yml rename to infra/ansible/playbooks/datasources/influxdb.yml diff --git a/infra/ansible-docker/playbooks/datasources/prerequisites.yml b/infra/ansible/playbooks/datasources/prerequisites.yml similarity index 100% rename from infra/ansible-docker/playbooks/datasources/prerequisites.yml rename to infra/ansible/playbooks/datasources/prerequisites.yml diff --git a/infra/ansible-docker/playbooks/system/docker_image_ls.yml b/infra/ansible/playbooks/system/docker_image_ls.yml similarity index 100% rename from infra/ansible-docker/playbooks/system/docker_image_ls.yml rename to infra/ansible/playbooks/system/docker_image_ls.yml diff --git a/infra/ansible-docker/playbooks/system/docker_prune.yml b/infra/ansible/playbooks/system/docker_prune.yml similarity index 100% rename from infra/ansible-docker/playbooks/system/docker_prune.yml rename to infra/ansible/playbooks/system/docker_prune.yml diff --git a/infra/ansible-docker/requirements.in b/infra/ansible/requirements.in similarity index 100% rename from infra/ansible-docker/requirements.in rename to infra/ansible/requirements.in diff --git a/infra/ansible-docker/requirements.txt b/infra/ansible/requirements.txt similarity index 100% rename from infra/ansible-docker/requirements.txt rename to infra/ansible/requirements.txt diff --git a/infra/ansible/roles/base/defaults/main.yml b/infra/ansible/roles/base/defaults/main.yml new file mode 100644 index 00000000..f4d47ec9 --- /dev/null +++ b/infra/ansible/roles/base/defaults/main.yml @@ -0,0 +1,3 @@ +--- +base_timezone: "Europe/London" +base_package_update: true diff --git a/infra/ansible/roles/base/tasks/main.yml b/infra/ansible/roles/base/tasks/main.yml new file mode 100644 index 00000000..59016846 --- /dev/null +++ b/infra/ansible/roles/base/tasks/main.yml @@ -0,0 +1,46 @@ +--- +- name: Check if APT keys exist + ansible.builtin.stat: + path: "/etc/apt/trusted.gpg.d/{{ item.filename }}" + loop: "{{ base_additional_apt_keys }}" + register: file_stats + +- name: Fetch missing apt GPG keys + become: true + ansible.builtin.get_url: + url: "{{ file_stat.item.url }}" + dest: "/etc/apt/trusted.gpg.d/{{ file_stat.item.filename }}" + mode: 644 + when: not file_stat.stat.exists + loop: "{{ file_stats.results }}" + loop_control: + loop_var: file_stat + +- name: Ensure timezones are available + become: true + ansible.builtin.apt: + update_cache: true + pkg: + - tzdata + state: present + +- name: Set timezone + become: true + community.general.timezone: + name: "{{ base_timezone }}" + +- name: Apply package updates + include_role: + name: packages_update + +- name: Install Python packages + ansible.builtin.include_role: + name: geerlingguy.pip + apply: + become: true + +- name: Install Docker + ansible.builtin.include_role: + name: geerlingguy.docker + apply: + become: true diff --git a/infra/ansible-docker/roles/cephfs_mount/defaults/main.yml b/infra/ansible/roles/cephfs_mount/defaults/main.yml similarity index 100% rename from infra/ansible-docker/roles/cephfs_mount/defaults/main.yml rename to infra/ansible/roles/cephfs_mount/defaults/main.yml diff --git a/infra/ansible-docker/roles/cephfs_mount/tasks/Debian.yml b/infra/ansible/roles/cephfs_mount/tasks/Debian.yml similarity index 100% rename from infra/ansible-docker/roles/cephfs_mount/tasks/Debian.yml rename to infra/ansible/roles/cephfs_mount/tasks/Debian.yml diff --git a/infra/ansible-docker/roles/cephfs_mount/tasks/main.yml b/infra/ansible/roles/cephfs_mount/tasks/main.yml similarity index 100% rename from infra/ansible-docker/roles/cephfs_mount/tasks/main.yml rename to infra/ansible/roles/cephfs_mount/tasks/main.yml diff --git a/infra/ansible-docker/roles/docs/defaults/main.yml b/infra/ansible/roles/docs/defaults/main.yml similarity index 100% rename from infra/ansible-docker/roles/docs/defaults/main.yml rename to infra/ansible/roles/docs/defaults/main.yml diff --git a/infra/ansible-docker/roles/docs/tasks/main.yml b/infra/ansible/roles/docs/tasks/main.yml similarity index 100% rename from infra/ansible-docker/roles/docs/tasks/main.yml rename to infra/ansible/roles/docs/tasks/main.yml diff --git a/infra/ansible/roles/elt/defaults/main.yml b/infra/ansible/roles/elt/defaults/main.yml new file mode 100644 index 00000000..cb6c7cac --- /dev/null +++ b/infra/ansible/roles/elt/defaults/main.yml @@ -0,0 +1,4 @@ +--- +elt_secrets_root_dir: /secrets/warehouses +elt_uv_version: "0.9.29" +elt_uv_python_version: "3.13" diff --git a/infra/ansible-docker/playbooks/elt/elt_cron_tasks.yml b/infra/ansible/roles/elt/tasks/elt_cron_tasks.yml similarity index 100% rename from infra/ansible-docker/playbooks/elt/elt_cron_tasks.yml rename to infra/ansible/roles/elt/tasks/elt_cron_tasks.yml diff --git a/infra/ansible/roles/elt/tasks/main.yml b/infra/ansible/roles/elt/tasks/main.yml new file mode 100644 index 00000000..56c67275 --- /dev/null +++ b/infra/ansible/roles/elt/tasks/main.yml @@ -0,0 +1,154 @@ +--- +# ---------- UV & Python ---------- +- name: Check if uv exists + stat: + path: /usr/local/bin/uv + register: stat_uv + +- name: Fetch uv installer + ansible.builtin.get_url: + url: "https://astral.sh/uv/{{ elt_uv_version }}/install.sh" + dest: /tmp/uv-install.sh + when: not stat_uv.stat.exists + +- name: Execute uv installer + become: true + shell: env UV_UNMANAGED_INSTALL=/usr/local/bin sh /tmp/uv-install.sh + when: not stat_uv.stat.exists + +- name: Remove uv installer script + file: + path: /tmp/uv-install.sh + state: absent + when: not stat_uv.stat.exists + +- name: Find compatible Python version + ansible.builtin.command: uv python find --quiet "{{ elt_uv_python_version }}" + ignore_errors: true + register: uv_python_find + +- name: Ensure Python is installed + ansible.builtin.command: uv python install "{{ elt_uv_python_version }}" + when: uv_python_find.rc != 0 + +# ---------- AWS S3 ---------- +- name: Ensure S3 configuration directory exists + ansible.builtin.file: + path: "{{ ansible_env['HOME'] }}/.aws" + state: directory + mode: "u=rwx,g=rx,o=rx" + +- name: Ensure S3 config file is absent + ansible.builtin.file: + path: "{{ ansible_env['HOME'] }}/.aws/config" + state: absent + +- name: Ensure AWS configuration is present + ansible.builtin.copy: + dest: "{{ ansible_env['HOME'] }}/.aws/credentials" + content: |+ + [default] + endpoint = {{ s3_endpoint }} + request_checksum_calculation = when_required + region = local-01 + aws_access_key_id = {{ s3_access_key_id }} + aws_secret_access_key = {{ s3_access_secret }} + mode: "u=rw,o=r,g=r" + +# ---------- dlt ---------- +- name: Ensure dlt configuration directory exists + ansible.builtin.file: + path: "{{ ansible_env['HOME'] }}/.dlt" + state: directory + mode: "u=rwx,g=rx,o=rx" + +- name: Generate .dlt/config.toml + no_log: "{{ not (elt_enable_logs | default(False)) }}" + ansible.builtin.template: + src: dlt/config.toml.j2 + dest: "{{ ansible_env['HOME'] }}/.dlt/config.toml" + mode: "u=rwx,g=r,o=r" + +# ---------- secrets ---------- +- name: Ensure secrets directories exist + become: true + no_log: "{{ not (elt_enable_logs | default(False)) }}" + ansible.builtin.file: + path: "{{ elt_secrets_root_dir }}/{{ warehouse.name }}" + state: directory + mode: "u=rwx,g=,o=" + owner: "{{ ansible_env['USER'] }}" + group: "{{ ansible_env['USER'] }}" + loop: "{{ elt_warehouses }}" + loop_control: + loop_var: warehouse + label: warehouse.name + +- name: Generate secrets + become: true + no_log: "{{ not (elt_enable_logs | default(False)) }}" + ansible.builtin.template: + src: ./secrets/envvars.j2 + dest: "{{ elt_secrets_root_dir }}/{{ warehouse.name }}/envvars" + mode: "u=r,g=,o=" + owner: "{{ ansible_env['USER'] }}" + group: "{{ ansible_env['USER'] }}" + + loop: "{{ elt_warehouses }}" + loop_control: + loop_var: warehouse + label: "{{ warehouse.name }}" + +# ---------- ELT jobs ---------- +- name: Set ELT job variables + ansible.builtin.set_fact: + elt_git_clone_dir: "{{ ansible_env['HOME'] }}/var/git" + elt_cron_root_dir: "{{ ansible_env['HOME'] }}/var/cron" + elt_cron_logs_root_dir: "{{ ansible_env['HOME'] }}/var/cron/logs" + +- name: Ensure cron scripts directory exists + ansible.builtin.file: + path: "{{ elt_cron_root_dir }}" + state: directory + mode: "u=rwx,g=rx,o=rx" + +- name: Create/update ELT cron script template + ansible.builtin.template: + src: ./cron/elt_task.sh.j2 + dest: "{{ elt_cron_root_dir }}/elt_task.sh" + mode: "u=rwx,g=rx,o=rx" + +- name: Ensure cron logs directory exists + ansible.builtin.file: + path: "{{ elt_cron_logs_root_dir }}/{{ warehouse.name }}" + state: directory + mode: "u=rwx,g=rx,o=rx" + loop: "{{ elt_warehouses }}" + loop_control: + loop_var: warehouse + +- ansible.builtin.include_tasks: elt_cron_tasks.yml + loop: "{{ elt_warehouses }}" + loop_control: + loop_var: warehouse + +# ---------- Iceberg maintenance jobs ---------- +- name: Create/update Iceberg maintenance script template + ansible.builtin.template: + src: ./cron/iceberg_maintenance.sh.j2 + dest: "{{ elt_cron_root_dir }}/iceberg_maintenance.sh" + mode: "u=rwx,g=rx,o=rx" + +- name: Ensure Iceberg table maintenance jobs exist + ansible.builtin.cron: + name: "iceberg-maintenance-{{ warehouse.name }}" + state: "{{ warehouse.maintenance.state }}" + hour: 3 + minute: 2 + job: > + NO_COLOR=1 UV_NO_PROGRESS=1 ELT_SECRETS={{ elt_secrets_root_dir }}/{{ warehouse.name }}/envvars + {{ elt_cron_root_dir }}/iceberg_maintenance.sh + > {{ elt_cron_logs_root_dir }}/{{ warehouse.name }}/iceberg_maintenance-$(date +\%Y\%m\%d_\%H\%M\%S).log 2>&1 + loop: "{{ elt_warehouses }}" + loop_control: + loop_var: warehouse diff --git a/infra/ansible-docker/playbooks/elt/templates/cron/elt_task.sh.j2 b/infra/ansible/roles/elt/templates/cron/elt_task.sh.j2 similarity index 100% rename from infra/ansible-docker/playbooks/elt/templates/cron/elt_task.sh.j2 rename to infra/ansible/roles/elt/templates/cron/elt_task.sh.j2 diff --git a/infra/ansible/roles/elt/templates/cron/iceberg_maintenance.sh.j2 b/infra/ansible/roles/elt/templates/cron/iceberg_maintenance.sh.j2 new file mode 100644 index 00000000..d9490a85 --- /dev/null +++ b/infra/ansible/roles/elt/templates/cron/iceberg_maintenance.sh.j2 @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Constants +ELT_SECRETS=${ELT_SECRETS:?} +ELT_GIT_REF=${ELT_GIT_REF:-main} + +# arguments +if [ $# -eq 0 ]; then + echo "Usage $0 " + exit 1 +fi + +git_url=$1 +shift 1 + +# Read secrets +set -o allexport; source $ELT_SECRETS; set +o allexport + +# Run maintenance tasks on all tables +uv_git_ref="git+${git_url}@${ELT_GIT_REF}#subdirectory=elt-common" +uvx \ + --with "${uv_git_ref}[iceberg-maintenance]" \ + python -m elt_common.iceberg.maintenance $* diff --git a/infra/ansible-docker/playbooks/elt/templates/dlt/config.toml.j2 b/infra/ansible/roles/elt/templates/dlt/config.toml.j2 similarity index 100% rename from infra/ansible-docker/playbooks/elt/templates/dlt/config.toml.j2 rename to infra/ansible/roles/elt/templates/dlt/config.toml.j2 diff --git a/infra/ansible-docker/playbooks/elt/templates/secrets/envvars.j2 b/infra/ansible/roles/elt/templates/secrets/envvars.j2 similarity index 60% rename from infra/ansible-docker/playbooks/elt/templates/secrets/envvars.j2 rename to infra/ansible/roles/elt/templates/secrets/envvars.j2 index f03b5d58..6a18225e 100644 --- a/infra/ansible-docker/playbooks/elt/templates/secrets/envvars.j2 +++ b/infra/ansible/roles/elt/templates/secrets/envvars.j2 @@ -1,29 +1,29 @@ -SOURCES__M365__CREDENTIALS__TENANT_ID={{ vault_ingestion_msgraph_directory_id }} -SOURCES__M365__CREDENTIALS__CLIENT_ID={{ vault_ingestion_msgraph_client_id }} -SOURCES__M365__CREDENTIALS__CLIENT_SECRET={{ vault_ingestion_msgraph_client_secret }} +SOURCES__M365__CREDENTIALS__TENANT_ID={{ vault_elt_msgraph_directory_id }} +SOURCES__M365__CREDENTIALS__CLIENT_ID={{ vault_elt_msgraph_client_id }} +SOURCES__M365__CREDENTIALS__CLIENT_SECRET={{ vault_elt_msgraph_client_secret }} OPRALOGWEB__SOURCES__CREDENTIALS__DRIVERNAME=mssql+pymssql OPRALOGWEB__SOURCES__CREDENTIALS__DATABASE=OpralogWebDB OPRALOGWEB__SOURCES__CREDENTIALS__SCHEMA=dbo OPRALOGWEB__SOURCES__CREDENTIALS__PORT=1433 -OPRALOGWEB__SOURCES__CREDENTIALS__HOST={{ vault_ingestion_opralogweb_host }} -OPRALOGWEB__SOURCES__CREDENTIALS__USERNAME={{ vault_ingestion_opralogweb_user }} -OPRALOGWEB__SOURCES__CREDENTIALS__PASSWORD={{ vault_ingestion_opralogweb_passwd }} +OPRALOGWEB__SOURCES__CREDENTIALS__HOST={{ vault_elt_opralogweb_host }} +OPRALOGWEB__SOURCES__CREDENTIALS__USERNAME={{ vault_elt_opralogweb_user }} +OPRALOGWEB__SOURCES__CREDENTIALS__PASSWORD={{ vault_elt_opralogweb_passwd }} DESTINATION__PYICEBERG__BUCKET_URL=s3://{{ lakekeeper_catalog.warehouses[warehouse.name].storage.bucket_name }} DESTINATION__PYICEBERG__CREDENTIALS__URI={{ lakekeeper_catalog.uri }} DESTINATION__PYICEBERG__CREDENTIALS__WAREHOUSE={{ warehouse.name }} DESTINATION__PYICEBERG__CREDENTIALS__ACCESS_DELEGATION=remote-signing -DESTINATION__PYICEBERG__CREDENTIALS__OAUTH2_SERVER_URI={{ keycloak_token_endpoint_url_isis }} -DESTINATION__PYICEBERG__CREDENTIALS__CLIENT_ID={{ vault_ingestion_catalog_client_id }} -DESTINATION__PYICEBERG__CREDENTIALS__CLIENT_SECRET={{ vault_ingestion_catalog_client_secret }} +DESTINATION__PYICEBERG__CREDENTIALS__OAUTH2_SERVER_URI={{ keycloak_realm_url }}/protocol/openid-connect/token +DESTINATION__PYICEBERG__CREDENTIALS__CLIENT_ID=elt +DESTINATION__PYICEBERG__CREDENTIALS__CLIENT_SECRET={{ vault_keycloak_client_secrets['elt'] }} DESTINATION__PYICEBERG__CREDENTIALS__SCOPE=lakekeeper DBT_TRINO_HTTP_SCHEME=https DBT_TRINO_HOST={{ top_level_domain }} DBT_TRINO_PORT={{ trino_https_port }} -DBT_TRINO_USER={{ trino_ingestion_username }} -DBT_TRINO_PASSWORD={{ vault_trino_ingestion_passwd }} +DBT_TRINO_USER=elt +DBT_TRINO_PASSWORD={{ vault_trino_users['elt'].password }} DBT_TRINO_THREADS={{ elt_threads }} DBT_TRINO_CATALOG={{ warehouse.name }} DBT_TRINO_CATALOG_SCHEMA_PREFIX= diff --git a/infra/ansible/roles/keycloak/defaults/main.yml b/infra/ansible/roles/keycloak/defaults/main.yml new file mode 100644 index 00000000..37744e63 --- /dev/null +++ b/infra/ansible/roles/keycloak/defaults/main.yml @@ -0,0 +1,12 @@ +keycloak_image: quay.io/keycloak/keycloak:26.3.5 +keycloak_image_optimized_name: keycloak_optimized +keycloak_log_level: INFO +keycloak_working_dir: /var/keycloak + +# Every client will have this set of default scopes +keycloak_client_default_scopes_minimum: + ["web-origins", "acr", "profile", "roles", "basic", "email"] +# Every client will have this set of optional scopes plus those defined +# by 'optional_scopes_additional' for each client +keycloak_client_optional_scopes_minimum: + ["address", "phone", "offline_access", "organization", "microprofile-jwt"] diff --git a/infra/ansible-docker/roles/keycloak/tasks/main.yml b/infra/ansible/roles/keycloak/tasks/main.yml similarity index 61% rename from infra/ansible-docker/roles/keycloak/tasks/main.yml rename to infra/ansible/roles/keycloak/tasks/main.yml index 10303d38..2f6ab5e5 100644 --- a/infra/ansible-docker/roles/keycloak/tasks/main.yml +++ b/infra/ansible/roles/keycloak/tasks/main.yml @@ -1,19 +1,19 @@ --- -- name: Ensure Keycloak working directory exists +- name: Create Keycloak working directory become: true ansible.builtin.file: path: "{{ keycloak_working_dir }}" state: directory mode: "u=rwx,g=rx,o=rx" -- name: Ensure certificates directory exists +- name: Create certificates directory become: true ansible.builtin.file: path: "{{ keycloak_working_dir }}/certs" state: directory mode: "u=rwx,g=rx,o=rx" -- name: Ensure certificates are present +- name: Copy certificates become: true no_log: true ansible.builtin.copy: @@ -27,7 +27,7 @@ loop_control: loop_var: certificate -- name: Ensure Dockerfile is up to date +- name: Generate Dockerfile for optimized build become: true ansible.builtin.template: src: "{{ template_file.src }}" @@ -38,40 +38,42 @@ loop_control: loop_var: template_file -- name: Ensure optimized Keycloak image is built - become: true +- name: Build optimized Keycloak image community.docker.docker_image_build: name: "{{ keycloak_image_optimized_name }}" path: "{{ keycloak_working_dir }}" rebuild: always -- name: Ensure Keycloak container is running - become: true +- name: Bootstrap Keycloak admin + no_log: "{{ not (keycloak_bootstrap_logging | default(false)) }}" community.docker.docker_container: - name: keycloak + name: keycloak_bootstrap image: "{{ keycloak_image_optimized_name }}" + command: + [ + "bootstrap-admin", + "user", + "--username={{ keycloak_bootstrap_admin_user }}", + "--password:env=KC_BOOTSTRAP_ADMIN_PASSWORD", + "--optimized", + "--no-prompt", + ] state: started cleanup: true - detach: true - restart_policy: unless-stopped + detach: false + restart_policy: "no" recreate: true - published_ports: - - "{{ keycloak_http_port }}:8080" - - "{{ keycloak_http_management_port }}:9000" env: # comment to keep formatter moving this up one line - KC_BOOTSTRAP_ADMIN_USERNAME: admin KC_BOOTSTRAP_ADMIN_PASSWORD: "{{ keycloak_bootstrap_admin_passwd }}" - KC_DB_URL: "jdbc:postgresql://{{ vault_db_host }}:{{ vault_db_port }}/{{ keycloak_db_dbname }}" + KC_DB_URL: "jdbc:postgresql://{{ keycloak_db_host }}:{{ keycloak_db_port }}/keycloak" KC_DB_USERNAME: "{{ keycloak_db_user }}" KC_DB_PASSWORD: "{{ keycloak_db_passwd }}" - networks: - - name: "{{ keycloak_container_network }}" - volumes: - - "{{ keycloak_working_dir }}/certs/stfc-ca.pem:/opt/keycloak/conf/truststores/stfc-ca.pem:ro" - comparisons: - networks: strict - env: strict + +- name: Run Keycloak + community.docker.docker_container: + name: keycloak + image: "{{ keycloak_image_optimized_name }}" command: [ "start", @@ -79,11 +81,30 @@ "--http-enabled=true", "--hostname=https://{{ top_level_domain }}{{ keycloak_base_path }}", "--proxy-headers=xforwarded", - "--proxy-trusted-addresses={{ openstack_reverse_proxy_fip }},127.0.0.0/8", - "--log-level=INFO", + "--proxy-trusted-addresses=127.0.0.0/8", + "--log-level={{ keycloak_log_level }}", ] + state: healthy + cleanup: true + detach: true + restart_policy: unless-stopped + recreate: true + env: + # comment to keep formatter moving this up one line + KC_DB_URL: "jdbc:postgresql://{{ keycloak_db_host }}:{{ keycloak_db_port }}/keycloak" + KC_DB_USERNAME: "{{ keycloak_db_user }}" + KC_DB_PASSWORD: "{{ keycloak_db_passwd }}" + network_mode: host + volumes: + - "{{ keycloak_working_dir }}/certs/stfc-ca.pem:/opt/keycloak/conf/truststores/stfc-ca.pem:ro" + comparisons: + env: strict healthcheck: test: "exec 3<>/dev/tcp/localhost/9000 && echo -e 'GET /health/ready HTTP/1.1\\r\\nHost: localhost\\r\\nConnection: close\\r\\n\\r\\n' >&3 && cat <&3 | grep -q '200 OK'" interval: 5s timeout: 2s retries: 15 + +- ansible.builtin.import_tasks: setup-realm.yml + vars: + target_realm: "{{ keycloak_realm.name }}" diff --git a/infra/ansible/roles/keycloak/tasks/setup-ldap.yml b/infra/ansible/roles/keycloak/tasks/setup-ldap.yml new file mode 100644 index 00000000..f56906ee --- /dev/null +++ b/infra/ansible/roles/keycloak/tasks/setup-ldap.yml @@ -0,0 +1,47 @@ +--- +- name: Connect to LDAP + community.general.keycloak_user_federation: + auth_client_id: admin-cli + auth_keycloak_url: "{{ keycloak_url }}" + auth_realm: master + auth_username: "{{ keycloak_bootstrap_admin_user }}" + auth_password: "{{ keycloak_bootstrap_admin_passwd }}" + realm: "{{ target_realm }}" + name: "STFC LDAP" + state: present + provider_id: ldap + connection_timeout: 20 + config: + enabled: true + editMode: READ_ONLY + importEnabled: true + syncRegistrations: false + vendor: "Active Directory" + # this is the mail in AD but using 'mail' gives a lookup error in Keycloak as some users don't have that set + usernameLDAPAttribute: userPrincipalName + rdnLDAPAttribute: cn + uuidLDAPAttribute: objectGUID + userObjectClasses: "person, organizationalPerson, user" + connectionUrl: "{{ vault_keycloak_ldap_connection_url }}" + usersDn: "OU=FBU,DC=fed,DC=cclrc,DC=ac,DC=uk" + authType: none + searchScope: 1 + validatePasswordPolicy: false + trustEmail: true + useTruststoreSpi: ldapsOnly + mappers: + - name: "first name" + providerId: "user-attribute-ldap-mapper" + config: + ldap.attribute: givenName + user.model.attribute: firstName + is.mandatory.in.ldap: false + attribute.force.default: true + is.binary.attribute: false + read.only: true + write.only: false + remove_unspecified_mappers: false + # The resource can be slow to create. The ansible role successfully creates it but then can generate + # a 404 trying to retrieve the new id that is not quite registered yet. Retry to avoid this. + retries: 3 + delay: 1 diff --git a/infra/ansible/roles/keycloak/tasks/setup-realm.yml b/infra/ansible/roles/keycloak/tasks/setup-realm.yml new file mode 100644 index 00000000..4651823c --- /dev/null +++ b/infra/ansible/roles/keycloak/tasks/setup-realm.yml @@ -0,0 +1,55 @@ +--- +# This is a hacky way so we can reuse the Keycloak auth_ vars +- ansible.builtin.debug: + msg: "" + vars: &keycloak_auth_vars + auth_client_id: admin-cli + auth_keycloak_url: "{{ keycloak_url }}" + auth_realm: master + auth_username: "{{ keycloak_bootstrap_admin_user }}" + auth_password: "{{ keycloak_bootstrap_admin_passwd }}" + when: false + +- name: Create Keycloak realm + community.general.keycloak_realm: + <<: *keycloak_auth_vars + realm: "{{ keycloak_realm.name }}" + display_name: "{{ keycloak_realm.display_name }}" + state: present + enabled: true + +- name: Create client scopes + community.general.keycloak_clientscope: + <<: *keycloak_auth_vars + realm: "{{ keycloak_realm.name }}" + name: "{{ item.name }}" + protocol: "{{ item.protocol }}" + protocol_mappers: "{{ item.protocol_mappers }}" + state: present + loop: "{{ keycloak_client_scopes }}" + +- name: Create service user clients + community.general.keycloak_client: + <<: *keycloak_auth_vars + realm: "{{ keycloak_realm.name }}" + client_id: "{{ item.client_id }}" + name: "{{ item.name }}" + protocol: "{{ item.protocol }}" + public_client: "{{ item.public_client }}" + secret: "{{ item.secret | default('') }}" + redirect_uris: "{{ item.redirect_uris | default([]) }}" + standard_flow_enabled: "{{ item.standard_flow_enabled }}" + implicit_flow_enabled: "{{ item.implicit_flow_enabled }}" + direct_access_grants_enabled: "{{ item.direct_access_grants_enabled }}" + service_accounts_enabled: "{{ item.service_accounts_enabled }}" + authorization_services_enabled: "{{ item.authorization_services_enabled }}" + default_client_scopes: "{{ keycloak_client_default_scopes_minimum }}" + optional_client_scopes: "{{ keycloak_client_optional_scopes_minimum + item.optional_client_scopes_additional }}" + attributes: "{{ item.attributes }}" + loop: "{{ keycloak_clients }}" + loop_control: + label: "{{ item.client_id }}" + +- ansible.builtin.import_tasks: setup-ldap.yml + vars: + target_realm: "{{ keycloak_realm.name }}" diff --git a/infra/ansible-docker/roles/keycloak/templates/Dockerfile.j2 b/infra/ansible/roles/keycloak/templates/Dockerfile.j2 similarity index 100% rename from infra/ansible-docker/roles/keycloak/templates/Dockerfile.j2 rename to infra/ansible/roles/keycloak/templates/Dockerfile.j2 diff --git a/infra/ansible-docker/roles/lakekeeper/defaults/main.yml b/infra/ansible/roles/lakekeeper/defaults/main.yml similarity index 100% rename from infra/ansible-docker/roles/lakekeeper/defaults/main.yml rename to infra/ansible/roles/lakekeeper/defaults/main.yml diff --git a/infra/ansible-docker/roles/lakekeeper/files/.gitignore b/infra/ansible/roles/lakekeeper/files/.gitignore similarity index 100% rename from infra/ansible-docker/roles/lakekeeper/files/.gitignore rename to infra/ansible/roles/lakekeeper/files/.gitignore diff --git a/infra/ansible-docker/roles/lakekeeper/files/bootstrap-warehouse.py b/infra/ansible/roles/lakekeeper/files/bootstrap-warehouse.py similarity index 84% rename from infra/ansible-docker/roles/lakekeeper/files/bootstrap-warehouse.py rename to infra/ansible/roles/lakekeeper/files/bootstrap-warehouse.py index 0fc3aaa5..caefe0cf 100644 --- a/infra/ansible-docker/roles/lakekeeper/files/bootstrap-warehouse.py +++ b/infra/ansible/roles/lakekeeper/files/bootstrap-warehouse.py @@ -29,8 +29,6 @@ LOGGER_FORMAT = "%(asctime)s|%(message)s" KEYCLOAK_MACHINE_USER_PREFIX = "service-account-" -LAKEKEEPER_PROJECT_ID_DEFAULT = "00000000-0000-0000-0000-000000000000" -LAKEKEEPER_PROJECT_NAME_DEFAULT = "Default Project" OIDC_PREFIX = "oidc~" REQUESTS_TIMEOUT_DEFAULT = 60.0 REQUESTS_DEFAULT_KWARGS: Dict[str, Any] = { @@ -178,45 +176,14 @@ def is_bootstrapped(self) -> bool: ) return response.json()["bootstrapped"] - def get_or_create_project( - self, name: str, new_project_id: str | None = None - ) -> str: - """Get the ID of a project with the given name or create one if it does not exist.""" - project_id = self.get_project(name) - if project_id is None: - project_id = self.create_project(name, new_project_id) - - return project_id - - def create_project(self, name: str, new_project_id: str | None = None): - """Create a new project, optionally specifying the ID""" - payload = {"project-name": name} - if new_project_id is not None: - payload["project-id"] = new_project_id + def rename_default_project(self, project_name: str): response = _request_with_auth( requests.post, - url=self.management_url + "/project", + url=self.management_url + "/project/rename", access_token=self.access_token, - json=payload, + json={"new-name": project_name}, ) response.raise_for_status() - project_id = response.json()["project-id"] - LOGGER.debug(f"Project '{name}' created with id '{project_id}'.") - return project_id - - def get_project(self, name: str) -> str | None: - """Get a project by name""" - response = _request_with_auth( - requests.get, - self.management_url + "/project-list", - self.access_token, - ) - for warehouse in response.json()["projects"]: - if warehouse["project-name"] == name: - LOGGER.debug(f"Project with name '{name}' already exists.'") - return warehouse["project-id"] - - return None def provision_user(self, access_token: str): """Provision a user through the /catalog/v1/config endpoint @@ -230,12 +197,11 @@ def provision_user(self, access_token: str): # but the user is provisioned internally. pass - def warehouse_exists(self, project_id: str, warehouse_name: str) -> bool: + def warehouse_exists(self, warehouse_name: str) -> bool: response = _request_with_auth( requests.get, self.management_url + "/warehouse", self.access_token, - headers={"x-project-id": project_id}, ) for warehouse in response.json()["warehouses"]: if warehouse["name"] == warehouse_name: @@ -243,23 +209,22 @@ def warehouse_exists(self, project_id: str, warehouse_name: str) -> bool: return False - def create_warehouse(self, project_id: str, warehouse_config: Dict[str, Any]): + def create_warehouse(self, warehouse_config: Dict[str, Any]): """Create a warehouse in the server with the given profile. If the bucket does not exist it is created. """ warehouse_name = warehouse_config["warehouse-name"] - if self.warehouse_exists(project_id, warehouse_name): + if self.warehouse_exists(warehouse_name): LOGGER.info( f"Warehouse '{warehouse_name}' already exists. Skipping warehouse creation." ) return - LOGGER.info(f"Creating warehouse '{warehouse_name}' in '{project_id}'") + LOGGER.info(f"Creating warehouse '{warehouse_name}'") _ensure_s3_bucket_exists( warehouse_config["storage-credential"], warehouse_config["storage-profile"] ) - warehouse_config["project-id"] = project_id _request_with_auth( requests.post, self.management_url + "/warehouse", @@ -322,7 +287,7 @@ def _ensure_s3_bucket_exists( s3.create_bucket(Bucket=storage_profile["bucket"]) except botocore.exceptions.ClientError as exc: code = str(exc.response.get("Error", {}).get("Code", "")) - if code in ("BucketAlreadyOwnedByYou", "BucketAlreadyExists"): + if code not in ("BucketAlreadyOwnedByYou", "BucketAlreadyExists"): raise @@ -330,13 +295,8 @@ def _ensure_s3_bucket_exists( @click.argument("lakekeeper-url") @click.option( "--project-name", - default=LAKEKEEPER_PROJECT_NAME_DEFAULT, help="Project name other than the default", ) -@click.option( - "--new-project-id", - help="An optional project ID for the new project. Default is a newly generated one.", -) @click.option("--keycloak-url", help="Base endpoint of Keycloak") @click.option("--keycloak-realm", help="Realm name to retrieve access token") @click.option( @@ -361,7 +321,6 @@ def _ensure_s3_bucket_exists( def main( lakekeeper_url: str, project_name: str, - new_project_id: str | None, keycloak_url: str | None, keycloak_realm: str | None, bootstrap_credentials: Credential | None, @@ -413,7 +372,9 @@ def main( server = LakekeeperRestV1(lakekeeper_url, access_token) server.bootstrap() - project_id = server.get_or_create_project(project_name, new_project_id) + if project_name is not None: + server.rename_default_project(project_name) + if server_admin is not None and identity_provider is not None: # Server admins are human users and will be provisioned the first time they log in server.assign_permissions( @@ -422,7 +383,10 @@ def main( ) server.assign_permissions( identity_provider.oidc_ids(server_admin, server.access_token), - entities={"server": ["admin"], f"project/{project_id}": ["project_admin"]}, + entities={ + "server": ["admin"], + "project": ["project_admin"], + }, ) if additional_user is not None and identity_provider is not None: @@ -437,7 +401,7 @@ def main( LOGGER.debug(f"Creating warehouse using file: {warehouse_json_file}") with open(warehouse_json_file) as fp: warehouse_json = json.load(fp) - server.create_warehouse(project_id, warehouse_json) + server.create_warehouse(warehouse_json) if __name__ == "__main__": diff --git a/infra/ansible-docker/roles/lakekeeper/tasks/main.yml b/infra/ansible/roles/lakekeeper/tasks/main.yml similarity index 60% rename from infra/ansible-docker/roles/lakekeeper/tasks/main.yml rename to infra/ansible/roles/lakekeeper/tasks/main.yml index 035fad23..57934ae9 100644 --- a/infra/ansible-docker/roles/lakekeeper/tasks/main.yml +++ b/infra/ansible/roles/lakekeeper/tasks/main.yml @@ -1,20 +1,4 @@ --- -- name: Ensure S3 buckets exist - amazon.aws.s3_bucket: - name: "{{ warehouse.value.storage.bucket_name }}" - endpoint_url: "{{ s3_endpoint }}" - access_key: "{{ s3_access_key_id }}" - secret_key: "{{ s3_access_secret }}" - ceph: true - loop: "{{ lakekeeper_catalog.warehouses | dict2items }}" - loop_control: - loop_var: warehouse - -- name: Get info on lakekeeper container - community.docker.docker_container_info: - name: "{{ lakekeeper_container_name }}" - register: result - - name: Stop Lakekeeper instance community.docker.docker_container: name: "{{ lakekeeper_container_name }}" @@ -35,14 +19,9 @@ LAKEKEEPER__PG_ENCRYPTION_KEY="{{ lakekeeper_metadb_encryption_key }}" LAKEKEEPER__PG_DATABASE_URL_READ="postgresql://{{ lakekeeper_metadb_user }}:{{ lakekeeper_metadb_passwd }}@{{ lakekeeper_metadb_host }}:5432/{{ lakekeeper_metadb_name }}" LAKEKEEPER__PG_DATABASE_URL_WRITE="postgresql://{{ lakekeeper_metadb_user }}:{{ lakekeeper_metadb_passwd }}@{{ lakekeeper_metadb_host }}:5432/{{ lakekeeper_metadb_name }}" - LAKEKEEPER__OPENID_PROVIDER_URI="http://{{ top_level_domain }}/auth/realms/{{ keycloak_realm }}" - LAKEKEEPER__OPENID_AUDIENCE="lakekeeper" - LAKEKEEPER__UI__OPENID_CLIENT_ID="lakekeeper" RUST_LOG="error" - networks: - - name: "{{ lakekeeper_container_network }}" + network_mode: host comparisons: - networks: strict env: strict - name: Start Lakekeeper container @@ -55,20 +34,18 @@ detach: true restart_policy: unless-stopped recreate: true - published_ports: - - "{{ lakekeeper_http_port }}:8181" env: # LAKEKEEPER__PG_ENCRYPTION_KEY="{{ lakekeeper_metadb_encryption_key }}" LAKEKEEPER__PG_DATABASE_URL_READ="postgresql://{{ lakekeeper_metadb_user }}:{{ lakekeeper_metadb_passwd }}@{{ lakekeeper_metadb_host }}:5432/{{ lakekeeper_metadb_name }}" LAKEKEEPER__PG_DATABASE_URL_WRITE="postgresql://{{ lakekeeper_metadb_user }}:{{ lakekeeper_metadb_passwd }}@{{ lakekeeper_metadb_host }}:5432/{{ lakekeeper_metadb_name }}" - LAKEKEEPER__OPENID_PROVIDER_URI="http://{{ top_level_domain }}/auth/realms/{{ keycloak_realm }}" - LAKEKEEPER__OPENID_AUDIENCE="lakekeeper" - LAKEKEEPER__UI__OPENID_CLIENT_ID="lakekeeper" - LAKEKEEPER__UI__OPENID_PROVIDER_URI="http://{{ top_level_domain }}/auth/realms/{{ keycloak_realm }}" + LAKEKEEPER__OPENID_PROVIDER_URI="{{ keycloak_realm_url }}" + LAKEKEEPER__OPENID_AUDIENCE=lakekeeper + LAKEKEEPER__UI__OPENID_PROVIDER_URI="{{ keycloak_realm_url }}" + LAKEKEEPER__UI__OPENID_CLIENT_ID=lakekeeper-ui + LAKEKEEPER__UI__OPENID_SCOPE="lakekeeper openid profile email" RUST_LOG="error" - networks: - - name: "{{ lakekeeper_container_network }}" + network_mode: host comparisons: networks: strict env: strict @@ -87,7 +64,7 @@ dest: "{{ lakekeeper_working_dir }}/bootstrap-warehouse.py" mode: "0755" -- name: Generate warehouse template(s) +- name: Generate warehouse templates become: true ansible.builtin.template: src: bootstrap-warehouse.json.j2 @@ -98,24 +75,21 @@ loop_var: warehouse register: warehouse_json_files -- name: Ensure Lakekeeper is bootstrapped +- name: Bootstrap Lakekeeper community.docker.docker_container: name: "{{ lakekeeper_container_name }}_bootstrap" image: ghcr.io/astral-sh/uv:python3.13-bookworm-slim - command: - [ - "uv", - "run", - "/opt/work/bootstrap-warehouse.py", - "--keycloak-url {{ keycloak_url }}", - "--keycloak-realm {{ keycloak_realm }}", - "--bootstrap-credentials ansible:{{ vault_keycloak_client_secret_ansible }}", - "--token-scope lakekeeper", - "--log-level {{ lakekeeper_bootstrap_log_level }}", - "--server-admin {{ vault_lakekeeper_admin_user }}", - "--warehouse-json-file", - "{{ warehouse_json_file | replace(lakekeeper_working_dir, '/opt/work') }}", - ] + command: >- + uv run + /opt/work/bootstrap-warehouse.py + --project-name "{{ lakekeeper_project_name }}" + --keycloak-url {{ keycloak_url }} + --keycloak-realm {{ keycloak_realm.name }} + --bootstrap-credentials {{ lakekeeper_bootstrap_credentials }} + --token-scope lakekeeper + --log-level {{ lakekeeper_bootstrap_log_level }} + --warehouse-json-file {{ warehouse_json_file | replace(lakekeeper_working_dir, '/opt/work') }} + http://localhost:{{ lakekeeper_http_port }} state: started cleanup: true detach: false @@ -125,8 +99,7 @@ # UV_LINK_MODE=copy UV_PROJECT_ENVIRONMENT=/opt/uv-venv - networks: - - name: "{{ lakekeeper_container_network }}" + network_mode: host volumes: - "{{ lakekeeper_working_dir }}:/opt/work" loop: "{{ warehouse_json_files.results | map(attribute='dest') | list }}" diff --git a/infra/ansible-docker/roles/lakekeeper/templates/bootstrap-warehouse.json.j2 b/infra/ansible/roles/lakekeeper/templates/bootstrap-warehouse.json.j2 similarity index 100% rename from infra/ansible-docker/roles/lakekeeper/templates/bootstrap-warehouse.json.j2 rename to infra/ansible/roles/lakekeeper/templates/bootstrap-warehouse.json.j2 diff --git a/infra/ansible/roles/minio/defaults/main.yml b/infra/ansible/roles/minio/defaults/main.yml new file mode 100644 index 00000000..90a4900e --- /dev/null +++ b/infra/ansible/roles/minio/defaults/main.yml @@ -0,0 +1,6 @@ +--- +minio_image: quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z +minio_api_port: 9000 +minio_console_port: 9001 +minio_console_base_path: /minio-ui +minio_working_dir: /var/minio diff --git a/infra/ansible/roles/minio/tasks/main.yml b/infra/ansible/roles/minio/tasks/main.yml new file mode 100644 index 00000000..773574d8 --- /dev/null +++ b/infra/ansible/roles/minio/tasks/main.yml @@ -0,0 +1,33 @@ +--- +- name: Create MinIO working directory + become: true + ansible.builtin.file: + path: "{{ minio_working_dir }}/data" + state: directory + mode: "u=rwx,g=rx,o=rx" + +- name: Ensure MinIO container is running + community.docker.docker_container: + name: minio + image: "{{ minio_image }}" + command: + [ + "server", + "/data", + "--address", + ":{{ minio_api_port }}", + "--console-address", + ":{{ minio_console_port }}", + ] + state: healthy + cleanup: true + detach: true + network_mode: host + restart_policy: unless-stopped + env: + # + MINIO_ROOT_USER: "{{ minio_root_user }}" + MINIO_ROOT_PASSWORD: "{{ minio_root_password }}" + MINIO_BROWSER_REDIRECT_URL: "https://{{ top_level_domain }}/{{ minio_console_base_path }}" + volumes: + - "{{ minio_working_dir }}/data:/data" diff --git a/infra/ansible-docker/roles/packages_update/tasks/main.yml b/infra/ansible/roles/packages_update/tasks/main.yml similarity index 86% rename from infra/ansible-docker/roles/packages_update/tasks/main.yml rename to infra/ansible/roles/packages_update/tasks/main.yml index f1647f9f..432df455 100644 --- a/infra/ansible-docker/roles/packages_update/tasks/main.yml +++ b/infra/ansible/roles/packages_update/tasks/main.yml @@ -1,6 +1,4 @@ --- -- ansible.builtin.include_tasks: ../../../tasks/ensure_openstack_apt_keys.yml - - name: Ensure all packages are up to date become: true ansible.builtin.apt: diff --git a/infra/ansible/roles/postgres/defaults/main.yml b/infra/ansible/roles/postgres/defaults/main.yml new file mode 100644 index 00000000..24f809ad --- /dev/null +++ b/infra/ansible/roles/postgres/defaults/main.yml @@ -0,0 +1,6 @@ +--- +postgres_db_data_path: /var/postgres/data +postgres_db_port: 5432 +postgres_db_user: postgres +postgres_docker_initdb_path: /var/postgres/docker-entrypoint-initdb.d +postgres_image: ghcr.io/cloudnative-pg/postgresql:16.9-bookworm diff --git a/infra/ansible/roles/postgres/tasks/main.yml b/infra/ansible/roles/postgres/tasks/main.yml new file mode 100644 index 00000000..b0a8685a --- /dev/null +++ b/infra/ansible/roles/postgres/tasks/main.yml @@ -0,0 +1,67 @@ +--- +- name: Install Python psycopg2 package + ansible.builtin.include_role: + name: geerlingguy.pip + apply: + become: true + vars: + pip_install_packages: + - name: psycopg2-binary + version: ">=2.5.1" + extra_args: --break-system-packages + +- name: Create postgres data directory exists + become: true + ansible.builtin.file: + path: "{{ postgres_db_data_path }}" + state: directory + owner: 26 + group: 999 + mode: "u=rwx,g=rx,o=rx" + +- name: Create secrets directory exists + become: true + ansible.builtin.file: + path: /secrets + state: directory + mode: "u=rwx,g=,o=" + +- name: Create postgres secrets + become: true + ansible.builtin.copy: + dest: /secrets/postgres-passwd + content: "{{ postgres_db_passwd }}" + owner: root + group: root + mode: "u=r,g=r,o=r" + +- name: Run Postgresql + become: true + community.docker.docker_container: + name: postgres + image: "{{ postgres_image }}" + state: healthy + cleanup: true + detach: true + network_mode: host + restart_policy: unless-stopped + env: + # + POSTGRES_USER="{{ postgres_db_user }}" + POSTGRES_PASSWORD_FILE=/secrets/postgres-passwd + volumes: + - "{{ postgres_db_data_path }}:/var/lib/postgresql/data" + - /secrets:/secrets:ro + +# The postgresql_ping check doesn't seem to be enough to check that postgres is ready +# to create a database. Instead just try until we succeed +- name: Create required databases + become: true + community.postgresql.postgresql_db: + name: "{{ item }}" + login_host: localhost + login_user: "{{ postgres_db_user }}" + login_password: "{{ postgres_db_passwd }}" + loop: "{{ postgres_db_names }}" + retries: 5 + delay: 2 diff --git a/infra/ansible/roles/redis/defaults/main.yml b/infra/ansible/roles/redis/defaults/main.yml new file mode 100644 index 00000000..6ddb522a --- /dev/null +++ b/infra/ansible/roles/redis/defaults/main.yml @@ -0,0 +1,4 @@ +--- +redis_work_dir: /var/redis +redis_image: redis:8.4-bookworm +redis_port: 6379 diff --git a/infra/ansible/roles/redis/tasks/main.yml b/infra/ansible/roles/redis/tasks/main.yml new file mode 100644 index 00000000..98c996eb --- /dev/null +++ b/infra/ansible/roles/redis/tasks/main.yml @@ -0,0 +1,25 @@ +--- +- name: Create redis working directory + become: true + ansible.builtin.file: + path: "{{ redis_work_dir }}" + state: directory + mode: "u=rwx,g=rx,o=rx" + +- name: Run Redis + become: true + community.docker.docker_container: + name: redis + image: "{{ redis_image }}" + command: + ["redis-server", "--port", "{{ redis_port }}", "--bind", "127.0.0.1 ::1"] + state: healthy + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 5s + retries: 3 + cleanup: true + detach: true + network_mode: host + restart_policy: unless-stopped diff --git a/infra/ansible/roles/superset_farm/defaults/main.yml b/infra/ansible/roles/superset_farm/defaults/main.yml new file mode 100644 index 00000000..be94c748 --- /dev/null +++ b/infra/ansible/roles/superset_farm/defaults/main.yml @@ -0,0 +1,4 @@ +--- +superset_farm_certs_dir: /var/certs +superset_farm_image: ghcr.io/isisneutronmuon/analytics-data-platform-superset:a4f0e8011f34785d759269dceedfa2693db81c19 +superset_farm_work_dir_prefix: /var/superset_farm diff --git a/infra/ansible-docker/roles/superset/files/docker/pythonpath/superset_config.py b/infra/ansible/roles/superset_farm/files/docker/pythonpath/superset_config.py similarity index 88% rename from infra/ansible-docker/roles/superset/files/docker/pythonpath/superset_config.py rename to infra/ansible/roles/superset_farm/files/docker/pythonpath/superset_config.py index 10b1d69b..ea9600e3 100644 --- a/infra/ansible-docker/roles/superset/files/docker/pythonpath/superset_config.py +++ b/infra/ansible/roles/superset_farm/files/docker/pythonpath/superset_config.py @@ -149,27 +149,23 @@ def _ldap_calculate_user_roles( # Caching layer REDIS_HOST = os.getenv("REDIS_HOST", "redis") REDIS_PORT = os.getenv("REDIS_PORT", "6379") - -CACHE_CONFIG = { +REDIS_DB = int(os.getenv("REDIS_DB", "0")) +COMMON_CACHE_CONFIG = { "CACHE_TYPE": "RedisCache", "CACHE_DEFAULT_TIMEOUT": 300, - "CACHE_KEY_PREFIX": "superset_metadata_cache", "CACHE_REDIS_HOST": REDIS_HOST, "CACHE_REDIS_PORT": REDIS_PORT, + "CACHE_REDIS_DB": REDIS_DB, } -DATA_CACHE_CONFIG = { - "CACHE_TYPE": "RedisCache", - "CACHE_DEFAULT_TIMEOUT": 300, - "CACHE_KEY_PREFIX": "superset_charting_data_cache", - "CACHE_REDIS_HOST": REDIS_HOST, - "CACHE_REDIS_PORT": REDIS_PORT, -} -##### - -##### +CACHE_CONFIG = dict(**COMMON_CACHE_CONFIG, CACHE_KEY_PREFIX="superset_metadata_cache") +DATA_CACHE_CONFIG = dict( + **COMMON_CACHE_CONFIG, CACHE_KEY_PREFIX="superset_charting_data_cache" +) +# FAB rate limiter +RATELIMIT_STORAGE_URI = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" # SQL Lab RESULTS_BACKEND = RedisCache( - host=REDIS_HOST, port=REDIS_PORT, key_prefix="superset_results_backend" + host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB, key_prefix="superset_results_backend" ) SQLLAB_CTAS_NO_LIMIT = True ##### @@ -178,9 +174,9 @@ def _ldap_calculate_user_roles( ##### # Celery class CeleryConfig: - broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0" + broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" imports = ("superset.sql_lab",) - result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0" + result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" worker_prefetch_multiplier = 1 task_acks_late = False beat_schedule = { @@ -202,25 +198,14 @@ class CeleryConfig: "https://{{ top_level_domain }}{os.environ.get('SUPERSET_APP_ROOT', '')}/" ) -##### -# Theming. -# Workaround missing brand logo: https://github.com/apache/superset/pull/34935. Once merged this can -# be removed. -THEME_DEFAULT = { - "token": { - "brandLogoUrl": f"{os.environ.get('SUPERSET_APP_ROOT', '')}/static/assets/images/superset-logo-horiz.png", - } -} - ##### # Misc features # fmt: off FEATURE_FLAGS = { - "ALERT_REPORTS": True, + "ALERT_REPORTS": False, "ENABLE_TEMPLATE_PROCESSING": True, "TAGGING_SYSTEM": True } # fmt: on -ALERT_REPORTS_NOTIFICATION_DRY_RUN = True SQLLAB_CTAS_NO_LIMIT = True ##### diff --git a/infra/ansible/roles/superset_farm/tasks/main.yml b/infra/ansible/roles/superset_farm/tasks/main.yml new file mode 100644 index 00000000..ff5a31b8 --- /dev/null +++ b/infra/ansible/roles/superset_farm/tasks/main.yml @@ -0,0 +1,39 @@ +--- +- name: Ensure certificates directory exists + become: true + ansible.builtin.file: + path: "{{ superset_farm_certs_dir }}" + state: directory + mode: "u=rwx,g=rx,o=rx" + +- name: Ensure certificates are present + become: true + no_log: true + ansible.builtin.copy: + dest: "{{ superset_farm_certs_dir }}/{{ certificate.filename }}" + content: "{{ certificate.content }}" + owner: root + group: root + mode: "u=r,o=r,g=r" + loop: + - { filename: stfc-ca.pem, content: "{{ stfc_ca_cert }}" } + loop_control: + loop_var: certificate + +- name: Install Python psycopg2 package on host + ansible.builtin.include_role: + name: geerlingguy.pip + apply: + become: true + vars: + pip_install_packages: + - name: psycopg2-binary + version: ">=2.5.1" + extra_args: --break-system-packages + +- name: Setup Superset instances + ansible.builtin.include_tasks: ./setup-instance.yml + loop: "{{ superset_farm_instances | dict2items }}" + loop_control: + loop_var: instance + label: "{{ instance.key }}" diff --git a/infra/ansible/roles/superset_farm/tasks/setup-instance.yml b/infra/ansible/roles/superset_farm/tasks/setup-instance.yml new file mode 100644 index 00000000..a21277b7 --- /dev/null +++ b/infra/ansible/roles/superset_farm/tasks/setup-instance.yml @@ -0,0 +1,110 @@ +--- +- name: Create a new DB schema ("{{ instance.key }}") + community.postgresql.postgresql_schema: + login_host: "{{ superset_farm_metadb_host }}" + login_db: "{{ superset_farm_metadb_name }}" + login_user: "{{ superset_farm_metadb_user }}" + login_password: "{{ superset_farm_metadb_passwd }}" + name: "{{ instance.key }}" + comment: "Superset schema for {{ instance.key }}" + +- ansible.builtin.set_fact: + instance_work_dir: "{{ superset_farm_work_dir_prefix }}/{{ instance.key }}" + +- name: Create working directory + become: true + ansible.builtin.file: + path: "{{ instance_work_dir }}" + state: directory + mode: "u=rwx,g=rx,o=rx" + +- name: Ensure configuration files are synchronized + become: true + ansible.posix.synchronize: + use_ssh_args: true + src: ./ + dest: "{{ instance_work_dir }}" + archive: false + checksum: true + recursive: true + delete: true + +- name: Ensure templated configuration files are synchronized + become: true + ansible.builtin.template: + src: "{{ superset_template.src }}" + dest: "{{ instance_work_dir }}/{{ superset_template.dest }}" + mode: "{{ superset_template.mode|default('u=rw,g=r,o=r') }}" + loop: + - { + src: superset-cli.sh.j2, + dest: superset-cli.sh, + mode: "u=rwx,g=rx,o=rx", + } + - { src: docker/dotenv.j2, dest: docker/.env } + - { + src: docker/docker-init.sh.j2, + dest: docker/docker-init.sh, + mode: "u=rwx,g=rx,o=r", + } + loop_control: + loop_var: superset_template + +- name: Ensure superset_config.yml exists + become: true + ansible.builtin.copy: + dest: "{{ instance_work_dir }}/docker/pythonpath/superset_config.yml" + content: "{{ instance.value.users_yaml | to_nice_yaml }}" + mode: "u=rw,g=r,o=r" + +- name: Run Superset ("{{ instance.key }}") + become: true + community.docker.docker_container: + name: "superset-{{ instance.key }}" + image: "{{ superset_farm_image }}" + command: ["/app/docker/docker-bootstrap.sh", "app-gunicorn"] + env_file: &superset-env-file "{{ instance_work_dir }}/docker/.env" + volumes: &superset-volumes + - "{{ instance_work_dir }}/docker/docker-init.sh:/app/docker/docker-init.sh:ro" + - "{{ instance_work_dir }}/docker/pythonpath:/app/docker/pythonpath:ro" + - "{{ superset_farm_certs_dir }}:/certs:ro" + state: healthy + healthy_wait_timeout: 120 + recreate: true + cleanup: true + detach: true + network_mode: host + restart_policy: on-failure + restart_retries: 3 + +- name: Run Superset init ("{{ instance.key }}") + become: true + community.docker.docker_container: + name: "superset-{{ instance.key }}-init" + image: "{{ superset_farm_image }}" + command: ["/app/docker/docker-init.sh"] + env_file: *superset-env-file + volumes: *superset-volumes + state: started + recreate: true + cleanup: true + detach: false + network_mode: host + restart_policy: on-failure + restart_retries: 3 + +- name: Run Superset worker ("{{ instance.key }}") + become: true + community.docker.docker_container: + name: "superset-{{ instance.key }}-worker" + image: "{{ superset_farm_image }}" + command: ["/app/docker/docker-bootstrap.sh", "worker"] + env_file: *superset-env-file + volumes: *superset-volumes + network_mode: host + state: started + recreate: true + cleanup: true + detach: true + restart_policy: on-failure + restart_retries: 3 diff --git a/infra/ansible-docker/roles/superset/templates/docker/docker-init.sh.j2 b/infra/ansible/roles/superset_farm/templates/docker/docker-init.sh.j2 similarity index 92% rename from infra/ansible-docker/roles/superset/templates/docker/docker-init.sh.j2 rename to infra/ansible/roles/superset_farm/templates/docker/docker-init.sh.j2 index 6677dd0d..0de68a5e 100755 --- a/infra/ansible-docker/roles/superset/templates/docker/docker-init.sh.j2 +++ b/infra/ansible/roles/superset_farm/templates/docker/docker-init.sh.j2 @@ -51,9 +51,9 @@ echo_step "2" "Complete" "Setting up roles and perms" # Create a database connection(s) echo_step "3" "Starting" "Setting up Iceberg catalog connection(s)" # shellcheck disable=1000-9999 -{% for analytic_db in superset_analytic_dbs %} +{% for analytic_db in instance.value.analytic_dbs %} superset set-database-uri \ - --database_name {{ analytic_db.database_name }} --uri "{{ analytic_db.uri }}" + --database_name {{ analytic_db.name }} --uri "{{ analytic_db.db_url }}" # shellcheck disable=1000-9999 {% endfor %} echo_step "3" "Complete" "Setting up data source database connection(s)" diff --git a/infra/ansible/roles/superset_farm/templates/docker/dotenv.j2 b/infra/ansible/roles/superset_farm/templates/docker/dotenv.j2 new file mode 100644 index 00000000..8ce7f2ab --- /dev/null +++ b/infra/ansible/roles/superset_farm/templates/docker/dotenv.j2 @@ -0,0 +1,28 @@ +# Metadata DB +SUPERSET_DB_DIALECT=postgresql +SUPERSET_DB_HOST={{ superset_farm_metadb_host }} +SUPERSET_DB_USER={{ superset_farm_metadb_user }} +SUPERSET_DB_PASSWORD={{ superset_farm_metadb_passwd }} +SUPERSET_DB_NAME={{ superset_farm_metadb_name }} +SUPERSET_DB_PORT={{ superset_farm_metadb_port }} +SUPERSET_DB_SCHEMA_NAME={{ instance.key }} + +# Caching +REDIS_HOST={{ superset_farm_redis_host }} +REDIS_PORT={{ superset_farm_redis_port }} + +# Environment/development settings +PYTHONPATH=/app/docker/pythonpath +SCARF_ANALYTICS=false +FLASK_DEBUG={{ instance.value.flask_debug }} +SUPERSET_LOG_LEVEL={{ "DEBUG" if instance.value.flask_debug else "INFO" }} +SUPERSET_ENV={{ instance.value.env_name }} +SUPERSET_PORT={{ instance.value.http_port }} +SUPERSET_APP_ROOT={{ instance.value.app_root }} + +# Make sure you set this to a unique secure random value on production +SUPERSET_SECRET_KEY={{ instance.value.secret_key }} + +# Auth +LDAP_SERVER={{ stfc_ldap_server }} +LDAP_CACERTFILE=/certs/stfc-ca.pem diff --git a/infra/ansible-docker/roles/superset/templates/superset-cli.sh.j2 b/infra/ansible/roles/superset_farm/templates/superset-cli.sh.j2 similarity index 52% rename from infra/ansible-docker/roles/superset/templates/superset-cli.sh.j2 rename to infra/ansible/roles/superset_farm/templates/superset-cli.sh.j2 index 63f231ec..91c88273 100644 --- a/infra/ansible-docker/roles/superset/templates/superset-cli.sh.j2 +++ b/infra/ansible/roles/superset_farm/templates/superset-cli.sh.j2 @@ -1,12 +1,14 @@ #!/bin/bash # Thin wrapper around superset command -CONTAINER_IMAGE={{ superset_container_image }} -SUPERSET_CONFIG_MOUNT={{ superset_working_dir }}/docker/pythonpath -ENV_FILE={{ superset_working_dir }}/docker/.env +CONTAINER_IMAGE={{ superset_farm_image }} +SUPERSET_CONFIG_MOUNT={{ instance_work_dir }}/docker/pythonpath +ENV_FILE={{ instance_work_dir }}/docker/.env docker run \ --rm \ --env-file $ENV_FILE \ --volume ${SUPERSET_CONFIG_MOUNT}:/app/docker/pythonpath:ro \ + --interactive \ + --tty \ ${CONTAINER_IMAGE} \ superset $* diff --git a/infra/ansible-docker/roles/traefik/defaults/main.yml b/infra/ansible/roles/traefik/defaults/main.yml similarity index 100% rename from infra/ansible-docker/roles/traefik/defaults/main.yml rename to infra/ansible/roles/traefik/defaults/main.yml diff --git a/infra/ansible/roles/traefik/tasks/container.yml b/infra/ansible/roles/traefik/tasks/container.yml new file mode 100644 index 00000000..6997211a --- /dev/null +++ b/infra/ansible/roles/traefik/tasks/container.yml @@ -0,0 +1,19 @@ +--- +- name: Set Traefik container state {{ traefik_container_state | default('started')}} + become: true + community.docker.docker_container: + name: traefik + image: "{{ traefik_image }}" + state: "{{ traefik_container_state | default('started') }}" + cleanup: true + detach: true + restart: true + recreate: true + restart_policy: unless-stopped + network_mode: host + volumes: + - "{{ traefik_config_root }}/{{ traefik_etc_path_relative }}:/etc/traefik" + - "{{ traefik_certs_root }}:/certs:ro" + - "{{ traefik_config_root }}/logs/traefik:/var/log/traefik" + - /var/run/docker.sock:/var/run/docker.sock:ro + - /etc/localtime:/etc/localtime:ro diff --git a/infra/ansible/roles/traefik/tasks/dynamic-config.yml b/infra/ansible/roles/traefik/tasks/dynamic-config.yml new file mode 100644 index 00000000..37b6906d --- /dev/null +++ b/infra/ansible/roles/traefik/tasks/dynamic-config.yml @@ -0,0 +1,32 @@ +--- +- name: Create temporary build directory for dynamic configurations + ansible.builtin.tempfile: + state: directory + register: traefik_dynamic_confs_tempdir + +- name: Regenerate new Traefik dynamic configuration (staging location) + become: true + ansible.builtin.template: + src: "{{ traefik_template }}" + dest: "{{ traefik_dynamic_confs_tempdir.path }}/{{ traefik_template | basename | replace('.j2', '') }}" + mode: "u=rw,g=r,o=r" + with_fileglob: "templates/{{ traefik_etc_dynamic_path_relative }}/*.j2" + loop_control: + loop_var: traefik_template + +- name: Synchronize new dynamic configuration + become: true + ansible.posix.synchronize: + use_ssh_args: true + src: "{{ traefik_dynamic_confs_tempdir.path }}/" + dest: "{{ traefik_config_root }}/{{ traefik_etc_dynamic_path_relative }}/" + archive: false + checksum: true + recursive: true + delete: true + delegate_to: "{{ inventory_hostname }}" + +- name: Remove temporary build directory for dynamic configurations + ansible.builtin.file: + path: "{{ traefik_dynamic_confs_tempdir.path }}" + state: absent diff --git a/infra/ansible/roles/traefik/tasks/main.yml b/infra/ansible/roles/traefik/tasks/main.yml new file mode 100644 index 00000000..462c9386 --- /dev/null +++ b/infra/ansible/roles/traefik/tasks/main.yml @@ -0,0 +1,8 @@ +--- +- ansible.builtin.import_tasks: static-config.yml + when: not (traefik_container_only | default(False)) + +- ansible.builtin.import_tasks: container.yml + +- ansible.builtin.import_tasks: dynamic-config.yml + when: not (traefik_container_only | default(False)) diff --git a/infra/ansible/roles/traefik/tasks/static-config.yml b/infra/ansible/roles/traefik/tasks/static-config.yml new file mode 100644 index 00000000..41b3e18d --- /dev/null +++ b/infra/ansible/roles/traefik/tasks/static-config.yml @@ -0,0 +1,49 @@ +--- +- name: Ensure Traefik directories exist + become: true + ansible.builtin.file: + path: "{{ traefik_dir.path }}" + state: directory + mode: "{{ traefik_dir.mode }}" + owner: "root" + group: "root" + loop: + - { + path: "{{ traefik_config_root }}/{{ traefik_etc_dynamic_path_relative }}", + mode: "u=rwx,g=rx,o=rx", + } + - { + path: "{{ traefik_config_root }}/logs/traefik", + mode: "u=rwx,g=rx,o=rx", + } + - { path: "{{ traefik_certs_root }}", mode: "u=rwx,g=,o=" } + loop_control: + loop_var: traefik_dir + +- name: Ensure TLS certifcates are present + become: true + no_log: true + ansible.builtin.copy: + dest: "{{ traefik_tls.filename }}" + content: "{{ traefik_tls.content }}" + owner: root + group: root + mode: "u=r,o=,g=" + loop: + - { + filename: "{{ traefik_certs_root }}/cert__domain.pem", + content: "{{ vault_tls_cert }}", + } + - { + filename: "{{ traefik_certs_root }}/key__domain.pem", + content: "{{ vault_tls_key }}", + } + loop_control: + loop_var: traefik_tls + +- name: Regenerate Traefik static configuration + become: true + ansible.builtin.template: + src: "templates/{{ traefik_etc_path_relative }}/traefik.yml.j2" + dest: "{{ traefik_config_root }}/{{ traefik_etc_path_relative }}/traefik.yml" + mode: "u=rw,g=r,o=r" diff --git a/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/datastore.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/datastore.yml.j2 new file mode 100644 index 00000000..b0e19f1f --- /dev/null +++ b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/datastore.yml.j2 @@ -0,0 +1,33 @@ +{% if 'datastore' in groups %} +http: + routers: + minioapi: + entryPoints: minio_tls + rule: Host(`{{ top_level_domain }}`) + service: minioapi + tls: {} + + minioconsole: + entryPoints: websecure + rule: Host(`{{ top_level_domain }}`) && PathPrefix(`{{ minio_console_base_path }}`) + service: minioconsole + middlewares: + - minioconsole-strip-prefix + tls: {} + + middlewares: + minioconsole-strip-prefix: + stripPrefix: + prefixes: + - "{{ minio_console_base_path }}" + + services: + minioapi: + loadBalancer: + servers: + - url: "http://{{ groups['datastore'][0] }}:{{ minio_api_port }}" + minioconsole: + loadBalancer: + servers: + - url: "http://{{ groups['datastore'][0] }}:{{ minio_console_port }}" +{% endif %} diff --git a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/docs.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/docs.yml.j2 similarity index 100% rename from infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/docs.yml.j2 rename to infra/ansible/roles/traefik/templates/etc/traefik/dynamic/docs.yml.j2 diff --git a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/keycloak.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/keycloak.yml.j2 similarity index 77% rename from infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/keycloak.yml.j2 rename to infra/ansible/roles/traefik/templates/etc/traefik/dynamic/keycloak.yml.j2 index 4ea4a3da..5d8fca20 100644 --- a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/keycloak.yml.j2 +++ b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/keycloak.yml.j2 @@ -1,3 +1,4 @@ +{% if 'keycloak' in groups %} http: routers: # TLS @@ -19,4 +20,5 @@ http: keycloak: loadBalancer: servers: - - url: "http://{{ hostvars[groups['keycloak'][0]]['ansible_host'] }}:{{ keycloak_http_port }}" + - url: "http://{{ groups['keycloak'][0] }}:{{ keycloak_http_port }}" +{% endif %} diff --git a/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/lakekeeper.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/lakekeeper.yml.j2 new file mode 100644 index 00000000..f2f8b5d9 --- /dev/null +++ b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/lakekeeper.yml.j2 @@ -0,0 +1,23 @@ +{% if 'lakekeeper' in groups %} +http: + routers: + lakeeper: + entryPoints: websecure + rule: Host(`{{ top_level_domain }}`) && PathPrefix(`{{ lakekeeper_base_path }}`) + service: lakekeeper + middlewares: + - lakekeeper-strip-prefix + tls: {} + + middlewares: + lakekeeper-strip-prefix: + stripPrefix: + prefixes: + - "{{ lakekeeper_base_path }}" + + services: + lakekeeper: + loadBalancer: + servers: + - url: "http://{{ groups['lakekeeper'][0] }}:{{ lakekeeper_http_port }}" +{% endif %} diff --git a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/root.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/root.yml.j2 similarity index 100% rename from infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/root.yml.j2 rename to infra/ansible/roles/traefik/templates/etc/traefik/dynamic/root.yml.j2 diff --git a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/superset.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/superset_farm.yml.j2 similarity index 53% rename from infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/superset.yml.j2 rename to infra/ansible/roles/traefik/templates/etc/traefik/dynamic/superset_farm.yml.j2 index 0c5d8604..5a3c1e05 100644 --- a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/superset.yml.j2 +++ b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/superset_farm.yml.j2 @@ -1,6 +1,7 @@ +{% if 'superset_farm' in groups %} http: routers: - {% for instance_name, props in superset_instances.items() %} + {% for instance_name, props in superset_farm_instances.items() %} {{ instance_name }}: entryPoints: websecure rule: Host(`{{ top_level_domain }}`) && PathPrefix(`{{ props.app_root }}`) @@ -9,9 +10,10 @@ http: {% endfor %} services: - {% for instance_name, props in superset_instances.items() %} + {% for instance_name, props in superset_farm_instances.items() %} {{ instance_name }}: loadBalancer: servers: - - url: "http://{{ hostvars[groups[instance_name][0]]['ansible_host'] }}:{{ props.http_port }}" + - url: "http://{{ groups['superset_farm'][0] }}:{{ props.http_port }}" {% endfor %} +{% endif %} diff --git a/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/tls.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/tls.yml.j2 new file mode 100644 index 00000000..df2c72d0 --- /dev/null +++ b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/tls.yml.j2 @@ -0,0 +1,4 @@ +tls: + certificates: + - certFile: /certs/cert__domain.pem + keyFile: /certs/key__domain.pem diff --git a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/traefik.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/traefik.yml.j2 similarity index 80% rename from infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/traefik.yml.j2 rename to infra/ansible/roles/traefik/templates/etc/traefik/dynamic/traefik.yml.j2 index 0008f208..7f1baf96 100644 --- a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/traefik.yml.j2 +++ b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/traefik.yml.j2 @@ -13,7 +13,7 @@ http: middlewares: dashboardauth: basicauth: - users: admin:$2y$10$xRntwpemRqOvRaDz6kLt1ed3d5JcHZSjTdlO3tCiKmnN42hdrQsA6 + users: admin:{{ vault_traefik_admin_passwd | password_hash('bcrypt', rounds=10, ident='2y') }} traefik-strip-prefix: stripPrefix: prefixes: diff --git a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/trino.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/trino.yml.j2 similarity index 65% rename from infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/trino.yml.j2 rename to infra/ansible/roles/traefik/templates/etc/traefik/dynamic/trino.yml.j2 index cb44f823..fa1e5bc2 100644 --- a/infra/ansible-docker/roles/traefik/templates/etc/traefik/dynamic/trino.yml.j2 +++ b/infra/ansible/roles/traefik/templates/etc/traefik/dynamic/trino.yml.j2 @@ -1,3 +1,4 @@ +{% if 'trino' in groups %} http: routers: trino: @@ -10,4 +11,5 @@ http: trino: loadBalancer: servers: - - url: "http://{{ hostvars[groups['trino'][0]]['ansible_host'] }}:{{ trino_http_port }}" + - url: "http://{{ groups['trino'][0] }}:{{ trino_http_port }}" +{% endif %} diff --git a/infra/ansible-docker/roles/traefik/templates/etc/traefik/traefik.yml.j2 b/infra/ansible/roles/traefik/templates/etc/traefik/traefik.yml.j2 similarity index 85% rename from infra/ansible-docker/roles/traefik/templates/etc/traefik/traefik.yml.j2 rename to infra/ansible/roles/traefik/templates/etc/traefik/traefik.yml.j2 index 373b8877..270b949a 100644 --- a/infra/ansible-docker/roles/traefik/templates/etc/traefik/traefik.yml.j2 +++ b/infra/ansible/roles/traefik/templates/etc/traefik/traefik.yml.j2 @@ -28,6 +28,10 @@ entryPoints: address: ":443" trino_tls: address: ":{{ trino_https_port }}" +{% if 'datastore' in groups %} + minio_tls: + address: ":{{ minio_api_port }}" +{% endif %} providers: file: diff --git a/infra/ansible-docker/roles/trino/defaults/main.yml b/infra/ansible/roles/trino/defaults/main.yml similarity index 72% rename from infra/ansible-docker/roles/trino/defaults/main.yml rename to infra/ansible/roles/trino/defaults/main.yml index 15ca216a..96fdc05c 100644 --- a/infra/ansible-docker/roles/trino/defaults/main.yml +++ b/infra/ansible/roles/trino/defaults/main.yml @@ -1,5 +1,7 @@ --- -trino_image: trinodb/trino:477 trino_http_port: 8060 trino_https_port: 8443 +trino_image: trinodb/trino:477 +trino_jvm_xmx: 4G +trino_shared_secret: "" trino_working_dir: /var/trino diff --git a/infra/ansible-docker/roles/trino/files/etc/access-control.properties b/infra/ansible/roles/trino/files/etc/access-control.properties similarity index 100% rename from infra/ansible-docker/roles/trino/files/etc/access-control.properties rename to infra/ansible/roles/trino/files/etc/access-control.properties diff --git a/infra/ansible-docker/roles/trino/files/etc/catalog/.gitkeep b/infra/ansible/roles/trino/files/etc/catalog/.gitkeep similarity index 100% rename from infra/ansible-docker/roles/trino/files/etc/catalog/.gitkeep rename to infra/ansible/roles/trino/files/etc/catalog/.gitkeep diff --git a/infra/ansible-docker/roles/trino/files/etc/log.properties b/infra/ansible/roles/trino/files/etc/log.properties similarity index 100% rename from infra/ansible-docker/roles/trino/files/etc/log.properties rename to infra/ansible/roles/trino/files/etc/log.properties diff --git a/infra/ansible-docker/roles/trino/files/etc/node.properties b/infra/ansible/roles/trino/files/etc/node.properties similarity index 100% rename from infra/ansible-docker/roles/trino/files/etc/node.properties rename to infra/ansible/roles/trino/files/etc/node.properties diff --git a/infra/ansible-docker/roles/trino/files/etc/password-authenticator.properties b/infra/ansible/roles/trino/files/etc/password-authenticator.properties similarity index 100% rename from infra/ansible-docker/roles/trino/files/etc/password-authenticator.properties rename to infra/ansible/roles/trino/files/etc/password-authenticator.properties diff --git a/infra/ansible-docker/roles/trino/tasks/main.yml b/infra/ansible/roles/trino/tasks/main.yml similarity index 76% rename from infra/ansible-docker/roles/trino/tasks/main.yml rename to infra/ansible/roles/trino/tasks/main.yml index 33b26191..0a1745a7 100644 --- a/infra/ansible-docker/roles/trino/tasks/main.yml +++ b/infra/ansible/roles/trino/tasks/main.yml @@ -34,18 +34,6 @@ loop_control: loop_var: trino_template -- name: Ensure isis catalog is enabled - become: true - ansible.builtin.template: - src: "{{ trino_template }}" - dest: "{{ trino_working_dir }}/{{ trino_template | trim('.j2') }}" - mode: "u=rw,g=r,o=r" - loop: - - etc/catalog/isis.properties.j2 - loop_control: - loop_var: trino_template - when: trino_enable_isis_catalog | default(False) - - name: Generate Trino catalog property files become: true ansible.builtin.copy: @@ -56,8 +44,8 @@ iceberg.rest-catalog.warehouse={{ warehouse_name }} iceberg.rest-catalog.vended-credentials-enabled=false iceberg.rest-catalog.security=OAUTH2 - iceberg.rest-catalog.oauth2.server-uri={{ keycloak_token_endpoint_url_isis }} - iceberg.rest-catalog.oauth2.credential={{ vault_trino_catalog_client_id }}:{{ vault_trino_catalog_client_secret }} + iceberg.rest-catalog.oauth2.server-uri={{ keycloak_realm_url }}/protocol/openid-connect/token + iceberg.rest-catalog.oauth2.credential=trino:{{ vault_keycloak_client_secrets['trino'] }} iceberg.rest-catalog.oauth2.scope=lakekeeper offline_access # Our S3 implementation does not have STS enabled so we need to provide # S3 credentials too @@ -73,6 +61,21 @@ loop_control: loop_var: warehouse_name +- name: Generate Trino access rules JSON + become: true + ansible.builtin.copy: + content: | + {{ trino_rules | to_json(indent=4) }} + dest: "{{ trino_working_dir }}/etc/rules.json" + mode: "u=rw,g=r,o=r" + vars: + trino_rules: + catalogs: + [ + { "user": "elt", "catalog": "(.*)", "allow": "all" }, + { "user": "superset", "catalog": "(.*)", "allow": "read-only" }, + ] + - name: Ensure credentials directory exists become: true ansible.builtin.file: @@ -82,26 +85,26 @@ - name: Ensure Trino passwords are available become: true + no_log: "{{ not (trino_passwords_logging | default(false)) }}" ansible.builtin.lineinfile: path: "{{ trino_working_dir }}/creds/trino.pass" - regexp: "^{{ user.username }}" - line: "{{ user.username }}:{{ user.password | password_hash('bcrypt', rounds=10, ident='2y') }}" + regexp: "^{{ user.key }}" + line: "{{ user.key }}:{{ user.value.password | password_hash('bcrypt', rounds=10, ident='2y') }}" state: present create: true owner: root group: root mode: u=rw,g=r,o=r - loop: "{{ trino_users }}" + loop: "{{ trino_users | dict2items }}" loop_control: loop_var: user - no_log: true + label: "{{ user.key }}" - name: Ensure Trino container is running - become: true community.docker.docker_container: name: trino image: "{{ trino_image }}" - state: started + state: healthy detach: true recreate: true restart: true diff --git a/infra/ansible-docker/roles/trino/templates/etc/config.properties.j2 b/infra/ansible/roles/trino/templates/etc/config.properties.j2 similarity index 87% rename from infra/ansible-docker/roles/trino/templates/etc/config.properties.j2 rename to infra/ansible/roles/trino/templates/etc/config.properties.j2 index 87140d54..46586166 100644 --- a/infra/ansible-docker/roles/trino/templates/etc/config.properties.j2 +++ b/infra/ansible/roles/trino/templates/etc/config.properties.j2 @@ -4,7 +4,7 @@ node-scheduler.include-coordinator=true catalog.management=static transaction.idle-timeout=8h -discovery.uri=http://trino:{{ trino_http_port }} +discovery.uri=http://localhost:{{ trino_http_port }} http-server.http.port={{ trino_http_port }} http-server.process-forwarded=true diff --git a/infra/ansible-docker/roles/trino/templates/etc/jvm.config.j2 b/infra/ansible/roles/trino/templates/etc/jvm.config.j2 similarity index 100% rename from infra/ansible-docker/roles/trino/templates/etc/jvm.config.j2 rename to infra/ansible/roles/trino/templates/etc/jvm.config.j2 diff --git a/infra/ansible/site.yml b/infra/ansible/site.yml new file mode 100644 index 00000000..2e1a92b8 --- /dev/null +++ b/infra/ansible/site.yml @@ -0,0 +1,48 @@ +--- +- name: Deploy traefik + hosts: traefik + roles: + - role: base + - role: docs + become: true + - role: traefik + +- name: Deploy data services (if required) + hosts: datastore + roles: + - role: base + - role: postgres + - role: minio + +- name: Deploy Keycloak + hosts: keycloak + roles: + - role: base + - role: keycloak + +- name: Deploy Lakekeeper + hosts: lakekeeper + roles: + - role: base + - role: lakekeeper + +- name: Deploy Trino + hosts: trino + roles: + - base + - trino + +- name: Deploy Superset instances + hosts: superset_farm + roles: + - base + - redis + - superset_farm + +- name: Deploy ELT node + hosts: elt + roles: + - base + - role: robertdebock.logrotate + become: true + - role: elt diff --git a/infra/ansible/site_system_update.yml b/infra/ansible/site_system_update.yml new file mode 100644 index 00000000..6eb60b52 --- /dev/null +++ b/infra/ansible/site_system_update.yml @@ -0,0 +1,27 @@ +--- +- name: Disable Traefik during updates + hosts: traefik + vars: + traefik_container_only: true + traefik_container_state: absent + roles: + - role: traefik + +# Skip Traefik as it serves as a jumphost +- name: Apply system package updates (except Traefik) + hosts: all:!traefik + roles: + - role: packages_update + +- name: Apply system package updates (Traefik) + hosts: traefik + roles: + - role: packages_update + +- name: Enable Traefik during updates + hosts: traefik + vars: + traefik_container_only: true + traefik_container_state: started + roles: + - role: traefik diff --git a/infra/ansible/terraform/.gitignore b/infra/ansible/terraform/.gitignore new file mode 100644 index 00000000..a2965b81 --- /dev/null +++ b/infra/ansible/terraform/.gitignore @@ -0,0 +1,29 @@ +# Terraform State and Local Files +terraform.tfstate +terraform.tfstate.backup +.terraform/ +.terraform.lock.hcl + +# Ansible inventories +*.ini + +# Crash log files +crash.log +crash.*.log + +# Ignore override files +override.tf +override.tf.json +*_override.tf +*_override.tf.json + +# IDE +.idea/ +.vscode/ +*.swp +*.swo +*~ +.DS_Store + +# Logs +*.log diff --git a/infra/ansible/terraform/README.md b/infra/ansible/terraform/README.md new file mode 100644 index 00000000..c01a9348 --- /dev/null +++ b/infra/ansible/terraform/README.md @@ -0,0 +1 @@ +# Cloud infrastructure provisioning diff --git a/infra/ansible/terraform/ansible-inventory.tmpl b/infra/ansible/terraform/ansible-inventory.tmpl new file mode 100644 index 00000000..270d0bbc --- /dev/null +++ b/infra/ansible/terraform/ansible-inventory.tmpl @@ -0,0 +1,10 @@ +[all:vars] +ansible_user=${ vm_user } +ansible_python_interpreter=${ python_interpreter } +ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q ${ vm_user }@${ floating_ip }"' + +%{ for index, group in ansible_groups ~} + +[${ansible_groups[index]}] +${ansible_ips[index]} +%{ endfor ~} diff --git a/infra/ansible/terraform/environments/dev.tfvars b/infra/ansible/terraform/environments/dev.tfvars new file mode 100644 index 00000000..decea7e2 --- /dev/null +++ b/infra/ansible/terraform/environments/dev.tfvars @@ -0,0 +1,75 @@ +# general +vm_user = "ubuntu" +vm_key_pair = "martyngigg" +vm_image = "ubuntu-noble-24.04-nogui" + +# networking +network_name = "lakehouse-dev" +network_subnet_cidr = "192.168.44.0/24" +external_network_id = "5283f642-8bd8-48b6-8608-fa3006ff4539" +security_groups = { + # 80 & 443 already exist as default groups in the cloud + "traefik-dev" = { + ports_ingress = [8443, 9000] + }, + "datastore-dev" = { + ports_ingress = [5432, 9000, 9001] + } + "keycloak-dev" = { + ports_ingress = [8080, 9000] + }, + "lakekeeper-dev" = { + ports_ingress = [8181] + }, + "trino-dev" = { + ports_ingress = [8060] + }, + "superset_farm-dev" = { + ports_ingress = [8088] + } +} + +# compute +instance_configs = { + "traefik-dev" = { + flavor_name = "l3.nano" + ansible_group = "traefik" + additional_security_groups = ["HTTP", "HTTPS", "traefik-dev"] + }, + "datastore-dev" = { + flavor_name = "l3.nano" + ansible_group = "datastore" + additional_security_groups = ["datastore-dev"] + }, + "keycloak-dev" = { + flavor_name = "l3.nano" + ansible_group = "keycloak" + additional_security_groups = ["keycloak-dev"] + }, + "lakekeeper-dev" = { + flavor_name = "l3.nano" + ansible_group = "lakekeeper" + additional_security_groups = ["lakekeeper-dev"] + }, + "trino-dev" = { + flavor_name = "l3.nano" + ansible_group = "trino" + additional_security_groups = ["trino-dev"] + }, + "superset_farm-dev" = { + flavor_name = "l3.nano" + ansible_group = "superset_farm" + additional_security_groups = ["superset_farm-dev"] + } + "elt-dev" = { + flavor_name = "l3.nano" + ansible_group = "elt" + } +} +floating_ip_info = { + vm_name = "traefik-dev" + floating_ip = "130.246.81.245" +} + +# ansible +ansible_inventory_filename = "inventory-dev.ini" diff --git a/infra/ansible/terraform/main.tf b/infra/ansible/terraform/main.tf new file mode 100644 index 00000000..f8361964 --- /dev/null +++ b/infra/ansible/terraform/main.tf @@ -0,0 +1,114 @@ +terraform { + required_version = ">= 0.14.0" + required_providers { + openstack = { + source = "terraform-provider-openstack/openstack" + version = "~> 3.4.0" + } + } +} + +provider "openstack" { + cloud = var.cloud_name +} + +# networking +resource "openstack_networking_network_v2" "network" { + name = var.network_name + external = false + admin_state_up = true +} + +resource "openstack_networking_subnet_v2" "subnet" { + name = "${var.network_name}-1" + network_id = openstack_networking_network_v2.network.id + cidr = var.network_subnet_cidr + ip_version = 4 +} + +resource "openstack_networking_router_v2" "router" { + name = "${var.network_name}-router" + admin_state_up = true + external_network_id = var.external_network_id +} + +resource "openstack_networking_router_interface_v2" "router_interface" { + router_id = openstack_networking_router_v2.router.id + subnet_id = openstack_networking_subnet_v2.subnet.id +} + +resource "openstack_networking_secgroup_v2" "sec_group" { + for_each = var.security_groups + + name = each.key + description = "Ingress ports for ${each.key}" +} + +locals { + sec_groups_map = merge([ + for sg_name, sg_config in var.security_groups : { + for port in sg_config.ports_ingress : "${sg_name}-${port}" => { + sec_group_id = openstack_networking_secgroup_v2.sec_group[sg_name].id + port_ingress = port + } + } + ]...) +} + +resource "openstack_networking_secgroup_rule_v2" "secgroup_ingress" { + for_each = local.sec_groups_map + + direction = "ingress" + ethertype = "IPv4" + protocol = "tcp" + port_range_min = each.value.port_ingress + port_range_max = each.value.port_ingress + remote_ip_prefix = "0.0.0.0/0" + security_group_id = each.value.sec_group_id +} + +# compute +resource "openstack_compute_instance_v2" "vm" { + for_each = var.instance_configs + + name = each.key + flavor_name = each.value.flavor_name + image_name = var.vm_image + key_pair = var.vm_key_pair + security_groups = concat( + ["default"], + each.value.additional_security_groups != null ? each.value.additional_security_groups : [] + ) + + network { + name = var.network_name + } + + depends_on = [openstack_networking_subnet_v2.subnet] +} + +data "openstack_networking_port_v2" "vm_port" { + device_id = openstack_compute_instance_v2.vm[var.floating_ip_info.vm_name].id + depends_on = [openstack_compute_instance_v2.vm] +} + +resource "openstack_networking_floatingip_associate_v2" "vm_fip" { + floating_ip = var.floating_ip_info.floating_ip + port_id = data.openstack_networking_port_v2.vm_port.id +} + +resource "local_file" "ansible_inventory" { + content = templatefile("ansible-inventory.tmpl", + { + vm_user = var.vm_user + python_interpreter = var.python_interpreter_path + floating_ip = var.floating_ip_info.floating_ip + ansible_groups = [ + for vm in openstack_compute_instance_v2.vm : + var.instance_configs[vm.name].ansible_group + ] + ansible_ips = [for vm in openstack_compute_instance_v2.vm : vm.access_ip_v4] + } + ) + filename = var.ansible_inventory_filename +} diff --git a/infra/ansible/terraform/outputs.tf b/infra/ansible/terraform/outputs.tf new file mode 100644 index 00000000..f5659195 --- /dev/null +++ b/infra/ansible/terraform/outputs.tf @@ -0,0 +1,4 @@ +output "ansible_inventory" { + description = "The full path to the generated Ansible inventory" + value = local_file.ansible_inventory.filename +} diff --git a/infra/ansible/terraform/variables.tf b/infra/ansible/terraform/variables.tf new file mode 100644 index 00000000..0e1239a6 --- /dev/null +++ b/infra/ansible/terraform/variables.tf @@ -0,0 +1,70 @@ +variable "cloud_name" { + description = "Name of the cloud in the ~/.config/openstack/clouds.yaml file." + type = string +} + +variable "vm_user" { + description = "Name of the vm user automatically provisioned by the cloud and used by Ansible." + type = string +} +variable "vm_image" { + description = "Name of the vm image." + type = string +} + +variable "vm_key_pair" { + description = "Name of the keypair to add to the machine" + type = string +} + +variable "python_interpreter_path" { + description = "Path to the Python interpreter for Ansible." + type = string + default = "/usr/bin/python3" +} + +variable "external_network_id" { + description = "Id of the existing External network on the cloud" + type = string +} + +variable "network_name" { + description = "Name of the network" + type = string +} + +variable "network_subnet_cidr" { + description = "CIDR of network subnet" + type = string +} + +variable "security_groups" { + description = "Details of the ports to open on each compute instance" + type = map(object({ + ports_ingress = list(number) + })) +} + +variable "instance_configs" { + description = "Map of instance configurations where the key defines the name. The instance is attached to the network defined by 'network_name'" + type = map(object({ + flavor_name = string + ansible_group = string + additional_security_groups = optional(list(string)) + })) + +} + +variable "floating_ip_info" { + description = "Define the instance that has the floating IP attached" + type = object({ + vm_name = string + floating_ip = string + } + ) +} + +variable "ansible_inventory_filename" { + description = "Filename for the generated ansible inventory" + type = string +} diff --git a/infra/local/docker-compose.yml b/infra/local/docker-compose.yml index bf63a8e6..e44c653a 100644 --- a/infra/local/docker-compose.yml +++ b/infra/local/docker-compose.yml @@ -6,8 +6,8 @@ x-keycloak-image: &keycloak-image quay.io/keycloak/keycloak:26.3 x-openfga-image: &openfga-image openfga/openfga:v1.10 x-lakekeeper-image: &lakekeeper-image quay.io/lakekeeper/catalog:v0.11.1 x-minio-image: &minio-image quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z -x-minio-mc-image: &minio-mc-image quay.io/minio/mc:RELEASE.2025-08-13T08-35-41Z x-postgres-image: &postgres-image postgres:16.9-bookworm +x-redis-image: &redis-image redis:8.4-bookworm x-superset-image: &superset-image ghcr.io/isisneutronmuon/analytics-data-platform-superset:a4f0e8011f34785d759269dceedfa2693db81c19 x-traefik-image: &traefik-image traefik:v3.5.3 x-trino-image: &trino-image trinodb/trino:477 @@ -298,7 +298,6 @@ services: for name in dev_isis_raw dev_isis_cleaned; do uv run /opt/work/bootstrap-warehouse.py \ --project-name "$$KC_REALM_NAME" \ - --new-project-id "$$ADP_LAKEKEEPER_PROJECT_ID" \ --keycloak-url "$$KEYCLOAK_URL_INTERNAL" \ --keycloak-realm "$$KC_REALM_NAME" \ --bootstrap-credentials "machine-infra:s3cr3t" \ @@ -314,7 +313,7 @@ services: lakekeeper: condition: service_healthy volumes: - - "../ansible-docker/roles/lakekeeper/files/bootstrap-warehouse.py:/opt/work/bootstrap-warehouse.py" + - "../ansible/roles/lakekeeper/files/bootstrap-warehouse.py:/opt/work/bootstrap-warehouse.py" - "./warehouses/lakekeeper:/opt/work/warehouses:ro" - *certs-volume networks: *local-networks @@ -353,7 +352,7 @@ services: - superset container_name: local-lakehouse-superset-redis env_file: *env-file - image: redis:8 + image: *redis-image restart: unless-stopped volumes: - shared_redis_data:/data diff --git a/infra/local/env-local b/infra/local/env-local index 26538128..cbe99f7a 100644 --- a/infra/local/env-local +++ b/infra/local/env-local @@ -64,7 +64,6 @@ SHARED_DB__LAKEKEEPER_DB_NAME=lakekeeper LAKEKEEPER__PG_ENCRYPTION_KEY=This-is-NOT-Secure! LAKEKEEPER__PG_DATABASE_URL_READ=postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@$SHARED_DB__HOST:$SHARED_DB__PORT/$SHARED_DB__LAKEKEEPER_DB_NAME LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@$SHARED_DB__HOST:$SHARED_DB__PORT/$SHARED_DB__LAKEKEEPER_DB_NAME -LAKEKEEPER__ENABLE_DEFAULT_PROJECT=false LAKEKEEPER__OPENID_PROVIDER_URI=$KEYCLOAK_REALM_INTERNAL LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS=$KEYCLOAK_REALM_EXTERNAL LAKEKEEPER__OPENID_AUDIENCE=lakekeeper @@ -75,9 +74,6 @@ LAKEKEEPER__UI__OPENID_PROVIDER_URI=$KEYCLOAK_REALM_EXTERNAL LAKEKEEPER__UI__OPENID_CLIENT_ID=lakekeeper LAKEKEEPER__UI__OPENID_SCOPE="lakekeeper openid profile email" -# New project ID -ADP_LAKEKEEPER_PROJECT_ID=c4fcd44f-7ce7-4446-9f7c-dcc7ba76dd22 - ############################################################################### # Trino ############################################################################### @@ -102,6 +98,7 @@ SUPERSET_DB_DIALECT=postgresql # Caching layer (match service name and port in compose file) REDIS_HOST=superset-redis REDIS_PORT=6379 +REDIS_DB=1 # Admin user SUPERSET_ADMIN_USER=superset_admin diff --git a/infra/local/superset/docker/pythonpath/superset_config.py b/infra/local/superset/docker/pythonpath/superset_config.py index 9d2c420d..8fc44b04 100644 --- a/infra/local/superset/docker/pythonpath/superset_config.py +++ b/infra/local/superset/docker/pythonpath/superset_config.py @@ -36,7 +36,6 @@ ) if SUPERSET_DB_SCHEMA_NAME: SQLALCHEMY_DATABASE_URI += f"?options=-c%20search_path={SUPERSET_DB_SCHEMA_NAME}" -##### ##### # Auth @@ -103,27 +102,21 @@ def load_user_jwt(self, _jwt_header, jwt_data): # Caching layer REDIS_HOST = os.getenv("REDIS_HOST", "redis") REDIS_PORT = os.getenv("REDIS_PORT", "6379") - -CACHE_CONFIG = { +REDIS_DB = int(os.getenv("REDIS_DB", "0")) +COMMON_CACHE_CONFIG = { "CACHE_TYPE": "RedisCache", "CACHE_DEFAULT_TIMEOUT": 300, - "CACHE_KEY_PREFIX": "superset_metadata_cache", "CACHE_REDIS_HOST": REDIS_HOST, "CACHE_REDIS_PORT": REDIS_PORT, + "CACHE_REDIS_DB": REDIS_DB, } -DATA_CACHE_CONFIG = { - "CACHE_TYPE": "RedisCache", - "CACHE_DEFAULT_TIMEOUT": 300, - "CACHE_KEY_PREFIX": "superset_charting_data_cache", - "CACHE_REDIS_HOST": REDIS_HOST, - "CACHE_REDIS_PORT": REDIS_PORT, -} -##### - -##### -# SQL Lab +CACHE_CONFIG = dict(**COMMON_CACHE_CONFIG, CACHE_KEY_PREFIX="superset_metadata_cache") +DATA_CACHE_CONFIG = dict( + **COMMON_CACHE_CONFIG, CACHE_KEY_PREFIX="superset_charting_data_cache" +) +# SQL lab RESULTS_BACKEND = RedisCache( - host=REDIS_HOST, port=REDIS_PORT, key_prefix="superset_results_backend" + host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB, key_prefix="superset_results_backend" ) SQLLAB_CTAS_NO_LIMIT = True @@ -131,9 +124,9 @@ def load_user_jwt(self, _jwt_header, jwt_data): ##### # Celery class CeleryConfig: - broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0" + broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" imports = ("superset.sql_lab",) - result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0" + result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}" worker_prefetch_multiplier = 1 task_acks_late = False beat_schedule = { diff --git a/infra/local/warehouses/trino/dev_isis_cleaned.properties b/infra/local/warehouses/trino/dev_isis_cleaned.properties index 58786181..20d98b67 100644 --- a/infra/local/warehouses/trino/dev_isis_cleaned.properties +++ b/infra/local/warehouses/trino/dev_isis_cleaned.properties @@ -1,6 +1,6 @@ connector.name=iceberg iceberg.catalog.type=rest -iceberg.rest-catalog.warehouse=${ENV:ADP_LAKEKEEPER_PROJECT_ID}/dev_isis_cleaned +iceberg.rest-catalog.warehouse=dev_isis_cleaned iceberg.rest-catalog.uri=http://${ENV:ROUTER_HOSTNAME_INTERNAL}:${ENV:ROUTER_PORT_HTTP}/iceberg/catalog iceberg.rest-catalog.vended-credentials-enabled=false iceberg.rest-catalog.security=OAUTH2 diff --git a/infra/local/warehouses/trino/dev_isis_raw.properties b/infra/local/warehouses/trino/dev_isis_raw.properties index a8062277..1b5736e1 100644 --- a/infra/local/warehouses/trino/dev_isis_raw.properties +++ b/infra/local/warehouses/trino/dev_isis_raw.properties @@ -1,6 +1,6 @@ connector.name=iceberg iceberg.catalog.type=rest -iceberg.rest-catalog.warehouse=${ENV:ADP_LAKEKEEPER_PROJECT_ID}/dev_isis_raw +iceberg.rest-catalog.warehouse=dev_isis_raw iceberg.rest-catalog.uri=http://${ENV:ROUTER_HOSTNAME_INTERNAL}:${ENV:ROUTER_PORT_HTTP}/iceberg/catalog iceberg.rest-catalog.vended-credentials-enabled=false iceberg.rest-catalog.security=OAUTH2