Add auto yaml to java generation #2984

derrickaw · 2025-11-13T19:33:14Z

Update KafkaToBigQuery.yaml to jive with existing KafkaToBigQuery.java template.
Add generation script to convert yaml blueprint to java interface.
~~3. Add workflow to inspect each pull request opened to rerun generation script and create or modify java templates.~~
Add java template file as a baseline file.
Add generic readme on how to add new blueprints, templates, etc.

codecov · 2025-11-13T19:48:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.24%. Comparing base (59d8c15) to head (437942b).
⚠️ Report is 16 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##               main    #2984    +/-   ##
==========================================
  Coverage     50.23%   50.24%            
- Complexity     5001     5018    +17     
==========================================
  Files           966      967     +1     
  Lines         59157    59261   +104     
  Branches       6445     6458    +13     
==========================================
+ Hits          29719    29777    +58     
- Misses        27334    27377    +43     
- Partials       2104     2107     +3

Components	Coverage Δ
spanner-templates	`70.42% <ø> (-0.03%)`	⬇️
spanner-import-export	`68.96% <ø> (-0.03%)`	⬇️
spanner-live-forward-migration	`79.69% <ø> (ø)`
spanner-live-reverse-replication	`77.03% <ø> (-0.05%)`	⬇️
spanner-bulk-migration	`88.33% <ø> (ø)`
see 6 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

update script name update workflow debug fix step names checkout branch try again more debug update to fetch and checkout add permissions add token fix names etc

…ould result from the new workflow

…rectory and file

damccorm

Thanks - the general flow LGTM, just had questions about the workflow structure

damccorm · 2025-11-17T22:05:24Z

.github/workflows/yaml_blueprint_to_java_template.yml

+name: YAML to Java Template Generation or Modification
+
+on:
+  pull_request:


I think we probably want to run this on different conditions (probably commits to this branch on the main branch or on a schedule). We should also add a workflow_dispatch trigger

Oh, I see - you're committing it back to the pull request branch. In general, I think it is probably better to have this as a standalone thing rather than tying it to PRs - that way it can be run if we update the underlying generation infrastructure (or make similar other changes)

I also think this won't work on forks since I don't think the workflow has permissions on forks. I'd recommend following the pattern in https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/.github/workflows/update-python-deps.yml

Alternatively, you could require that a user generate these as part of their pull request and fail the check if they don't (similar to how linting checks work). To do this, you could do all the same things you're already doing, but finish with git diff --exit-code to fail the job if there are differences detected.

In general, I'd stay away from modifying a user's branch though, since it is surprising and will run into those permission problems.

i like the final suggestion, at least for the initial version. Let us give instructions for users to generate java interface from yaml using your script(We did similar setup for ManagedIO docs). Committing to working branch might not be good. I am happy not to have a validation check (PR vs script output) too. We might not want to create Template for every Blueprint.

For the initial version:

Removed the workflow.

Added a readme file explaining how to add the yaml blueprint, interface, IT, etc.

Added mvn spotless process to the generate script from the workflow file to minimize that probable check failure.

Thanks

damccorm · 2025-11-17T22:05:59Z

.github/workflows/yaml_blueprint_to_java_template.yml

+        with:
+          fetch-depth: 0


Any reason for this? This will slow clones

When I was iterating on this and with my understanding, it was required to have the full history to know about the branch and its context for pushing back to the same PR. Its a mute point now based on other comments, so will remove. Thanks.

tarun-google

LGTM

tarun-google · 2025-11-18T16:51:40Z

yaml/src/main/yaml/KafkaToBigQuery.yaml


    - name: "kafkaReadTopics"
-      help: "Kafka topic to read the input from."
+      description: "Kafka topic(s) to read the input from."


QQ: Are we displaying these new fields in UI too?

This yaml file isn't used currently in the UI for Job Builder. The template fields in the java template are used currently in the Create Job from template.

chamikaramj

Thanks!

chamikaramj · 2025-11-18T18:20:32Z

yaml/README.md

+
+## Steps
+
+1.  **Add YAML Blueprint:** Create the YAML blueprint file that defines the template's structure and parameters.


Add a link on contribution instructions to this Github repo ?

done, thanks

chamikaramj · 2025-11-18T18:21:38Z

yaml/README.md

+```
+
+3.  **Create Integration Test:** Develop an integration test to validate the functionality of the new template.
+Place this file in [here](https://github.com/GoogleCloudPlatform/DataflowTemplates/yaml/src/test/java).


Any existing instructions on creating the integration tests for Flex templates that we can link to ?

done, thanks

chamikaramj · 2025-11-18T18:47:04Z

yaml/scripts/generate_yaml_java_templates.py

+
+
+def get_java_type(param_type):
+    """Maps a YAML parameter type to a Java type."""


Why are we limited to these three types here ?

Goes back to the limitation of https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/metadata/src/main/java/com/google/cloud/teleport/metadata/TemplateParameter.java#L24

Updated a little.

chamikaramj · 2025-11-18T18:47:16Z

yaml/scripts/generate_yaml_java_templates.py

+        return 'String'
+
+def get_template_parameter_type(param_type):
+    """Maps a YAML parameter type to a TemplateParameter annotation type."""


The template parameter types are limited based on this - https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/metadata/src/main/java/com/google/cloud/teleport/metadata/TemplateParameter.java#L24

Updated a little.

chamikaramj · 2025-11-18T18:48:29Z

yaml/scripts/generate_yaml_java_templates.py

+    with open(yaml_path, 'r') as f:
+        content = f.read()
+        # Remove Jinja variables before parsing
+        content = re.sub(r'{{.*?}}', '', content)


Probably we should have some validation to make sure we do not remove required params ?

Under normal situation jinja variables are rendered before the they are loaded. In this case we do not care about the jinja variables since this exercise really only cares about the metadata of the file, but it prevents us from loading the file correctly, so just doing a dummy replace for now.

chamikaramj · 2025-11-18T18:49:28Z

yaml/scripts/generate_yaml_java_templates.py

+            param_code += "  @Validation.Required\n"
+
+        # default param
+        if 'default' in param:


Ditto regarding types.

see other comments, thanks

chamikaramj · 2025-11-18T18:51:33Z

yaml/scripts/generate_yaml_java_templates.py

+
+    # Convert each YAML file to a Java interface
+    try:
+        for yaml_path in yaml_files:


Is it possible that we might have other YAML files that are not intended to be pipelines ?

The idea is that this location would only contain the golden yaml blueprints.

chamikaramj

Thanks. LGTM other than one comment.

chamikaramj · 2025-11-25T00:03:38Z

yaml/scripts/generate_yaml_java_templates.py

+        print(f"Error running mvn spotless:apply: {e}", file=sys.stderr)
+        return e
+
+def generate_java_interface(yaml_path, java_path):


Is it possible to add some unit testing to cover these functions ?

derrickaw added 4 commits November 13, 2025 14:25

initial generate code

b339532

update yaml to jive with java template file

1fccd9b

add java template file

3b64df9

update generate file

3549765

pull-request-size bot added the size/L label Nov 13, 2025

derrickaw added addition New feature or request java Pull requests that update Java code improvement yaml Pull requests that update Yaml code labels Nov 13, 2025

add new workflow for yaml blueprint to java template

996004d

update script name update workflow debug fix step names checkout branch try again more debug update to fetch and checkout add permissions add token fix names etc

derrickaw force-pushed the addAutoYamlToJavaGeneration branch from a398dbd to 996004d Compare November 13, 2025 21:32

derrickaw added 10 commits November 13, 2025 21:34

add the resulting changes to the KafkaToBigQueryYaml.java file that w…

457ba94

…ould result from the new workflow

with mvn spotless

0d50633

remove mvn spotless

f660460

update template

148006f

update a different template :)

83437e8

add distribution for jdk

455936e

spotless apply on yaml template

893cea5

make spotless run only on yaml module

7797e73

add -B

b477955

add more comments

fdc8a8e

derrickaw marked this pull request as ready for review November 14, 2025 21:01

derrickaw requested review from chamikaramj, damccorm and tarun-google November 14, 2025 21:03

remove redundant workflow step since changed script to handle both di…

2ef9170

…rectory and file

damccorm reviewed Nov 17, 2025

View reviewed changes

derrickaw added 2 commits November 18, 2025 15:27

remove workflow for now - will revisit on next iteration

4d772af

add note abote spotless

e741603

add generic readme on how to add new templates

b007e38

tarun-google previously approved these changes Nov 18, 2025

View reviewed changes

tarun-google reviewed Nov 18, 2025

View reviewed changes

chamikaramj reviewed Nov 18, 2025

View reviewed changes

derrickaw added 2 commits November 19, 2025 14:14

update readme per comments

ded6680

update generate script per comments

437942b

derrickaw dismissed tarun-google’s stale review via 437942b November 19, 2025 14:19

derrickaw requested a review from chamikaramj November 19, 2025 16:44

chamikaramj reviewed Nov 25, 2025

View reviewed changes


		## Steps

		1. Add YAML Blueprint: Create the YAML blueprint file that defines the template's structure and parameters.



		def get_java_type(param_type):
		"""Maps a YAML parameter type to a Java type."""

Add auto yaml to java generation #2984

Are you sure you want to change the base?

Add auto yaml to java generation #2984

Uh oh!

Conversation

derrickaw commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

damccorm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derrickaw Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tarun-google left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derrickaw Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derrickaw Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derrickaw commented Nov 13, 2025 •

edited

Loading

codecov bot commented Nov 13, 2025 •

edited

Loading

derrickaw Nov 18, 2025 •

edited

Loading

derrickaw Nov 19, 2025 •

edited

Loading

derrickaw Nov 19, 2025 •

edited

Loading