POC "Fairness implemented" and "Fairness evaluated" conclusions #1

NicolasLacroix · 2025-08-15T18:49:16Z

No description provided.

…eadability

…ndencies used in the steps

…ub.com/jpipe-mcscert/jpipe-examples into llm_fates/fairness_benchmarks_datasets

github-actions · 2025-08-15T19:05:43Z

Justification process failed!

Download Diagram Artifact

Runner Output


ERROR - validate():317 - 2025-08-15 19:05:41,430 - [EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'multilingual_benchmark_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'multilingual_benchmark_is_present' is correctly producing at least one variable.

[EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'training_code_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'training_code_is_present' is correctly producing at least one variable.

[EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'bbq_benchmark_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'bbq_benchmark_is_present' is correctly producing at least one variable.

[EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'multilingual_dataset_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'multilingual_dataset_is_present' is correctly producing at least one variable.

[EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'evaluation_code_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'evaluation_code_is_present' is correctly producing at least one variable.

github-actions · 2025-08-15T19:06:31Z

Justification process failed!

No Diagram artifact was generated - please check the logs for details.

github-actions · 2025-08-15T19:08:33Z

Justification process failed!

No Diagram artifact was generated - please check the logs for details.

github-actions · 2025-08-15T19:19:12Z

Justification process failed!

Download Diagram Artifact

Runner Output


ERROR - validate():317 - 2025-08-15 19:19:10,254 - [EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'multilingual_benchmark_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'multilingual_benchmark_is_present' is correctly producing at least one variable.

[EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'training_code_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'training_code_is_present' is correctly producing at least one variable.

[EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'bbq_benchmark_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'bbq_benchmark_is_present' is correctly producing at least one variable.

[EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'multilingual_dataset_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'multilingual_dataset_is_present' is correctly producing at least one variable.

[EvidenceDependencyValidator]
Pipeline validation error: evidence node does not produce any variables.
  • Evidence: 'evaluation_code_is_present'
  • Problem: This evidence does not produce any output variables.
  • Impact: Strategies connected above this evidence will not receive required inputs.
  • Recommendation: Ensure that the function associated with evidence 'evaluation_code_is_present' is correctly producing at least one variable.

github-actions · 2025-08-15T19:25:19Z

Justification process failed!

Download Diagram Artifact

Runner Output

              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/jpipe_runner/runtime.py", line 40, in __init__
    self.load_files(libraries or [])
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/jpipe_runner/runtime.py", line 54, in load_files
    self._import_file(file_path)
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/jpipe_runner/runtime.py", line 73, in _import_file
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/runner/work/jpipe-examples/jpipe-examples/llm_FATES/jpipe-libs/steps.py", line 4, in <module>
    import httpx
ModuleNotFoundError: No module named 'httpx'

github-actions · 2025-08-15T19:30:18Z

Justification process failed!

Download Diagram Artifact

Runner Output

              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/jpipe_runner/runtime.py", line 40, in __init__
    self.load_files(libraries or [])
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/jpipe_runner/runtime.py", line 54, in load_files
    self._import_file(file_path)
  File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/jpipe_runner/runtime.py", line 73, in _import_file
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/runner/work/jpipe-examples/jpipe-examples/llm_FATES/jpipe-libs/steps.py", line 4, in <module>
    import httpx
ModuleNotFoundError: No module named 'httpx'

github-actions · 2025-08-15T19:38:50Z

Justification process failed!

Download Diagram Artifact

Runner Output


WARNING - pretty_display():172 - 2025-08-15 19:38:47,285 - FileNotFoundError: [Errno 2] No such file or directory: '/mnt/c/Users/bapti/OneDrive/Documents/Travail/CDD/MCMASTER_UNIVERSITY/Project/jpipe-examples/llm_FATES/resources/eval/smollm3_base.txt'
WARNING - pretty_display():172 - 2025-08-15 19:38:47,285 - FunctionException: 
Function 'bbq_benchmark_is_present' returned False, indicating failure.
  - The function associated with node 'BBQ_BM' (label: 'BBQ benchmark is present') executed but did not pass its check.
  - Please review the implementation and input data for this function.
  - Returned value: False
  - The function must return True to indicate a successful check.
WARNING - pretty_display():172 - 2025-08-15 19:38:47,286 - FileNotFoundError: [Errno 2] No such file or directory: '/mnt/c/Users/bapti/OneDrive/Documents/Travail/CDD/MCMASTER_UNIVERSITY/Project/jpipe-examples/llm_FATES/resources/pretraining/stage1_8T.yaml'

github-actions · 2025-08-15T19:42:33Z

Justification process failed!

Download Diagram Artifact

Runner Output


WARNING - pretty_display():172 - 2025-08-15 19:42:31,544 - FileNotFoundError: [Errno 2] No such file or directory: 'resources/eval/smollm3_base.txt'
WARNING - pretty_display():172 - 2025-08-15 19:42:31,544 - FunctionException: 
Function 'bbq_benchmark_is_present' returned False, indicating failure.
  - The function associated with node 'BBQ_BM' (label: 'BBQ benchmark is present') executed but did not pass its check.
  - Please review the implementation and input data for this function.
  - Returned value: False
  - The function must return True to indicate a successful check.
WARNING - pretty_display():172 - 2025-08-15 19:42:31,544 - FileNotFoundError: [Errno 2] No such file or directory: 'resources/pretraining/stage1_8T.yaml'

github-actions · 2025-08-15T19:46:32Z

Justification process failed!

Download Diagram Artifact

Runner Output


WARNING - inject_arguments():97 - 2025-08-15 19:46:30,968 - Consumed variable 'dataset_configuration' is declared but not used in function 'training_model_using_the_multilingual_dataset'.
WARNING - inject_arguments():97 - 2025-08-15 19:46:30,969 - Consumed variable 'training_code' is declared but not used in function 'training_model_using_the_multilingual_dataset'.
WARNING - inject_arguments():97 - 2025-08-15 19:46:30,969 - Consumed variable 'eval_configuration' is declared but not used in function 'evaluating_model_using_the_multilingual_benchmark'.
WARNING - inject_arguments():97 - 2025-08-15 19:46:30,969 - Consumed variable 'evaluation_procedure' is declared but not used in function 'evaluating_model_using_the_multilingual_benchmark'.
WARNING - pretty_display():172 - 2025-08-15 19:46:30,970 - FunctionException: 
Function 'bbq_benchmark_is_present' returned False, indicating failure.
  - The function associated with node 'BBQ_BM' (label: 'BBQ benchmark is present') executed but did not pass its check.
  - Please review the implementation and input data for this function.
  - Returned value: False
  - The function must return True to indicate a successful check.

… pr comments

github-actions · 2025-08-18T14:19:50Z

Justification process failed!

Download Diagram Artifact

Runner Output


WARNING - inject_arguments():97 - 2025-08-18 14:19:47,561 - Consumed variable 'dataset_configuration' is declared but not used in function 'training_model_using_the_multilingual_dataset'.
WARNING - inject_arguments():97 - 2025-08-18 14:19:47,562 - Consumed variable 'training_code' is declared but not used in function 'training_model_using_the_multilingual_dataset'.
WARNING - inject_arguments():97 - 2025-08-18 14:19:47,562 - Consumed variable 'eval_configuration' is declared but not used in function 'evaluating_model_using_the_multilingual_benchmark'.
WARNING - inject_arguments():97 - 2025-08-18 14:19:47,562 - Consumed variable 'evaluation_procedure' is declared but not used in function 'evaluating_model_using_the_multilingual_benchmark'.
WARNING - pretty_display():172 - 2025-08-18 14:19:47,563 - FunctionException: 
Function 'bbq_benchmark_is_present' returned False, indicating failure.
  - The function associated with node 'BBQ_BM' (label: 'BBQ benchmark is present') executed but did not pass its check.
  - Please review the implementation and input data for this function.
  - Returned value: False
  - The function must return True to indicate a successful check.

github-actions · 2025-08-21T13:57:55Z

Justification process failed!

Download Diagram Artifact

Runner Output


WARNING - inject_arguments():97 - 2025-08-21 13:57:52,034 - Consumed variable 'dataset_configuration' is declared but not used in function 'training_model_using_the_multilingual_dataset'.
WARNING - inject_arguments():97 - 2025-08-21 13:57:52,034 - Consumed variable 'training_code' is declared but not used in function 'training_model_using_the_multilingual_dataset'.
WARNING - inject_arguments():97 - 2025-08-21 13:57:52,035 - Consumed variable 'eval_configuration' is declared but not used in function 'evaluating_model_using_the_multilingual_benchmark'.
WARNING - inject_arguments():97 - 2025-08-21 13:57:52,035 - Consumed variable 'evaluation_procedure' is declared but not used in function 'evaluating_model_using_the_multilingual_benchmark'.
WARNING - pretty_display():172 - 2025-08-21 13:57:52,035 - FunctionException: 
Function 'bbq_benchmark_is_present' returned False, indicating failure.
  - The function associated with node 'BBQ_BM' (label: 'BBQ benchmark is present') executed but did not pass its check.
  - Please review the implementation and input data for this function.
  - Returned value: False
  - The function must return True to indicate a successful check.

github-actions · 2025-08-22T18:13:27Z

Justification process failed!

Download Diagram Artifact

Runner Output


WARNING - inject_arguments():97 - 2025-08-22 18:13:23,792 - Consumed variable 'dataset_configuration' is declared but not used in function 'training_model_using_the_multilingual_dataset'.
WARNING - inject_arguments():97 - 2025-08-22 18:13:23,792 - Consumed variable 'training_code' is declared but not used in function 'training_model_using_the_multilingual_dataset'.
WARNING - inject_arguments():97 - 2025-08-22 18:13:23,793 - Consumed variable 'eval_configuration' is declared but not used in function 'evaluating_model_using_the_multilingual_benchmark'.
WARNING - inject_arguments():97 - 2025-08-22 18:13:23,793 - Consumed variable 'evaluation_procedure' is declared but not used in function 'evaluating_model_using_the_multilingual_benchmark'.
WARNING - pretty_display():172 - 2025-08-22 18:13:23,793 - FunctionException: 
Function 'bbq_benchmark_is_present' returned False, indicating failure.
  - The function associated with node 'BBQ_BM' (label: 'BBQ benchmark is present') executed but did not pass its check.
  - Please review the implementation and input data for this function.
  - Returned value: False
  - The function must return True to indicate a successful check.

github-actions · 2025-08-22T19:25:09Z

Justification process failed!

Download Diagram Artifact

Runner Output


ERROR - validate():317 - 2025-08-22 19:25:03,872 - [MissingVariableValidator]
Pipeline validation error: missing variable.
  • Function 'multilingual_dataset_is_present' declares that it consumes variable 'expected_smollm3_model_name',
    but no producer for this variable is found in the pipeline,
    nor is it provided in the 'main' context.
  • To fix:
    - Ensure that some earlier function produces 'expected_smollm3_model_name', or
    - Provide '{var}' via config/context,
    so that 'multilingual_dataset_is_present' can consume it.

[MissingVariableValidator]
Pipeline validation error: missing variable.
  • Function 'multilingual_dataset_is_present' declares that it consumes variable 'expected_pretraining_languages',
    but no producer for this variable is found in the pipeline,
    nor is it provided in the 'main' context.
  • To fix:
    - Ensure that some earlier function produces 'expected_pretraining_languages', or
    - Provide '{var}' via config/context,
    so that 'multilingual_dataset_is_present' can consume it.

[MissingVariableValidator]
Pipeline validation error: missing variable.
  • Function 'multilingual_dataset_is_present' declares that it consumes variable 'resources_dir',
    but no producer for this variable is found in the pipeline,
    nor is it provided in the 'main' context.
  • To fix:
    - Ensure that some earlier function produces 'resources_dir', or
    - Provide '{var}' via config/context,
    so that 'multilingual_dataset_is_present' can consume it.

[MissingVariableValidator]
Pipeline validation error: missing variable.
  • Function 'evaluation_code_is_present' declares that it consumes variable 'expected_evaluation_readme_location',
    but no producer for this variable is found in the pipeline,
    nor is it provided in the 'main' context.
  • To fix:
    - Ensure that some earlier function produces 'expected_evaluation_readme_location', or
    - Provide '{var}' via config/context,
    so that 'evaluation_code_is_present' can consume it.

NicolasLacroix and others added 6 commits August 15, 2025 14:48

refactor(steps): Remove try-except blocks to improve runner results r…

48ab4e8

…eadability

test: debug steps

579a362

test: debug steps

d67a8aa

build(dependencies): Update requirements.txt file to include all depe…

55fbe59

…ndencies used in the steps

feat(workflow): changing uses to main release

a755954

Merge branch 'llm_fates/fairness_benchmarks_datasets' of https://gith…

c54f3a9

…ub.com/jpipe-mcscert/jpipe-examples into llm_fates/fairness_benchmarks_datasets

feat(workflow): changing uses to main release

bc3d870

test: debug steps

a93fff1

fix(workflow): fixing issue with library path

ff1a0cb

BaptisteLacroix added 2 commits August 15, 2025 15:26

fix(workflow): installing requirement.txt

6da38db

fix(workflow): adding system dependencies installation

68ceaf1

fix(workflow): specifying python path

23cc656

fix: remove static link

67f8019

fix(steps): Update resources path to llm_FATES/resources

a379718

jpipe-mcscert deleted a comment from github-actions bot Aug 18, 2025

feat: adding development branch jpipe-runner to get images shwoing on…

5cedcb8

… pr comments

Update workflow.yml

f7d6558

jpipe-mcscert deleted a comment from github-actions bot Aug 21, 2025

feat(gpt-oss): Add POC for gpt-oss's prepared framework justification

be85efd

BaptisteLacroix added 2 commits August 22, 2025 14:15

chore: update workflow file

a31e311

chore: pull & merge

0cc9997

POC "Fairness implemented" and "Fairness evaluated" conclusions #1

Are you sure you want to change the base?

POC "Fairness implemented" and "Fairness evaluated" conclusions #1

Uh oh!

Conversation

NicolasLacroix commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Aug 18, 2025

Uh oh!

github-actions bot commented Aug 21, 2025

Uh oh!

github-actions bot commented Aug 22, 2025

Uh oh!

github-actions bot commented Aug 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants