Skip to content

Feature/spline 815 export import #1436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 19 commits into
base: release/0.8
Choose a base branch
from

Conversation

wajda
Copy link
Contributor

@wajda wajda commented Jul 11, 2025

Fixes #815

Adds CLI commands:

  • lineage-import
  • lineage-export

Adds Producer REST API endpoitns:

  • GET /execution-plans -- returns execution plan IDs
  • GET /execution-plans/{planID} -- returns the given ExecutionPlan (Producer API v1.1)
  • GET /execution-plans/{planID}/events -- returns a list of EvecutionEvents of the given ExecutionPlan

@wajda wajda requested a review from cerveada as a code owner July 11, 2025 23:00
@wajda wajda marked this pull request as draft July 11, 2025 23:00
@wajda
Copy link
Contributor Author

wajda commented Jul 11, 2025

Snyk checks have failed. 6 issues have been found so far.

Icon Severity Issues
Critical 0
High 2
Medium 1
Low 3

security/snyk check is complete. 6 issues have been found. (View Details)

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@wajda wajda linked an issue Jul 11, 2025 that may be closed by this pull request
@wajda wajda requested review from Copilot and removed request for cerveada July 11, 2025 23:20
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements import/export CLI commands and extends the Producer API with new endpoints for execution plan retrieval and events.

  • Adds lineage-import and lineage-export commands in the Admin CLI and integration tests.
  • Introduces repository methods (isDatabaseOk, fetchExecutionPlanIds, fetchExecutionPlan, fetchExecutionEvents) and updates the ExecutionProducerRepository interface.
  • Exposes new REST endpoints (GET /execution-plans, GET /execution-plans/{id}, GET /execution-plans/{id}/events) in ExecutionPlansController.

Reviewed Changes

Copilot reviewed 130 out of 130 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
ExecutionProducerRepositoryImpl.scala Added database health check and new fetch methods with AQL.
ExecutionProducerRepository.scala Extended interface for new repository methods.
ExecutionPlansController.scala Added GET endpoints and updated POST mapping for execution plans.
ExecutionPlanApiModelAssembler.scala New assembler for converting persistent models to API models.
integration-tests/AdminCLISpec.scala Integration tests for lineage-import and lineage-export.
Comments suppressed due to low confidence (4)

producer-rest-core/src/main/scala/za/co/absa/spline/producer/rest/controller/ExecutionPlansController.scala:194

  • [nitpick] Method name includes uppercase 'IDs', which is inconsistent with Scala camelCase conventions. Consider renaming to getExecutionPlanIds.
  def getExecutionPlanIDs: Future[Seq[UUID]] = {

producer-services/src/main/scala/za/co/absa/spline/producer/service/repo/ExecutionProducerRepositoryImpl.scala:140

  • Consider adding unit tests for fetchExecutionPlanIds to verify correct behavior when the database returns various result sets or errors.
  override def fetchExecutionPlanIds()(implicit ec: ExecutionContext): Future[Seq[UUID]] = {

producer-services/src/main/scala/za/co/absa/spline/producer/service/repo/ExecutionProducerRepositoryImpl.scala:56

  • NonFatal is not imported in this file, leading to a compilation error. Please add import scala.util.control.NonFatal.
      case NonFatal(_) => Future.successful(false)

producer-services/src/main/scala/za/co/absa/spline/producer/service/repo/ExecutionProducerRepositoryImpl.scala:141

  • [nitpick] This query loads all execution plan IDs into memory at once. For large datasets, consider adding pagination or streaming support to limit memory usage.
    db.queryAs[apiModel.ExecutionPlan.Id](

Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Export / Import lineage data
1 participant