Skip to content

ice: Added support for deleting partitions #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 34 commits into from
Jul 9, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
e93e775
ice: Added support for positional delete file.
subkanthi May 12, 2025
7b976f6
ice: Added support to delete file with partition name and value.
subkanthi May 14, 2025
732804a
Merge branch 'master' of github.com:Altinity/ice into 7-ice-support-d…
subkanthi May 14, 2025
615b430
ice: Added support to delete file with partition name and value.
subkanthi May 15, 2025
40838b4
ice: Fix formatting errors.
subkanthi May 15, 2025
539d889
ice: Remove wildcard imports
subkanthi May 15, 2025
40a313a
ice: Fix formatting.
subkanthi May 15, 2025
25ac38e
Merge branch 'master' of github.com:Altinity/ice into 7-ice-support-d…
subkanthi May 21, 2025
d8ca2ee
ice: Renamed DeleteFile to DeletePartition, Added --dry-run option.
subkanthi May 21, 2025
7aa2075
ice: DeletePartition , add support to delete a list of values.
subkanthi May 24, 2025
84247ec
Added Integration test to start rest catalog server with shutdown
subkanthi Jun 5, 2025
0f16095
Merged conflicts from main.
subkanthi Jun 5, 2025
dd5ee85
Merged conflicts from main.
subkanthi Jun 5, 2025
00dae46
Merged conflicts from main.
subkanthi Jun 5, 2025
c14da54
Merged conflicts from main.
subkanthi Jun 5, 2025
13e0751
Fixed imports
subkanthi Jun 5, 2025
2e5795a
revert devbox.json changes.
subkanthi Jun 5, 2025
c720c83
Merge branch 'master' of github.com:Altinity/ice into 7-ice-support-d…
subkanthi Jun 17, 2025
4f93ceb
Added RESTCatalogIT integration test to start ice rest server and val…
subkanthi Jun 18, 2025
86c0e7f
Reverted back change to shutDown token.
subkanthi Jun 19, 2025
2396bc1
formatting changes.
subkanthi Jun 19, 2025
cabb5a4
Merge branch 'master' of github.com:Altinity/ice into 7-ice-support-d…
subkanthi Jun 19, 2025
7f7f1a8
Modified deletePartition to use namespace.tablename as argument. Modi…
subkanthi Jun 20, 2025
b75d4af
Added insert command to RESTCatalogIT
subkanthi Jun 21, 2025
ede4b82
Merge branch 'master' of github.com:Altinity/ice into 7-ice-support-d…
subkanthi Jun 28, 2025
3b4ae26
Addressed PR review comments.
subkanthi Jun 28, 2025
12ab306
Reverted back removal of private
subkanthi Jun 28, 2025
bce53f2
Removed yaml files added for Integration test.
subkanthi Jun 28, 2025
271c864
reverted back private change to Main
subkanthi Jun 28, 2025
a474855
Updated README.md with example
subkanthi Jun 29, 2025
9f5a8be
Removed wildcard imports, replaced Iterable with CloseableIterable an…
subkanthi Jun 30, 2025
344b0b3
Add try/catch to close CloseableIterable.
subkanthi Jul 1, 2025
23c389a
Remove logging line before throwing exception.
subkanthi Jul 2, 2025
ee0c915
Removed catch block.
subkanthi Jul 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions examples/scratch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,9 @@ ice create-table flowers.iris_no_copy --schema-from-parquet=file://iris.parquet
local-mc cp iris.parquet local/bucket1/flowers/iris_no_copy/
ice insert flowers.iris_no_copy --no-copy s3://bucket1/flowers/iris_no_copy/iris.parquet

# delete partition(By default --dry-run=true is enabled to print the list of partitions that will be deleted)
ice delete nyc.taxis --dry-run=false

# inspect
ice describe

Expand Down
39 changes: 39 additions & 0 deletions ice/src/main/java/com/altinity/ice/cli/Main.java
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import com.altinity.ice.cli.internal.cmd.Check;
import com.altinity.ice.cli.internal.cmd.CreateNamespace;
import com.altinity.ice.cli.internal.cmd.CreateTable;
import com.altinity.ice.cli.internal.cmd.Delete;
import com.altinity.ice.cli.internal.cmd.DeleteNamespace;
import com.altinity.ice.cli.internal.cmd.DeleteTable;
import com.altinity.ice.cli.internal.cmd.Describe;
Expand All @@ -27,6 +28,7 @@
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
import java.io.IOException;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
Expand Down Expand Up @@ -426,6 +428,43 @@ void deleteNamespace(
}
}

@CommandLine.Command(name = "delete", description = "Delete data from catalog.")
void delete(
@CommandLine.Parameters(
arity = "1",
paramLabel = "<name>",
description = "Table name (e.g. ns1.table1)")
String name,
@CommandLine.Option(
names = {"--partition"},
description =
"JSON array of partition filters: [{\"name\": \"vendorId\", \"values\": [5, 6]}]. "
+ "For timestamp columns, use ISO Datetime format YYYY-MM-ddTHH:mm:ss")
String partitionJson,
@CommandLine.Option(
names = "--dry-run",
description = "Log files that would be deleted without actually deleting them",
defaultValue = "true")
boolean dryRun)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be worth it to have it true be default considering the nature of the operation

throws IOException {
try (RESTCatalog catalog = loadCatalog(this.configFile())) {
List<PartitionFilter> partitions = new ArrayList<>();
if (partitionJson != null && !partitionJson.isEmpty()) {
ObjectMapper mapper = newObjectMapper();
PartitionFilter[] parts = mapper.readValue(partitionJson, PartitionFilter[].class);
partitions = Arrays.asList(parts);
}
TableIdentifier tableId = TableIdentifier.parse(name);

Delete.run(catalog, tableId, partitions, dryRun);
} catch (URISyntaxException e) {
throw new RuntimeException(e);
}
}

public record PartitionFilter(
@JsonProperty("name") String name, @JsonProperty("values") List<Object> values) {}

private RESTCatalog loadCatalog() throws IOException {
return loadCatalog(this.configFile());
}
Expand Down
81 changes: 81 additions & 0 deletions ice/src/main/java/com/altinity/ice/cli/internal/cmd/Delete.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
/*
* Copyright (c) 2025 Altinity Inc and/or its affiliates. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*/
package com.altinity.ice.cli.internal.cmd;

import java.io.IOException;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.List;
import org.apache.iceberg.DataFile;
import org.apache.iceberg.FileScanTask;
import org.apache.iceberg.RewriteFiles;
import org.apache.iceberg.Table;
import org.apache.iceberg.TableScan;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.io.CloseableIterable;
import org.apache.iceberg.rest.RESTCatalog;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public final class Delete {

private static final Logger logger = LoggerFactory.getLogger(Delete.class);

private Delete() {}

public static void run(
RESTCatalog catalog,
TableIdentifier tableId,
List<com.altinity.ice.cli.Main.PartitionFilter> partitions,
boolean dryRun)
throws IOException, URISyntaxException {

Table table = catalog.loadTable(tableId);
TableScan scan = table.newScan();
if (partitions != null && !partitions.isEmpty()) {
org.apache.iceberg.expressions.Expression expr = null;
for (com.altinity.ice.cli.Main.PartitionFilter pf : partitions) {
org.apache.iceberg.expressions.Expression e = null;
for (Object value : pf.values()) {
org.apache.iceberg.expressions.Expression valueExpr =
org.apache.iceberg.expressions.Expressions.equal(pf.name(), value);
e = (e == null) ? valueExpr : org.apache.iceberg.expressions.Expressions.or(e, valueExpr);
}
expr = (expr == null) ? e : org.apache.iceberg.expressions.Expressions.and(expr, e);
}
scan = scan.filter(expr);
}
List<DataFile> filesToDelete = new ArrayList<>();

try (CloseableIterable<FileScanTask> tasks = scan.planFiles()) {
for (FileScanTask task : tasks) {
filesToDelete.add(task.file());
}
}

if (!filesToDelete.isEmpty()) {
if (dryRun) {
logger.info("Dry run: The following files would be deleted:");
for (DataFile file : filesToDelete) {
logger.info(" {}", file.path());
}
} else {
RewriteFiles rewrite = table.newRewrite();
for (DataFile deleteFile : filesToDelete) {
rewrite.deleteFile(deleteFile);
}
rewrite.commit();
logger.info("Partition(s) deleted.");
}
} else {
logger.info("No files found for the partition(s).");
}
}
}