Ozone in Presto Iceberg Connector - [Issue] Creates a file instead of a directory

[nishithakbhaskaran]

Hi folks,
I am connecting ozone using the iceberg connector from presto using the s3 gateway.
The create schema is failing as it is creating a file in ozone bucket instead of a directory. Hence I am getting below error when creating a table over the schema
Failed to write json to file: com.facebook.presto.iceberg.HdfsOutputFile@88839f48

Any thoughts on the above scenario? Thanks in advance!!

[errose28]

Some questions to help fill in the picture:

• What version of Ozone are you running?
• Are you accessing an FSO bucket through the S3 gateway?
• How did you determine that the FSO bucket contains a file where it should contain a directory?

Accessing FSO buckets through S3 gateway is expected to work, but there are some caveats because S3 and ofs (the Hadoop compatible interface) have fundamental differences. One of them is that S3 does not have any notion of a directory, so S3 gateway has to infer whether it should create a directory in the underlying FSO bucket by:

• The length of the file being zero AND
• The given key ending with a / character AND
• The config ozone.s3g.fso.directory.creation set to true to enable this behavior. True is the default value if the config is not specified.

(Code if you are interested)

The description sounds like HDDS-8437 but that has been fixed for a while.

If you are only accessing data through S3 gateway and do not need a hadoop compatible interface, you can also use an OBS bucket instead of an FSO bucket and not have to worry about the intricacies of converting between S3 and ofs protocols.