RFC-100: Lance File Format support in Hudi

[rahil-c]

✅ Lance File Format Integration Tasks

See the following feature for more context: #14127

In regards to the following new feature for supporting unstructured data in Hudi via formats like Lance that are focused on AI/ML use cases. Here is the initial scope of what we are targeting(Note this list will continue to grow as we find get deeper within the integration, for now it aims to first support the Hudi Spark Client):

• Add base HoodieFileWriter for Lance with a Spark implementation, PR (merged in feature branch)
• Add base HoodieFileReader for Lance with a Spark implementation, PR
• Add SparkColumnarFileReader implementation for Spark Datasource Integration with Lance, PR
• Implement bulk insert/ insert / upsert / delete validation for COW, PR
• Implement bulk insert/ insert / upsert / delete validation for MOR(with lance base file and avro log file), PR
• Ensure that basic compaction, and clustering works for Hudi Tables with Lance PR (requires rebase from above)
• Ensure that schema evolution(add column) works for Hudi Tables with Lance PR
• FileWriter improvements: Support canWrite() and bloom filter comment1, comment2
• Validate that vector and blob content works PR
• Integrate Lance as a log file format
• Add predicate (filter) push-down
• Support ColumnarBatch vectorized reading

Will be raising changes on the following open source feature branch: https://github.com/apache/hudi/tree/feature-branch-rfc100-unstructured-data

Improving File Format Integration Points

Changes that are modifying existing hudi master (1.2.0) to allow for better integration with file formats in the future.

• Consolidate file writer side interfaces, PR