Evaluate tables via Google DocAI and Azure Doc Intelligence

[davidRozycki]

Hey,

Do you plan on evaluating the capabilities and metrics of the table extraction via Google DocumentAI or Azure Document Intelligence? Both provide a well-structured table object, albeit both are imperfect in terms of dealing with structure, merged cells, value-splitting (for text spanning multiple lines in a cell), etc.

These processors are more cost-effective when compared to LLM's usually, that's why these results would interest me from a cost/performance pov.

Thanks!

[mandalsouvik3333]

Yeah. We will add task-specific models also. But, after we are finished with evaluating quantized models of the open-source VLM models, and adding the image to markdown task. This is because most non-enterprise people use the quantized models because of memory constraints. You can share the task-specific models, services you would like to be evaluated. We can look into those when we start.

[davidRozycki]

Awesome, thanks!

What comes to mind are the following:

1. Azure Document Intelligence prebuilt-layout model: https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/layout?view=doc-intel-4.0.0&tabs=rest%2Csample-code#tables
2. Google Document AI: https://cloud.google.com/document-ai/docs/handle-response#forms_and_tables
3. Mistral DocAI (from our limited testing, the weakest so far in terms of accuracy): https://docs.mistral.ai/

Note: all of these processors also do OCR, so you might want to consider reviewing those capabilities as well.