Merge pull request #63 from shcherbak-ai/dev

build: v0.15.0

Author: SergiiShcherbak

CHANGELOG.md

@@ -5,6 +5,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 - **Refactor**: Code reorganization that doesn't change functionality but improves structure or maintainability
 
+## [0.15.0](https://github.com/shcherbak-ai/contextgem/releases/tag/v0.15.0) - 2025-08-14
+### Added
+- Auto-pricing for LLMs: enable via `auto_pricing=True` to automatically estimate costs using pydantic's `genai-prices`; optional `auto_pricing_refresh=True` refreshes cached price data at runtime.
+
+### Refactor
+- Public API made more consistent and stable: user-facing classes are now thin, well-documented facades over internal implementations. No behavior changes.
+- Internal reorganization for maintainability and future-proofing.
+
+### Docs
+- Added guidance for configuring auto-pricing for LLMs.
+
 ## [0.14.4](https://github.com/shcherbak-ai/contextgem/releases/tag/v0.14.4) - 2025-08-08
 ### Fixed
 - Suppressed noisy LiteLLM proxy missing-dependency error logs (prompting to install `litellm[proxy]`) emitted by `litellm>=1.75.2` during LLM API calls. ContextGem does not require LiteLLM proxy features. Suppression is scoped to LiteLLM loggers.

CONTRIBUTING.md

@@ -67,19 +67,24 @@ contextgem/
 │
 ├── contextgem/
 │   │
-│   ├── public/                   # 🎯 User-facing API (start here for new features)
-│   │   ├── concepts.py           #   - Concepts API
-│   │   ├── aspects.py            #   - Aspects API  
-│   │   ├── documents.py          #   - Document processing
-│   │   ├── pipelines.py          #   - Document data extraction pipelines
-│   │   ├── llms.py               #   - LLM extraction functionality
-│   │   └── ...                   #   - More public modules
+│   ├── internal/                 # 🔧 Core implementation (start here for new features)
+│   │   ├── base/                 #   - Core abstractions & business logic
+│   │   │   ├── concepts.py       #     - Internal concept implementations
+│   │   │   ├── aspects.py        #     - Internal aspect implementations  
+│   │   │   ├── documents.py      #     - Internal document processing
+│   │   │   ├── llms.py           #     - Internal LLM functionality
+│   │   │   └── ...               #     - More internal implementations
+│   │   ├── prompts/              #   - LLM prompt templates
+│   │   ├── typings/              #   - Type definitions
+│   │   └── ...                   #   - More internal modules
 │   │
-│   └── internal/                 # 🔧 Internal implementation
-│       ├── base/                 #   - Core abstractions
-│       ├── prompts/              #   - LLM prompt templates
-│       ├── typings/              #   - Type definitions
-│       └── ...                   #   - More internal modules
+│   └── public/                   # 🎯 User-facing API (thin facades exposing internals)
+│       ├── concepts.py           #   - Public concept facades
+│       ├── aspects.py            #   - Public aspect facades 
+│       ├── documents.py          #   - Public document facades
+│       ├── pipelines.py          #   - Public pipeline facades
+│       ├── llms.py               #   - Public LLM facades
+│       └── ...                   #   - More public modules
 │
 ├── tests/
 │   ├── cassettes/                # 📼 VCR recordings (auto-generated)
@@ -102,12 +107,12 @@ contextgem/
 ```
 
 **🎯 Quick Start for Your Contribution:**
-- **Adding new functionality?** → Start in `contextgem/public/`, often requires `internal/` changes too
+- **Adding new functionality?** → Implement in `contextgem/internal/` (core logic). Then expose via a thin public facade in `contextgem/public/` using the registry.
 - **Writing tests?** → Add to `tests/test_all.py::TestAll`  
 - **Updating docs?** → Edit files in `docs/source/` or `dev/`
 - **Fixing README?** → Edit `dev/readme.template.md`
 
-> **💡 Note:** New public features typically require supporting changes in internal modules (base classes, prompt templates, type definitions, etc.). Think of `public/` as the user interface and `internal/` as the engine that powers it.
+> **💡 Note:** Implement functionality in `internal/` (base classes, validation, serialization, typing). Use `public/` to expose thin, documented facades that inherit from internal classes and are registered with `@_expose_in_registry` decorator to ensure deserialization and instance creation utils return public types. Do not import public classes in internal modules; use the registry for type resolution and publicization.
 
 
 ---

NOTICE

@@ -25,6 +25,7 @@ This software includes the following third-party components:
 
 Core Dependencies:
 - aiolimiter: Rate limiting for asynchronous operations
+- genai-prices: LLM pricing data and utilities (by Pydantic) to automatically estimate costs
 - Jinja2: Templating engine
 - litellm: LLM interface library (this software uses only MIT-licensed portions of LiteLLM and does not utilize any components from the enterprise/ directory)
 - loguru: Logging utility

README.md

@@ -484,6 +484,7 @@ This project is automatically scanned for security vulnerabilities using multipl
 ContextGem relies on these excellent open-source packages:
 
 - [aiolimiter](https://github.com/mjpieters/aiolimiter): Powerful rate limiting for async operations
+- [genai-prices](https://github.com/pydantic/genai-prices): LLM pricing data and utilities (by Pydantic) to automatically estimate costs
 - [Jinja2](https://github.com/pallets/jinja): Fast, expressive, extensible templating engine used for prompt rendering
 - [litellm](https://github.com/BerriAI/litellm): Unified interface to multiple LLM providers with seamless provider switching
 - [loguru](https://github.com/Delgan/loguru): Simple yet powerful logging that enhances debugging and observability

contextgem/__init__.py

@@ -20,7 +20,7 @@
 ContextGem - Effortless LLM extraction from documents
 """
 
-__version__ = "0.14.4"
+__version__ = "0.15.0"
 __author__ = "Shcherbak AI AS"
 
 from contextgem.public import (
@@ -52,7 +52,7 @@
 )
 
 
-__all__ = [
+__all__ = (
     # Aspects
     "Aspect",
     # Concepts
@@ -90,4 +90,4 @@
     "JsonObjectClassStruct",
     # Converters
     "DocxConverter",
-]
+)

contextgem/internal/__init__.py

@@ -17,17 +17,29 @@
 #
 
 from contextgem.internal.base import (
-    _AssignedAspectsProcessor,
-    _AssignedConceptsProcessor,
-    _AssignedInstancesProcessor,
-    _Concept,
-    _ExtractedItem,
-    _ExtractedItemsAttributeProcessor,
-    _InstanceBase,
-    _MarkdownTextAttributesProcessor,
-    _ParasAndSentsBase,
-    _PostInitCollectorMixin,
-    _RefParasAndSentsAttrituteProcessor,
+    _COST_QUANT,
+    _LOCAL_MODEL_PROVIDERS,
+    _Aspect,
+    _BooleanConcept,
+    _DateConcept,
+    _Document,
+    _DocumentLLM,
+    _DocumentLLMGroup,
+    _DocumentPipeline,
+    _ExtractionPipeline,
+    _Image,
+    _JsonObjectClassStruct,
+    _JsonObjectConcept,
+    _JsonObjectExample,
+    _LabelConcept,
+    _LLMPricing,
+    _NumericalConcept,
+    _Paragraph,
+    _RatingConcept,
+    _RatingScale,
+    _Sentence,
+    _StringConcept,
+    _StringExample,
 )
 from contextgem.internal.converters import _DocxConverterBase, _DocxPackage
 from contextgem.internal.data_models import (
@@ -37,7 +49,12 @@
     _LLMUsage,
     _LLMUsageOutputContainer,
 )
-from contextgem.internal.decorators import _post_init_method, _timer_decorator
+from contextgem.internal.decorators import (
+    _disable_direct_initialization,
+    _expose_in_registry,
+    _post_init_method,
+    _timer_decorator,
+)
 from contextgem.internal.exceptions import (
     DocxConverterError,
     LLMAPIError,
@@ -111,19 +128,31 @@
 )
 
 
-__all__ = [
+__all__ = (
     # Base
-    "_InstanceBase",
-    "_AssignedAspectsProcessor",
-    "_AssignedConceptsProcessor",
-    "_AssignedInstancesProcessor",
-    "_ExtractedItemsAttributeProcessor",
-    "_RefParasAndSentsAttrituteProcessor",
-    "_PostInitCollectorMixin",
-    "_Concept",
-    "_ExtractedItem",
-    "_ParasAndSentsBase",
-    "_MarkdownTextAttributesProcessor",
+    "_COST_QUANT",
+    "_LOCAL_MODEL_PROVIDERS",
+    "_Aspect",
+    "_BooleanConcept",
+    "_DateConcept",
+    "_Document",
+    "_DocumentLLM",
+    "_DocumentLLMGroup",
+    "_DocumentPipeline",
+    "_ExtractionPipeline",
+    "_Image",
+    "_JsonObjectClassStruct",
+    "_JsonObjectConcept",
+    "_JsonObjectExample",
+    "_LabelConcept",
+    "_LLMPricing",
+    "_NumericalConcept",
+    "_Paragraph",
+    "_RatingConcept",
+    "_RatingScale",
+    "_Sentence",
+    "_StringConcept",
+    "_StringExample",
     # LLM output structs
     "_get_aspect_extraction_output_struct",
     "_get_concept_extraction_output_struct",
@@ -166,6 +195,8 @@
     # Decorators
     "_post_init_method",
     "_timer_decorator",
+    "_disable_direct_initialization",
+    "_expose_in_registry",
     # Extracted items
     "_StringItem",
     "_IntegerItem",
@@ -204,4 +235,4 @@
     "LLMExtractionError",
     "LLMAPIError",
     "DocxConverterError",
-]
+)

contextgem/internal/base/__init__.py

@@ -16,38 +16,64 @@
 # limitations under the License.
 #
 
-from contextgem.internal.base.attrs import (
-    _AssignedAspectsProcessor,
-    _AssignedConceptsProcessor,
-    _AssignedInstancesProcessor,
-    _ExtractedItemsAttributeProcessor,
-    _RefParasAndSentsAttrituteProcessor,
+
+from contextgem.internal.base.aspects import _Aspect
+from contextgem.internal.base.concepts import (
+    _BooleanConcept,
+    _DateConcept,
+    _JsonObjectConcept,
+    _LabelConcept,
+    _NumericalConcept,
+    _RatingConcept,
+    _StringConcept,
+)
+from contextgem.internal.base.data_models import _LLMPricing, _RatingScale
+from contextgem.internal.base.documents import _Document
+from contextgem.internal.base.examples import _JsonObjectExample, _StringExample
+from contextgem.internal.base.images import _Image
+from contextgem.internal.base.llms import (
+    _COST_QUANT,
+    _LOCAL_MODEL_PROVIDERS,
+    _DocumentLLM,
+    _DocumentLLMGroup,
 )
-from contextgem.internal.base.concepts import _Concept
-from contextgem.internal.base.instances import _InstanceBase
-from contextgem.internal.base.items import _ExtractedItem
-from contextgem.internal.base.md_text import _MarkdownTextAttributesProcessor
-from contextgem.internal.base.mixins import _PostInitCollectorMixin
-from contextgem.internal.base.paras_and_sents import _ParasAndSentsBase
+from contextgem.internal.base.paras_and_sents import _Paragraph, _Sentence
+from contextgem.internal.base.pipelines import _DocumentPipeline, _ExtractionPipeline
+from contextgem.internal.base.utils import _JsonObjectClassStruct
 
 
-__all__ = [
-    # Instances
-    "_InstanceBase",
-    # Attrs processors
-    "_AssignedAspectsProcessor",
-    "_AssignedConceptsProcessor",
-    "_AssignedInstancesProcessor",
-    "_ExtractedItemsAttributeProcessor",
-    "_RefParasAndSentsAttrituteProcessor",
-    # Mixins
-    "_PostInitCollectorMixin",
+__all__ = (
+    # Aspects
+    "_Aspect",
     # Concepts
-    "_Concept",
-    # Extracted items
-    "_ExtractedItem",
+    "_BooleanConcept",
+    "_DateConcept",
+    "_JsonObjectConcept",
+    "_LabelConcept",
+    "_NumericalConcept",
+    "_RatingConcept",
+    "_StringConcept",
+    # Data models (base)
+    "_LLMPricing",
+    "_RatingScale",
+    # Documents
+    "_Document",
+    # Examples
+    "_JsonObjectExample",
+    "_StringExample",
+    # Images
+    "_Image",
+    # LLMs
+    "_COST_QUANT",
+    "_LOCAL_MODEL_PROVIDERS",
+    "_DocumentLLM",
+    "_DocumentLLMGroup",
     # Paragraphs and sentences
-    "_ParasAndSentsBase",
-    # Markdown text
-    "_MarkdownTextAttributesProcessor",
-]
+    "_Paragraph",
+    "_Sentence",
+    # Pipelines
+    "_DocumentPipeline",
+    "_ExtractionPipeline",
+    # Utils (base)
+    "_JsonObjectClassStruct",
+)