refactor: pdf extractor #18
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…imports - Updated langfuse version in pyproject.toml and poetry.lock files. - Modified import statements in langfuse_ragas_evaluator.py to reflect new package structure. - Adjusted langfuse_manager.py to use labels instead of is_active for prompt management. - Refactored langfuse_traced_chain.py to utilize the new CallbackHandler import. - Enhanced traced_chain.py to initialize langfuse client and update tracing logic.
- Introduced test suite for enhanced PDF extraction capabilities in `test_enhanced_pdfs.py`. - Created new test files for various PDF types including text-based, mixed content, and scanned documents. - Implemented detailed tests for PDFExtractor's classification, extraction, and linking functionalities in `test_pdf_extractorv2_new.py`. - Added quick functionality verification tests in `test_pdf_functionality.py` to ensure correct operation with real PDF files. - Established mock classes and fixtures to facilitate unit testing of PDF extraction methods.
- Added a new source for PyTorch and its related packages with CPU support in pyproject.toml. - Included additional dependencies: camelot-py, tabula, and easyocr. - Changed the import statement for PDFExtractor to use the new version (pdf_extractorv2) in dependency_container.py.
…imports - Updated langfuse dependency in pyproject.toml and poetry.lock files to version 3.0.0. - Changed import statement for DatasetClient in langfuse_ragas_evaluator.py to reflect new package structure. - Modified langfuse_manager.py to use labels instead of is_active for prompt management. - Updated tracing logic in langfuse_traced_chain.py to utilize the new CallbackHandler and langfuse client. - Refactored traced_chain.py to integrate langfuse client for better tracing capabilities.
…prehensive test suite for PDFExtractor class - Deleted outdated test files: test_pdf_extractorv2.py, test_pdf_extractorv2_new.py, and test_pdf_functionality.py. - Introduced a new comprehensive test suite for the PDFExtractor class, covering various functionalities including content extraction from different PDF types, error handling, and performance testing. - Added mock dependencies and fixtures to streamline testing processes. - Implemented tests for text extraction, table extraction, language detection, and related ID mapping. - Ensured compatibility with multiple PDF formats and validated metadata completeness in extracted content.
…t.py, ensuring comprehensive coverage and maintaining functionality. Removed old test file to streamline the testing structure.
…r improved readability and maintainability
…d consistency; enhance test suite with additional logging and assertions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces several updates across multiple libraries and modules, focusing on dependency upgrades, feature enhancements, and code refactoring. The most significant changes include upgrading dependencies, adding new sources and tools for text and table extraction, and improving PDF extraction logic with enhanced functionality.