Open Source Python Data Formats Software

OCRmyPDF adds an optical character recognition (OCR) text layer to scanned PDF files, allowing them to be searched. PDF is the best format for storing and exchanging scanned documents. Unfortunately, PDFs can be difficult to modify. OCRmyPDF makes it easy to apply image processing and OCR (recognized, searchable text) to existing PDFs.

pandas-profiling generates profile reports from a pandas DataFrame. The pandas df.describe() function is handy yet a little basic for exploratory data analysis. pandas-profiling extends pandas DataFrame with df.profile_report(), which automatically generates a standardized univariate and multivariate report for data understanding. High correlation warnings, based on different correlation metrics (Spearman, Pearson, Kendall, Cramér’s V, Phik). Most common categories (uppercase, lowercase, separator), scripts (Latin, Cyrillic) and blocks (ASCII, Cyrilic). File sizes, creation dates, dimensions, indication of truncated images and existance of EXIF metadata. Mostly global details about the dataset (number of records, number of variables, overall missigness and duplicates, memory footprint). Comprehensive and automatic list of potential data quality issues (high correlation, skewness, uniformity, zeros, missing values, constant values, between others).

A Python application to add watermarks (text or image) to PDF files, converts them into image and back to PDF with options for OCR and compression.

API and command line utility, written in Python, for querying Elasticsearch exporting result as documents into a CSV file. The search can be done using logical operators or ranges, in combination or alone. The output can be limited to the desired attributes. Also ToT can insert the querying to a Pandas Dataframe or/and save its in a HDF5 container (under development).

Cathnet is developing the infrastructure for the Catholic Semantic Web. Technologies involved include, but are not limited to, XML, RDF, NLP, Zope, Plone and Plone products.

DocArray is a library for nested, unstructured, multimodal data in transit, including text, image, audio, video, 3D mesh, etc. It allows deep-learning engineers to efficiently process, embed, search, recommend, store, and transfer multimodal data with a Pythonic API. Door to multimodal world: super-expressive data structure for representing complicated/mixed/nested text, image, video, audio, 3D mesh data. The foundation data structure of Jina, CLIP-as-service, DALL·E Flow, DiscoArt etc. Data science powerhouse: greatly accelerate data scientists’ work on embedding, k-NN matching, querying, visualizing, evaluating via Torch/TensorFlow/ONNX/PaddlePaddle on CPU/GPU. Data in transit: optimized for network communication, ready-to-wire at anytime with fast and compressed serialization in Protobuf, bytes, base64, JSON, CSV, DataFrame. Perfect for streaming and out-of-memory data. One-stop k-NN: Unified and consistent API for mainstream vector databases.

Sofa is a CUDA-based reasoner

A large crowd-sourced dataset for developing natural language interfaces for relational databases. WikiSQL is the dataset released along with our work Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. Regarding tokenization and Stanza, when WikiSQL was written 3-years ago, it relied on Stanza, a CoreNLP python wrapper that has since been deprecated. If you'd still like to use the tokenizer, please use the docker image. We do not anticipate switching to the current Stanza as changes to the tokenizer would render the previous results not reproducible.

ftc is a python script for content-based file type classification based on an file extension and magic number database, and several computational intelligence algorithms.

C++ library for working with OWL ontologies

A Python AIML interpreter with non-compliant extensions. PyAIMLng is an interpreter for AIML (the Artificial Intelligence Markup Language), forked from Cort Stratton's PyAIML. PyAIMLng adds additional features which are not part of the AIML 1.0.1 specification in order to provide the bot master with a rich set of tools from which to build a more believable AIML bot.