Skip to content

AJDA-214: Update ODBC driver #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 33 commits into
base: master
Choose a base branch
from
Open

AJDA-214: Update ODBC driver #44

wants to merge 33 commits into from

Conversation

MiroCillik
Copy link
Member

@MiroCillik MiroCillik commented May 20, 2025

https://keboola.atlassian.net/browse/AJDA-214

Story

The extractor was failing on segmentation fault --> updated driver (use Debian one, not RPM)
The new driver was throwing exception on getting primary keys --> needed to update db-extractor-adapter keboola/db-extractor-adapter#35
The I was experimenting with settings for odbc_longreadlen, suggested by AI, which helped when I put large enough number there :)
The last problem was corrupted utf-8 character, so I put utf8 sanitization into the extractor itself.

Changes

  • updated ODBC driver to clouderahiveodbc_2.8.2.1002-2
  • extend OdbcQueryResult from db-extractor-adapter and override the fetch() method to set odbc_longreadlen to 32MB and some other memory optimizations
  • add log record at the end of extraction
  • sanitize broken utf8 chars in HiveResultWriter class

@MiroCillik MiroCillik changed the title tmp: odbc verbose logging Update ODBC driver Jun 2, 2025
…es fetch() method - mainly to set the odbc_longreadlen property
Comment on lines +30 to +32
$parameters['DefaultStringColumnLength'] = '16777216';
$parameters['DefaultVarcharColumnLength'] = '16777216';
$parameters['BinaryColumnLength'] = '16777216';
Copy link
Member Author

@MiroCillik MiroCillik Jun 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timhle si nejsem jistej, jestli to ma opravdu vlyv, ale funguje to takhle, uz to tak necham.
Kdyby se objevil problem, tak to muzem postelovat

@MiroCillik MiroCillik requested a review from ondrajodas June 2, 2025 13:51
@MiroCillik
Copy link
Member Author

Tady je videt, ze to funguje, puvodni konfigurace i s procesorama a PKs (verze 14):
Screenshot 2025-06-02 at 15 53 52

A ze i data jsou cela, neni tam zadny null, nebo nekompletni XML:
Screenshot 2025-06-02 at 12 46 54
Screenshot 2025-06-02 at 12 46 45

A jeste potrebuji skusit verzi 15, kde jsou upravy OdbcQueryResult presunuty z db-extractor-adapter sem do tridy HiveOdbcQueryResult

@MiroCillik MiroCillik changed the title Update ODBC driver AJDA-214: Update ODBC driver Jun 3, 2025
@MiroCillik
Copy link
Member Author

Vali to i s verzi "15":

Screenshot 2025-06-03 at 8 56 05 Screenshot 2025-06-03 at 8 55 54 Screenshot 2025-06-03 at 8 55 47

Copy link
Contributor

@ondrajodas ondrajodas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jinak dobrý, ale ty datadir testy by se měly pouštět

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants