Skip to content

Add git to filesystem b 301 #350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Add git to filesystem b 301 #350

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
764ce42
support gitpythonfs fsspec implementation
deanja Feb 5, 2024
64d1d9d
dynamically create test git repo
Feb 6, 2024
ff9228a
Use dlt from feature branch
Feb 8, 2024
fbbec91
improve readability of test parameters
Feb 15, 2024
229f7a5
use distinct filesystem instances in pipeline threads
Feb 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
use distinct filesystem instances in pipeline threads 
- objects yielded by @dlt.resource can run in different threads
- note this does not address possible bug with filesystem instances being cached by fsspec itself.
  • Loading branch information
Your Name committed Feb 16, 2024
commit 229f7a5fac7dac86aad9a6685f783500df208d5f
10 changes: 7 additions & 3 deletions sources/filesystem/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,11 @@
import dlt
from dlt.common.typing import copy_sig, DictStrAny
from dlt.sources import DltResource
from dlt.sources.filesystem import FileItem, FileItemDict, fsspec_filesystem, glob_files
from dlt.sources.filesystem import FileItem, FileItemDict, glob_files
from dlt.sources.credentials import FileSystemCredentials
from dlt.common.storages.fsspec_filesystem import fsspec_from_config
from dlt.common.storages.configuration import FilesystemConfiguration


from .helpers import (
AbstractFileSystem,
Expand Down Expand Up @@ -90,14 +93,15 @@ def filesystem(
Returns:
Iterator[List[FileItem]]: The list of files.
"""
fs_config = FilesystemConfiguration(bucket_url, credentials, kwargs=kwargs)
if isinstance(credentials, AbstractFileSystem):
fs_client = credentials
else:
fs_client = fsspec_filesystem(bucket_url, credentials, kwargs=kwargs)[0]
fs_client = fsspec_from_config(fs_config)[0]

files_chunk: List[FileItem] = []
for file_model in glob_files(fs_client, bucket_url, file_glob):
file_dict = FileItemDict(file_model, fs_client)
file_dict = FileItemDict(file_model, fs_config)
if extract_content:
file_dict["file_content"] = file_dict.read_bytes()
files_chunk.append(file_dict) # type: ignore
Expand Down