Skip to content

tool local-ingest fails to ingest from stdin #5772

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tandr opened this issue May 20, 2025 · 0 comments
Open

tool local-ingest fails to ingest from stdin #5772

tandr opened this issue May 20, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@tandr
Copy link

tandr commented May 20, 2025

Describe the bug
local-ingest fails to ingest from stdin

$ cat > xaa <<EOF
{"_raw":"[email protected] Access denied from 127.0.0.2","_raw_hash2":-8546123884480137573,"email":"[email protected]", "ip":["127.0.0.2"]}
{"_raw":"[email protected] Login successful from 127.0.0.1",_raw_hash2":-3287367624800626342,"email":"[email protected]","ip":["127.0.0.1"]}
EOF

$ cat xaa |  quickwit tool local-ingest --index col5
❯ Ingesting documents locally...

---------------------------------------------------
 Connectivity checklist 
 ✔ metastore storage
 ✔ metastore
 ✔ index storage
 ✔ _ingest-cli-source

2025-05-20T17:20:41.475Z ERROR quickwit_indexing::actors::indexing_pipeline: error while spawning indexing pipeline, retrying after some time error=unknown source type `Stdin` (available source types are ingest API v2, ingest API v1, file, void, vec) retry_count=0 retry_delay=2s
2025-05-20T17:20:43.476Z ERROR quickwit_indexing::actors::indexing_pipeline: error while spawning indexing pipeline, retrying after some time error=unknown source type `Stdin` (available source types are ingest API v2, ingest API v1, file, void, vec) retry_count=1 retry_delay=4s
^C

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

  1. create an index
  2. create a jsonl file
  3. try to execute cat JSONLFILE | quickwit tool local-ingest --index INDEX

Expected behavior
A clear and concise description of what you expected to happen.

Configuration:
Please provide:

  1. Output of quickwit --version
$ quickwit --version
Quickwit 0.8.0 (x86_64-unknown-linux-gnu unknown unknown)

$ git rev-parse HEAD
6ec11c555aaf05912e1c7a394ad6ace48d095696
  1. The index_config.yaml
version: 0.7

index_id: col5

doc_mapping:
  field_mappings:
    - name: _id
      type: text
      tokenizer: raw
      stored: true
    - name: _raw
      type: text
      record: position
      stored: false
    - name: _raw_hash2
      type: i64
      fast: true
    - name: username
      type: array<text>
      record: position
    - name: ip
      type: array<ip>
      fast: true
    - name: domain
      type: array<text>
      fast: true
      record: position

 #
  dynamic_mapping:
    tokenizer: default
#    type: array<text>
    stored: true
    indexed: true

search_settings:
  default_search_fields: [_raw]

indexing_settings:
  commit_timeout_secs: 50
  1. The config file
version: 0.8
data_dir: /home/me/dev/quickwit/qwdata

indexer:
  enable_otlp_endpoint: ${QW_ENABLE_OTLP_ENDPOINT:-false}

ingest_api:
  max_queue_memory_usage: 8GiB
  max_queue_disk_usage: 40GiB
  content_length_limit: 1GiB
  shard_burst_limit: 2GiB
# i wish I could put this unlimited
  shard_throughput_limit: 20MiB

jaeger:
  enable_endpoint: ${QW_ENABLE_JAEGER_ENDPOINT:-false}

@tandr tandr added the bug Something isn't working label May 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant