issue：Exceeding 1 million images error

[SpiderJack0516]

What happened?

I just have 30,000 images, 140,000 targets.

operating version: https://github.com/visual-layer/fastdup/releases/download/v2.2_3.8/fastdup-2.2-cp38-cp38-linux_x86_64.whl

What did you expect to see?

No response

What version of fastdup were you runnning on?

2.2

What version of Python were you running on?

Python 3.8

Operating System

ubantu18.04

Reproduction steps

No response

Relevant log output

NoneType: None
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[3], line 2
      1 fd = fastdup.create(work_dir, image_dir)
----> 2 fd.run(annotations=coco_annotations, overwrite=True)

File ~/anaconda3/envs/fastdup380/lib/python3.8/site-packages/fastdup/engine.py:157, in Fastdup.run(self, input_dir, annotations, embeddings, subset, data_type, overwrite, model_path, distance, nearest_neighbors_k, threshold, outlier_percentile, num_threads, num_images, verbose, license, high_accuracy, cc_threshold, **kwargs)
    154     fastdup_func_params['model_path'] = model_path
    155 fastdup_func_params.update(kwargs)
--> 157 return super().run(annotations=annotations, input_dir=input_dir, subset=subset, data_type=data_type,
    158             overwrite=overwrite, embeddings=embeddings, **fastdup_func_params)

File ~/anaconda3/envs/fastdup380/lib/python3.8/site-packages/fastdup/sentry.py:144, in v1_sentry_handler.<locals>.inner_function(*args, **kwargs)
    142     else:
    143         fastdup_capture_exception(f"V1:{func.__name__}", ex)
--> 144     raise ex
    146 except Exception as ex:
    147     fastdup_capture_exception(f"V1:{func.__name__}", ex)

File ~/anaconda3/envs/fastdup380/lib/python3.8/site-packages/fastdup/sentry.py:135, in v1_sentry_handler.<locals>.inner_function(*args, **kwargs)
    133 try:
    134     start_time = time.time()
--> 135     ret = func(*args, **kwargs)
    136     fastdup_performance_capture(f"V1:{func.__name__}", start_time)
    137     return ret

File ~/anaconda3/envs/fastdup380/lib/python3.8/site-packages/fastdup/fastdup_controller.py:595, in FastdupController.run(self, input_dir, annotations, subset, embeddings, data_type, overwrite, print_summary, print_vl_datasets_ref, **fastdup_kwargs)
    593 # run fastdup - create embeddings
    594 if fastdup.run(self._set_fastdup_input(), work_dir=str(self._work_dir), **fastdup_kwargs) != 0:
--> 595     raise RuntimeError('Fastdup execution failed')
    597 #fastdup_convert_to_relpath(self._work_dir, self._filename_prefix)
    598 
    599 # post process - map fastdup-id to image (for bbox this is done in self._set_fastdup_input)
    600 if self._dtype == FD.IMG or self._run_mode == FD.MODE_CROP:

RuntimeError: Fastdup execution failed

Attach a screenshot [Optional]

No response

Contact Details [Optional]

No response

[SpiderJack0516]

sorry, I seem download wrong version, but I download other version from pypi and it would not install.

[SpiderJack0516]

The specific process is shown in the figure：

[dbickson]

Hi @SpiderJack0516
please send us the output of python3.9 -m pip debug --verbose
It seems the downloaded wheels are not matching the environment.
aarch64 - mean ARM cpu. Are you using intel cpu?

[SpiderJack0516]

@dbickson Here is the log, the highest version inside the output is cp39-cp39-manylinux_2_27_x86_64, but it doesn't match the version of the whl file.

[SpiderJack0516]

Here are the all logs.
fastdup-log.txt

[dbickson]

It seems you are using an older linux, conforming to 2_27, which is most likely ubuntu 18 or equivalent.
As Ubuntu 18 reached end of life we do not support it on our latest fastdup version. There is a legacy version in our github release page.
We recommend upgrading your machine to Ubuntu 20 or 22.

Regarding the 1M error, you need to upgrade fastdup to fix it.

[SpiderJack0516]

@dbickson Hi, the question been solved, I replaced ubantu 20.04, everything normal, thank you.