fix(ingest/rest): out-of-date structured report being sent

[anshbansal]

• Fix a bug which was causing structured report to have outdated information. Fix is explained here https://www.loom.com/share/122e6a79cd574f7baae3003a982bd378 As this is a multi-threading bug in the underlying structures being used not sure how to add tests for this.

  • the fix is in metadata-ingestion/src/datahub/ingestion/api/sink.py and metadata-ingestion/src/datahub/ingestion/sink/datahub_rest.py which is quite small.

• Attempt to add an initial CLAUDE.MD so AI agents can work with the repository effectively

• Add datahub-mock-data source to be able to produce mock data which helped in reproducing this bug reliably. This will be used later on for more testing of ingestion sources. Heavily unit tested

[codecov]

Codecov Report

Attention: Patch coverage is 92.85714% with 11 lines in your changes missing coverage. Please review.

❌ Unsupported file format

Upload processing failed due to unsupported file format. Please review the parser error message:

Error deserializing json
Caused by:

expected value at line 1 column 1

For more help, visit our troubleshooting guide.

Files with missing lines	Patch %	Lines
...gestion/src/datahub/ingestion/sink/datahub_rest.py	25.00%	6 Missing ⚠️
...ub/ingestion/source/mock_data/datahub_mock_data.py	95.76%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

[sgomezvillamor]

Having mechanisms to generate mock data is definitely valuable.
What concerns me is coupling that logic directly with an ingestion source.

Would it make more sense to have a general-purpose DataHub source that ingests arbitrary MCE file(s)? That feels more flexible and reusable.

[anshbansal]

• Ingesting arbitrary MCE can already be done using datahub ingest mcps FILE_NAME
• There is already a datahub source which is for migration.

Using the ingestion framework for this allows us to test for various bugs in here and iterate on ingestion framework items which isn't really possible otherwise. This is a separate isolated source which is clearly marked as being there for testing. Unless there are very strong opinions would request this be allowed in.

[sgomezvillamor]

Yes, thats what I meant, that we should have something as generic as datahub ingest mcps FILE_NAME` in the ingestion framework.
While I think that would be a more generic approach, I'm approving to unblock.