Echo Unmapped VID Investigation [VS-1671] #9202
Open
+1,105
−18
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mcovarr
force-pushed
the
vs_1671_vat_discrepancies
branch
from
d87e3e1 to
0b4f7f7
Compare
mcovarr
changed the title
There was a problem hiding this comment.
Pull Request Overview
This PR adds the investigation artifacts for Echo unmapped VIDs by introducing new scripts, workflows, prompts, documentation, and updated Docker setups to support extracting and merging JSON/TSV from gVCFs.
- Bumps the
variants_dockerimage tag and adds a newMergeJSONsWDL task to combine JSON shards. - Introduces Python scripts and Claude prompts for generating bcftools commands, filtering VCFs by VIDs, comparing VCFs, and processing gVCF variants.
- Updates Dockerfiles (base and final) to build and include GCS-enabled htslib/bcftools/vcftools and exposes the new workflow in
.dockstore.yml.
Reviewed Changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| scripts/variantstore/wdl/GvsUtils.wdl | Updated variants_docker tag; added MergeJSONs task to join JSON shards and emit TSV. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/read_gvcfs.prompt | New Claude prompt for reading gVCFs and reblocked gVCFs. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/generate_bcftools_commands.py | New script to parse VIDs and emit bcftools view commands. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/generate_bcftools_commands.prompt | New prompt describing VID-based bcftools command generation. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/filter_vcf_by_vids.py | New script to filter a VCF by matching VIDs. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/filter_vcf_by_vids.prompt | New prompt outlining the VCF filtering requirements. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/compare_vcfs.py | New script to compare input vs. left-aligned VCF variants. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/compare_vcfs.prompt | New prompt describing the VCF comparison and output format. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/SearchGVCFsForUnmappedVIDs.wdl | New WDL workflow to query BigQuery, read gVCFs, merge outputs, and upload content. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/README.md | Documentation for the pseudo-VID resolution procedure. |
| scripts/variantstore/variant-annotations-table/pseudo_vids_only_in_vat/ECHO_VID_INVESTIGATION.md | Detailed scientific-facing investigation write-up. |
| scripts/variantstore/scripts/reorder_gvcf_content_cols.py | New helper to reorder TSV columns before BigQuery load. |
| scripts/variantstore/scripts/process_gvcf_variants.py | New script to query gVCFs with bcftools and emit enriched JSON. |
| scripts/variantstore/scripts/build_base.Dockerfile | Updated to build and install htslib, bcftools, and vcftools; cleans up artifacts. |
| scripts/variantstore/scripts/Dockerfile | Updated final image to include htslib libraries and set LD_LIBRARY_PATH. |
| .dockstore.yml | Exposes the new vs_1671_vat_discrepancies branch and SearchGVCFsForUnmappedVIDs workflow. |
Comments suppressed due to low confidence (2)
scripts/variantstore/wdl/GvsUtils.wdl:130
- Typo in comment: 'handlful' should be 'handful'.
# there are a handlful of tasks that require the larger GNU libc-based `slim`.
scripts/variantstore/scripts/reorder_gvcf_content_cols.py:12
- The comment indicates JSON input, but this script reads a TSV file. Please update the comment to reflect that it processes TSV data.
# Load input JSON
| else: | ||
| search_range = 200 | ||
|
|
||
| start = position + 1 |
There was a problem hiding this comment.
The range calculation only searches downstream of the variant. To cover the intended ±range around the position, 'start' should be computed as 'position - search_range' (or adjusted per spec).
Suggested change
| start = position + 1 | |
| start = max(1, position - search_range) |
Copilot uses AI. Check for mistakes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Assorted artifacts of the Echo unmapped VID investigation.
Happy integration run with the updated Docker.
BQ tables
pseudo_vid_mappinglinks vids inpseudo_vids_only_in_vatto non-left aligned alleles that appear in the sites-only VCF used for making the VAT.pseudo_vid_sample_idlinks the non-left aligned alleles inpseudo_vid_mappingto samples containing these alleles.pseudo_vid_gvcf_contentincludes reblocked and unreblocked gVCF paths and file content for non-left aligned alleles on a per sample basis.Docs
README.mdECHO_VID_INVESTIGATION.mdTechnical artifacts