Tags: nkdAgility/azure-devops-migration-tools
Tags
Implement ExportWorkItemMappingTool (#3031) ## Implementation of `ExportWorkItemMappingTool` This tool is for saving mappings for work items between source and target. Work item IDs are saved as simple dictionary _source ID → target ID_. We need such a mappings for some post processing after work item migrations. In our case, we use wiki in source system, which will also be migrated to target. In the wiki, there are many links to backlog items. These links are written in the wiki as simple `#{workItemId}` – for example `#1234567`. So if we will have mappings from source to target, we can just replace those IDs and the wiki will be working in target system. I believe, some users can also find it useful for some post-processing. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Records source→target work‑item ID mappings as items are created/identified and persists them at migration end (configurable export tool with Enabled, TargetFile, PreserveExisting and validation). * **Tests** * Adds a mock export tool and wires it into test DI setups for unit tests. * **Documentation** * Adds docs and JSON schema for the export tool and updates ChangeSetMappingFile documentation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Refactor GetFieldMappings for missing mappings (#3060) Refactored the `GetFieldMappings` method in `FieldMappingTool.cs` to improve code clarity and functionality. Introduced a `result` list to aggregate both global (`"*"` wildcard) and specific mappings for a given work item type. The method now returns a combined list of mappings, ensuring both global and specific mappings are considered when applicable. Co-authored-by: Mattias Nielsen <[email protected]>
Refactor GetFieldMappings for missing mappings (#3060) Refactored the `GetFieldMappings` method in `FieldMappingTool.cs` to improve code clarity and functionality. Introduced a `result` list to aggregate both global (`"*"` wildcard) and specific mappings for a given work item type. The method now returns a combined list of mappings, ensuring both global and specific mappings are considered when applicable. Co-authored-by: Mattias Nielsen <[email protected]>
Fix warnings (#3056) I do not like warnings, because I treat them as potential errors. :) So I fixed all warnings in the project, that could be fixed. Remaining warnings during build are now only those regarding mixing old and new .NET. ## Fixed warnings - `CS0168` The variable 'ex' is declared but never used - `MSTEST0001` Explicitly enable or disable tests parallelization (https://learn.microsoft.com/dotnet/core/testing/mstest-analyzers/mstest0001) - `MSTEST0017` Assertion arguments should be passed in the correct order. 'actual' and 'expected'/'notExpected' arguments have been swapped. - `MSTEST0037` Use proper asertion methods: - Use 'Assert.IsEmpty' instead of 'Assert.AreEqual' - Use 'Assert.HasCount' instead of 'Assert.AreEqual' - Use 'Assert.IsTrue' instead of 'Assert.AreEqual' - Use 'Assert.AreEqual' instead of 'Assert.IsTrue' - `MSTEST0044` 'DataTestMethod' is obsolete. Use 'TestMethod' instead. - `MSTEST0056` Use the 'DisplayName' property instead of passing a string argument to TestMethodAttribute - `NU1504` Duplicate 'PackageReference' items found. Remove the duplicate items or use the Update functionality to ensure a consistent restore behavior. The duplicate 'PackageReference' items are: Newtonsoft.Json , Newtonsoft.Json. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Tests** * Enabled parallel test execution at class level across test suites. * Updated test assertions for consistency and standardisation. * **Chores** * Removed unused JSON serialisation package dependencies. * Cleaned up imports and improved code formatting throughout tests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Update TfsWorkItemTypeValidatorTool documentation (#3055) Update documentation of `TfsWorkItemTypeValidator` [based on this issue](#2903 (comment)). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Renamed configuration key to ExcludeWorkItemTypes and replaced per-field mappings with per-work-item-type excluded-field lists (ExcludeSourceFields). * Added ExcludeDefaultWorkItemTypes and ExcludeWorkItemTypes entries to control which types are skipped. * **Documentation** * Updated docs and schema defaults to describe the new ExcludeSourceFields structure, wildcard support and updated default values. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix: improve EmbededImagesRepairToolBase.GetImageFormat to support ad… …ditional image header variants (#3049) ### Summary This PR fixes an issue where `EmbededImagesRepairToolBase.GetImageFormat` could not correctly identify certain image formats due to incomplete header checks. ### Changes - Added support for additional JPEG APP markers (ICC profile, Photoshop IRB, Adobe). - Improved detection for PNG and TIFF by checking full signatures. ### Why Previously, some valid images were misclassified as Authentication errors or not detected at all. This update ensures broader and more accurate format detection. (Might still have gaps). ### Testing - Added unit tests for JPEG variants (APP2, APP13, APP14) and other formats. - Verified detection works for PNG, TIFF, BMP, GIF, both for "fake headers" and real test assets. Closes #3048 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added SVG image format support. * **Bug Fixes** * Improved detection and validation for BMP, GIF, PNG, TIFF and JPEG, with more robust handling of short or null inputs. * **Tests** * Added comprehensive tests for image format detection and included image assets for test runs. * **Chores** * Updated gitignore to exclude development container files. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fix measuring duration of processor and command (#3040) I noticed this when i had my migration run for a long time and at the end I wanted to know, how long it really took. The migration duration was zero. ## Processor Although the log outputs `Duration` of the processor activity, this is always zero. Its because `Duration` is not dynamic – it does not return real current duration. Its value is calculated and updated only when `Stop()` is called. ## Command base Basically the same problem as in `Processor`, but from the other side. The activity is stopped immediately after it is started, so it reports zero duration at the end. Even when the telemetry is disabled, there is no need to not measure duration of the command. Because even wihout telemetry, we want to see duration in the log outputs. And `Stop()` basically does nothing just updates duration. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Refactor** * Enhanced activity lifecycle management within telemetry and processing workflows. * Removed unused code dependencies. * **Style** * Minor formatting improvements. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Refactor tests and update using directives (#3035) Updated `using` directives in `FieldCalculationMapTests.cs` to include additional namespaces and consolidate them into a single line. Replaced `Assert.ThrowsException` with `Assert.Throws` in two test methods for improved consistency and readability. Performed minor formatting adjustments, including closing brace cleanup. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Chores** * Updated multiple dependency packages to their latest versions for improved compatibility and security. * **Tests** * Updated testing framework to the latest version, ensuring compatibility with current development standards. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fix StringManipulatorTool to support Unicode characters and strip emo… …jis for SOAP compatibility (#3016) ## Problem The `StringManipulatorTool`'s default regex pattern `[^( -~)\n\r\t]+` removes all non-ASCII characters, which breaks international migrations containing accented letters, Cyrillic, Chinese characters, and other Unicode content. For example, with the current implementation: - `"Café résumé"` becomes `"Caf rsum"` - `"Привет мир"` becomes `""` (empty) - `"你好世界"` becomes `""` (empty) Additionally, emojis can cause errors when data passes through SOAP interfaces and need to be stripped while preserving other Unicode symbols. This makes the tool unsuitable for any non-English Azure DevOps migrations. ## Solution Updated the default regex pattern from `[^( -~)\n\r\t]+` to `[\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F]|[\uD800-\uDBFF][\uDC00-\uDFFF]|\uFE0F` which: ✅ **Preserves Unicode content**: International characters are maintained ✅ **Strips emojis for SOAP compatibility**: Removes emoji surrogate pairs and variation selectors ✅ **Maintains security**: Still removes harmful control characters ✅ **Preserves formatting**: Keeps newlines (`\x0A`) and tabs (`\x09`) ✅ **Backward compatible**: All existing functionality preserved The new pattern specifically targets problematic content while preserving legitimate Unicode: - `\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F`: Control characters - `[\uD800-\uDBFF][\uDC00-\uDFFF]`: Emoji surrogate pairs (😀🔥💻🇺🇸) - `\uFE0F`: Variation selector for emoji presentation ## Testing - Added 27 comprehensive test cases covering Unicode and emoji scenarios - All 41 existing tests continue to pass - Verified no security vulnerabilities introduced - Demonstration of the fix: ``` Original Old Pattern New Pattern ---------------------------------------------------------------------- Héllo Wørld Hllo Wrld Héllo Wørld Привет мир Привет мир 你好世界 你好世界 Café résumé Caf rsum Café résumé Hello 😀 World Hll Wrld Hello World Math ∑ Symbol Mth Symbl Math ∑ Symbol Test\u0001Control TestControl TestControl ``` ## Documentation Updated the StringManipulatorTool documentation to: - Show examples with the new Unicode-friendly pattern - Provide specific emoji stripping examples for SOAP compatibility - Provide the legacy ASCII-only pattern for environments that require it - Clarify use cases for international migration scenarios This change enables the Azure DevOps Migration Tools to properly handle international content while maintaining SOAP compatibility and the same level of data safety and security. Fixes the issue raised in the GitHub discussion where users noted that non-ASCII character support was not available by default. Fixes #3006 <!-- START COPILOT CODING AGENT SUFFIX --> <details> <summary>Original prompt</summary> > > ---- > > *This section details on the original issue you should resolve* > > <issue_title>[Bug]: support for non-ASCII characters is not supported by default</issue_title> > <issue_description>By default "StringManipulatorTool" is enabled, but the regex pattern in AddDefaultManipulator() "[^( -~)\n\r\t]+" removes all non-ASCII characters, which is a problem for any non-english migrations. Could the pattern be updated to allow for at least unicode letters and numbers? > > Perhaps something like `[^\u0000-\uFFFF]+` would work?</issue_description> > > ## Comments on the Issue (you are @copilot in this section) > > <comments> > <comment_new><author>@MrHinsh</author><body> > Do you suggest something like `[^\u0000-\uFFFF]+` as the default?</body></comment_new> > </comments> > </details> Fixes #3006 <!-- START COPILOT CODING AGENT TIPS --> --- 💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click [here](https://survey3.medallia.com/?EAHeSx-AP01bZqG0Ld9QLQ) to start the survey.
Fix StringManipulatorTool to support Unicode characters and strip emo… …jis for SOAP compatibility (#3016) ## Problem The `StringManipulatorTool`'s default regex pattern `[^( -~)\n\r\t]+` removes all non-ASCII characters, which breaks international migrations containing accented letters, Cyrillic, Chinese characters, and other Unicode content. For example, with the current implementation: - `"Café résumé"` becomes `"Caf rsum"` - `"Привет мир"` becomes `""` (empty) - `"你好世界"` becomes `""` (empty) Additionally, emojis can cause errors when data passes through SOAP interfaces and need to be stripped while preserving other Unicode symbols. This makes the tool unsuitable for any non-English Azure DevOps migrations. ## Solution Updated the default regex pattern from `[^( -~)\n\r\t]+` to `[\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F]|[\uD800-\uDBFF][\uDC00-\uDFFF]|\uFE0F` which: ✅ **Preserves Unicode content**: International characters are maintained ✅ **Strips emojis for SOAP compatibility**: Removes emoji surrogate pairs and variation selectors ✅ **Maintains security**: Still removes harmful control characters ✅ **Preserves formatting**: Keeps newlines (`\x0A`) and tabs (`\x09`) ✅ **Backward compatible**: All existing functionality preserved The new pattern specifically targets problematic content while preserving legitimate Unicode: - `\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F`: Control characters - `[\uD800-\uDBFF][\uDC00-\uDFFF]`: Emoji surrogate pairs (😀🔥💻🇺🇸) - `\uFE0F`: Variation selector for emoji presentation ## Testing - Added 27 comprehensive test cases covering Unicode and emoji scenarios - All 41 existing tests continue to pass - Verified no security vulnerabilities introduced - Demonstration of the fix: ``` Original Old Pattern New Pattern ---------------------------------------------------------------------- Héllo Wørld Hllo Wrld Héllo Wørld Привет мир Привет мир 你好世界 你好世界 Café résumé Caf rsum Café résumé Hello 😀 World Hll Wrld Hello World Math ∑ Symbol Mth Symbl Math ∑ Symbol Test\u0001Control TestControl TestControl ``` ## Documentation Updated the StringManipulatorTool documentation to: - Show examples with the new Unicode-friendly pattern - Provide specific emoji stripping examples for SOAP compatibility - Provide the legacy ASCII-only pattern for environments that require it - Clarify use cases for international migration scenarios This change enables the Azure DevOps Migration Tools to properly handle international content while maintaining SOAP compatibility and the same level of data safety and security. Fixes the issue raised in the GitHub discussion where users noted that non-ASCII character support was not available by default. Fixes #3006 <!-- START COPILOT CODING AGENT SUFFIX --> <details> <summary>Original prompt</summary> > > ---- > > *This section details on the original issue you should resolve* > > <issue_title>[Bug]: support for non-ASCII characters is not supported by default</issue_title> > <issue_description>By default "StringManipulatorTool" is enabled, but the regex pattern in AddDefaultManipulator() "[^( -~)\n\r\t]+" removes all non-ASCII characters, which is a problem for any non-english migrations. Could the pattern be updated to allow for at least unicode letters and numbers? > > Perhaps something like `[^\u0000-\uFFFF]+` would work?</issue_description> > > ## Comments on the Issue (you are @copilot in this section) > > <comments> > <comment_new><author>@MrHinsh</author><body> > Do you suggest something like `[^\u0000-\uFFFF]+` as the default?</body></comment_new> > </comments> > </details> Fixes #3006 <!-- START COPILOT CODING AGENT TIPS --> --- 💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click [here](https://survey3.medallia.com/?EAHeSx-AP01bZqG0Ld9QLQ) to start the survey.
PreviousNext