Format-Hex: Improve mixed-collection piped input and piped streams of input #8674

vexx32 · 2019-01-17T16:59:54Z

PR Summary

Refactoring & Streamed Input Support

Handles continuous data stream input as a single input stream, and will display it in the same manner for both Format-Hex -InputObject $byteArray and $byteArray | Format-Hex
- This includes arrays / streams of all basic value types.
Handles jagged streams of input by clumping input objects according to their type. This is not expected to handle a jagged array passed directly to the -InputObject parameter by name.
Adds ByteCollection properties:
- Label — stores the type name and a random hexadecimal ID. Used to group objects in output formatting. This can be utilised by, for example, Group-Object, in order to enable downstream cmdlets to handle each group of ByteCollection objects separately, if needed.
- Ascii — stores the ASCII representation of the bytes in the ByteCollection
- HexBytes — stores the hexadecimal representation of the bytes in the ByteCollection
Filters ASCII display to replace control characters with the <?> unicode glyph

Additional tests have been added to cover this behaviour. Details below.

Behaviour of Piped Input

A Primitive Value

Each item is converted to bytes and collected into an input buffer.

The type of the last InputObject is stored in a private field.

If a new InputObject is received of the same type, it is converted to bytes and added to the buffer.

If a new InputObject is received of a different type, the buffer is divided into 16-byte segments and each segment is sent to the success output stream as a ByteCollection. Then, the process restarts from the beginning.

System.IO.FileInfo

If any data is in the primitives buffer, it is output as described above, the buffer is cleared, as well as the information on what the last type was (resetting the grouping logic).

One ByteCollection object is emitted per 16 bytes of the file.

All ByteCollection objects are tagged with the input file's path as their Label, which will group their output into a single table.

Offset and Count are interpreted as applying to each file, the same way as they would be for -Path inputs.

String

If any data is in the primitives buffer, it is output as described above, the buffer is cleared, as well as the information on what the last type was (resetting the grouping logic).

All strings are handled on an individual basis (no grouping), and are converted into bytes according to the -Encoding parameter's value.

One ByteCollection object is emitted per 16 bytes of the string, and each is tagged with the System.String type name and a shared random hexadecimal ID to ensure that each string is assigned a separate Label group.

Offset and Count are interpreted as applying to each input string.

Array of Primitives

If any data is in the primitives buffer, it is output as described above, the buffer is cleared, as well as the information on what the last type was (resetting the grouping logic).

Each array is treated as a single "group", and will not be grouped with other items in the output.

All items in the array are converted into a single chunk of bytes, which then is sliced according to Offset and Count.

One ByteCollection object is emitted per 16 bytes, and each is tagged with the array's type and a shared random hexadecimal ID to ensure that each array is assigned a single Label group.

Examples

PR Context

Format-Hex needs a bit more versatility and to be able to handle piped input in a more effective manner.

PR Checklist

vexx32 · 2019-01-17T19:18:12Z

RE the Codacy issues - _isHeterogenousPipedInput and _lastInputType exist as fields as their values must persist between calls to the method it's used in.

I'll take care of the rest.

The Static Analysis link failure is unrelated, possibly an older page was taken down:

2019-01-17T17:09:58.2571685Z       [-] http://www.powershellmagazine.com/2014/04/24/windows-powershell-4-0-and-other-quick-reference-guides/ should work 131.74s
2019-01-17T17:09:58.3083022Z         RuntimeException: retry of url failed with error: 
2019-01-17T17:09:58.3092376Z         at <ScriptBlock>, /home/vsts/work/1/s/test/common/markdown/markdown-link.tests.ps1: line 109

EDIT: Seems fine now. Weird.

vexx32 · 2019-01-17T19:29:57Z

Codacy is a little wonky at times, it seems. Its request to make the methods static cannot be resolved, for example if we try to make the methods static we get an error on the WriteObject() calls:

iSazonov · 2019-01-18T05:36:44Z

@vexx32 Thanks for the great contribution!
Could you please move all style changes to another PR (we will fast merge it)? It'll make the code review more easy and fast. Thanks!

vexx32 · 2019-01-18T10:59:58Z

@iSazonov yeah, that makes sense. Will do!

vexx32 · 2019-01-19T18:25:15Z

Close & Reopen to restart failing xUnit test on Windows. Don't think that should have been affected by anything here.

vexx32 · 2019-01-28T20:15:11Z

/cc @iSazonov @JamesWTruher @PaulHigin

Style changes PR was #8683 and is merged. This one's just pending your guys review, I think? Let me know. 😄

stale · 2019-02-27T20:18:55Z

This PR has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs within 10 days.
Thank you for your contributions.
Community members are welcome to grab these works.

vexx32 · 2019-02-27T20:24:49Z

Ping @PaulHigin for review. 😃

...rosoft.PowerShell.Commands.Utility/commands/utility/FormatAndOutput/format-hex/Format-Hex.cs

iSazonov · 2019-03-16T09:13:45Z

@vexx32 I reviewed this again. I think we could close the PR as controversial and continue the discussion in an issue:

Handles continuous data stream input as a single input stream

I doubt it is necessary. Perhaps this is exactly what users want in some scenarios.

Handles heterogenous streams of input by treating each input item as a discrete item to display

Looks useful in all scenarious.

vexx32 · 2019-03-16T11:35:22Z

If we must, we must. 🤷‍♂️

I'm inclined to think that users would find its current behaviour when piping a byte stream/array to it extremely useless, but that's just me.

But if I'm honest, it may well be just that it doesn't need the behaviour as it isn't used extensively.

stale · 2019-04-15T12:17:43Z

This PR has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs within 10 days.
Thank you for your contributions.
Community members are welcome to grab these works.

iSazonov · 2019-04-16T05:30:14Z

@vexx32 Please resolve the merge conflict. I think we need rebase to get latest CIs updates.

vexx32 · 2019-04-16T20:48:33Z

@iSazonov done and done. Let's see how CI runs fare. 😄

vexx32 · 2019-05-31T18:08:10Z

@anmenaga ping for review. 🙂

JamesWTruher

not blocking.
one comment about the new tests (and i suppose a reflection on all the tests for this).

test/powershell/Modules/Microsoft.PowerShell.Utility/Format-Hex.Tests.ps1

vexx32 · 2019-10-10T01:47:05Z

@iSazonov updated tests to account for behavioural changes and updated the PR description to hopefully clarify what's going on. If anything still isn't clear, please let me know! 💖

vexx32 · 2019-10-10T03:58:03Z

@PoshChan please retry macos

PoshChan · 2019-10-10T03:58:30Z

@vexx32, successfully started retry of PowerShell-CI-macOS

iSazonov · 2019-10-10T14:37:14Z

@PoshChan Please remind me in 1 day

PoshChan · 2019-10-11T14:37:35Z

@iSazonov, this is the reminder you requested 1 day ago

iSazonov · 2019-10-11T16:18:30Z

@SteveL-MSFT @TravisEz13 @rkeithhill The PR is ready for final code review.

SteveL-MSFT · 2019-10-12T04:25:48Z

...rosoft.PowerShell.Commands.Utility/commands/utility/FormatAndOutput/format-hex/Format-Hex.cs

+            }
+            else
+            {
+                WriteHexadecimal(bytes, offset: 0, label: GetGroupLabel(typeof(string)));


Should either consistently name the label param or don't, but it's a bit inconsistent.

I think I'll opt for naming it, since one of the other possible overloads also has a string param (path), so it's safer to name this one.

SteveL-MSFT

LGTM, one NIT comment

...rosoft.PowerShell.Commands.Utility/commands/utility/FormatAndOutput/format-hex/Format-Hex.cs

vexx32 · 2019-10-12T04:50:03Z

@PoshChan please retry static

PoshChan · 2019-10-12T04:50:21Z

@vexx32, successfully started retry of PowerShell-CI-static-analysis

...rosoft.PowerShell.Commands.Utility/commands/utility/FormatAndOutput/format-hex/Format-Hex.cs

vexx32 · 2019-10-12T05:05:46Z

Oh, I guess MSDN is down for a bit. 😄

TravisEz13 · 2019-10-14T18:06:36Z

retried the URL check

anmenaga · 2019-10-15T18:03:08Z

Added doc issue reference to PR template: MicrosoftDocs/PowerShell-Docs#4532

anmenaga · 2019-10-15T18:05:19Z

@vexx32 Thank you!

ghost · 2019-10-23T19:07:59Z

🎉v7.0.0-preview.5 has been released which incorporates this pull request.:tada:

Handy links:

Release Notes

… input (PowerShell#8674)

vexx32 requested review from JamesWTruher and PaulHigin as code owners January 17, 2019 16:59

vexx32 mentioned this pull request Jan 18, 2019

Cleanup Format-Hex #8683

Merged

11 tasks

vexx32 closed this Jan 19, 2019

vexx32 reopened this Jan 19, 2019

stale bot added the Stale label Feb 27, 2019

stale bot removed the Stale label Feb 27, 2019

iSazonov reviewed Feb 28, 2019

View reviewed changes

...rosoft.PowerShell.Commands.Utility/commands/utility/FormatAndOutput/format-hex/Format-Hex.cs Show resolved Hide resolved

iSazonov changed the title ~~Format-Hex: Improve mixed-collection piped input and piped streams of input~~ WIP: Format-Hex: Improve mixed-collection piped input and piped streams of input Mar 16, 2019

stale bot added the Stale label Apr 15, 2019

stale bot removed the Stale label Apr 16, 2019

vexx32 force-pushed the FormatHexRefactor branch from 5de1539 to d405003 Compare April 16, 2019 20:46

vexx32 mentioned this pull request Apr 18, 2019

Format-Hex should support all primitive types (including enums) and arrays of these types - instead of this random assortment. #3358

Closed

vexx32 changed the title ~~WIP: Format-Hex: Improve mixed-collection piped input and piped streams of input~~ Format-Hex: Improve mixed-collection piped input and piped streams of input Apr 18, 2019

SteveL-MSFT assigned anmenaga Apr 30, 2019

JamesWTruher reviewed Jun 22, 2019

View reviewed changes

test/powershell/Modules/Microsoft.PowerShell.Utility/Format-Hex.Tests.ps1 Outdated Show resolved Hide resolved

🎨 Set fixed width for the random ID

e12caad

🎨 Codacy/CodeFactor fixes

feb5d57

iSazonov approved these changes Oct 11, 2019

View reviewed changes

vexx32 requested review from TravisEz13, anmenaga and SteveL-MSFT October 12, 2019 03:26

SteveL-MSFT reviewed Oct 12, 2019

View reviewed changes

SteveL-MSFT approved these changes Oct 12, 2019

View reviewed changes

🎨 Use consistent param naming

a2ae168

PoshChan reviewed Oct 12, 2019

View reviewed changes

...rosoft.PowerShell.Commands.Utility/commands/utility/FormatAndOutput/format-hex/Format-Hex.cs Show resolved Hide resolved

PoshChan reviewed Oct 12, 2019

View reviewed changes

...rosoft.PowerShell.Commands.Utility/commands/utility/FormatAndOutput/format-hex/Format-Hex.cs Show resolved Hide resolved

anmenaga mentioned this pull request Oct 15, 2019

Document changes to Format-Hex for mixed-collection piped input MicrosoftDocs/PowerShell-Docs#4936

Closed

13 tasks

anmenaga merged commit 2e95179 into PowerShell:master Oct 15, 2019

vexx32 deleted the FormatHexRefactor branch October 15, 2019 18:09

vexx32 mentioned this pull request Oct 16, 2019

Format-Hex should not try to render unicode control characters that affect the display #7777

Closed

kilasuit pushed a commit to kilasuit/PowerShell that referenced this pull request Nov 9, 2019

Format-Hex: Improve mixed-collection piped input and piped streams of…

bb06133

… input (PowerShell#8674)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Format-Hex: Improve mixed-collection piped input and piped streams of input #8674

Format-Hex: Improve mixed-collection piped input and piped streams of input #8674

vexx32 commented Jan 17, 2019 •

edited

Loading

vexx32 commented Jan 17, 2019 •

edited

Loading

vexx32 commented Jan 17, 2019 •

edited

Loading

iSazonov commented Jan 18, 2019

vexx32 commented Jan 18, 2019

vexx32 commented Jan 19, 2019

vexx32 commented Jan 28, 2019

stale bot commented Feb 27, 2019

vexx32 commented Feb 27, 2019

iSazonov commented Mar 16, 2019

vexx32 commented Mar 16, 2019

stale bot commented Apr 15, 2019

iSazonov commented Apr 16, 2019

vexx32 commented Apr 16, 2019

vexx32 commented May 31, 2019

JamesWTruher left a comment

vexx32 commented Oct 10, 2019

vexx32 commented Oct 10, 2019

PoshChan commented Oct 10, 2019

iSazonov commented Oct 10, 2019

PoshChan commented Oct 11, 2019

iSazonov commented Oct 11, 2019

SteveL-MSFT Oct 12, 2019

vexx32 Oct 12, 2019

SteveL-MSFT left a comment

vexx32 commented Oct 12, 2019

PoshChan commented Oct 12, 2019

vexx32 commented Oct 12, 2019

TravisEz13 commented Oct 14, 2019

anmenaga commented Oct 15, 2019

anmenaga commented Oct 15, 2019

ghost commented Oct 23, 2019

Format-Hex: Improve mixed-collection piped input and piped streams of input #8674

Format-Hex: Improve mixed-collection piped input and piped streams of input #8674

Conversation

vexx32 commented Jan 17, 2019 • edited Loading

PR Summary

Refactoring & Streamed Input Support

Behaviour of Piped Input

A Primitive Value

System.IO.FileInfo

String

Array of Primitives

Examples

PR Context

PR Checklist

vexx32 commented Jan 17, 2019 • edited Loading

vexx32 commented Jan 17, 2019 • edited Loading

iSazonov commented Jan 18, 2019

vexx32 commented Jan 18, 2019

vexx32 commented Jan 19, 2019

vexx32 commented Jan 28, 2019

stale bot commented Feb 27, 2019

vexx32 commented Feb 27, 2019

iSazonov commented Mar 16, 2019

vexx32 commented Mar 16, 2019

stale bot commented Apr 15, 2019

iSazonov commented Apr 16, 2019

vexx32 commented Apr 16, 2019

vexx32 commented May 31, 2019

JamesWTruher left a comment

Choose a reason for hiding this comment

vexx32 commented Oct 10, 2019

vexx32 commented Oct 10, 2019

PoshChan commented Oct 10, 2019

iSazonov commented Oct 10, 2019

PoshChan commented Oct 11, 2019

iSazonov commented Oct 11, 2019

SteveL-MSFT Oct 12, 2019

Choose a reason for hiding this comment

vexx32 Oct 12, 2019

Choose a reason for hiding this comment

SteveL-MSFT left a comment

Choose a reason for hiding this comment

vexx32 commented Oct 12, 2019

PoshChan commented Oct 12, 2019

vexx32 commented Oct 12, 2019

TravisEz13 commented Oct 14, 2019

anmenaga commented Oct 15, 2019

anmenaga commented Oct 15, 2019

ghost commented Oct 23, 2019

vexx32 commented Jan 17, 2019 •

edited

Loading

vexx32 commented Jan 17, 2019 •

edited

Loading

vexx32 commented Jan 17, 2019 •

edited

Loading