UnknownComponentDependencyException with SBOM from cyclonedx-javascript-library

[finger42]

Hello,

I try to enhance a given SBOM, which is generated with
https://github.com/CycloneDX/cyclonedx-javascript-library

While writing the sbom with output_as_string I get the exception:
UnknownComponentDependencyException

cyclonedx.exception.model.UnknownComponentDependencyException: One or more Components have Dependency references to Components/Services that are not known in this BOM. They are: {<BomRef '@mui/styled-engine@npm:5.16.6 [3135b]' id=126477793907040>, <BomRef '@mui/material@npm:5.14.0 [cd63f]' id=126477793557152>, <BomRef 'react-redux@npm:8.1.1 [cd63f]' id=126477790002304>, <BomRef '@mui/system@npm:5.16.7 [c1a0d]' id=126477793910160>}

For example the component:

@mui/styled-engine@npm:5.16.6 [3135b]

is used in:

components: [
{
      "type": "library",
      "name": "styled-engine",
      "group": "@mui",
      "version": "5.16.6",
      "bom-ref": "@mui/styled-engine@npm:5.16.6 [3135b]",
      "author": "MUI Team",
      "description": "styled() API wrapper package for emotion.",
      "licenses": [
        {
          "license": {
            "id": "MIT"
          }
        }
      ],
      "purl": "pkg:npm/%40mui/[email protected]?vcs_url=git%2Bhttps%3A//github.com/mui/material-ui.git#packages/mui-styled-engine",
      "externalReferences": [
        {
          "url": "https://github.com/mui/material-ui/issues",
          "type": "issue-tracker",
          "comment": "as detected from PackageJson property \"bugs.url\""
        },
        {
          "url": "git+https://github.com/mui/material-ui.git#packages/mui-styled-engine",
          "type": "vcs",
          "comment": "as detected from PackageJson property \"repository.url\" and \"repository.directory\""
        },
        {
          "url": "https://mui.com/system/styled/",
          "type": "website",
          "comment": "as detected from PackageJson property \"homepage\""
        }
      ]
    }

and

dependencies: [
{
      "ref": "@mui/styled-engine@npm:5.16.6 [3135b]",
      "dependsOn": [
        "@babel/runtime@npm:7.26.0",
        "@emotion/cache@npm:11.13.5",
        "csstype@npm:3.1.3",
        "prop-types@npm:15.8.1",
        "react@npm:18.2.0"
      ]
    },
{
      "ref": "@mui/system@npm:5.16.7 [260c3]",
      "dependsOn": [
        "@babel/runtime@npm:7.26.0",
        "@mui/private-theming@npm:5.16.6 [3135b]",
        "@mui/styled-engine@npm:5.16.6 [3135b]",
        "@mui/types@npm:7.2.19 [260c3]",
        "@mui/utils@npm:5.16.6 [260c3]",
        "clsx@npm:2.1.1",
        "csstype@npm:3.1.3",
        "prop-types@npm:15.8.1",
        "react@npm:18.2.0"
      ]
    },

I dont know why the [3135b] is added to this component. It looks like an hash or something.
Could it be that the space between @mui/system@npm:5.16.7 and [260c3] causes the problems?

I am using cyclonedx-python-lib Version: 8.5.0

Any ideas or suggestions would help.

[jkowalleck]

Thanks for the report, @finger42 .

Could you provide a reproducible setup?
Maybe create a repository that hosts the relevant SBOM(-Part), a requirements.txt declaring the needed dependencies, and a python script with the code you tried to run?

[finger42]

Hello,

Thx for the hint do to my homework ;-)

from json import loads as json_loads
from cyclonedx.model.bom import Bom
from cyclonedx.output.json import Json as JsonOutputter
from cyclonedx.output.json import JsonV1Dot5

json_data = """
    {
  "$schema": "http://cyclonedx.org/schema/bom-1.5.schema.json",
  "bomFormat": "CycloneDX",
  "specVersion": "1.5",
  "version": 1,
  "metadata": {
      "component": {
      "type": "application",
      "name": "example",
      "version": "1.2.3",
      "bom-ref": "topref"
    }
  },
  "components": [
    {
      "type": "library",
      "name": "styled-engine",
      "version": "5.16.6",
      "bom-ref": "@mui/styled-engine@npm:5.16.6 [296f2]"
    },
    {
      "type": "library",
      "name": "styled-engine",
      "version": "5.16.6",
      "bom-ref": "@mui/styled-engine@npm:5.16.6 [3135b]"
    }
  ],
  "dependencies": [
    {
      "ref": "topref",
      "dependsOn": [
        "@mui/styled-engine@npm:5.16.6 [296f2]",
        "@mui/styled-engine@npm:5.16.6 [3135b]"
      ]
    }
  ]
}
    """
bom = Bom.from_json(json_loads(json_data))
my_json_outputter: "JsonOutputter" = JsonV1Dot5(bom)
serialized_json = my_json_outputter.output_as_string(indent=2)

I think the problem is that there are two components, which are exactly the same ( only the bom-ref differs )

If I modify one property in one of the comonents. e.g. version to something other, the problem is gone.

Now my question: Is this a valid SBOM ?

If not, then the generator of the original SBOM , or another script, needs to strip the duplicated components ?

Beside that, the error msg:

cyclonedx.exception.model.UnknownComponentDependencyException: One or more Components have Dependency references to Components/Services that are not known in this BOM. They are: {<BomRef '@mui/styled-engine@npm:5.16.6 [3135b]' id=129184359355344>

could be improved to point out that there are duplicated components in the dependency tree ?

[jkowalleck]

I think the problem is that there are two components, which are exactly the same ( only the bom-ref differs )

yepp, exactly that should be the reason.

The python library does some deduplication, since it uses set() for collections.
And the component's hash function does not take a bom-ref into account, since there is novalue to it - a bom-ref is a mere anchor, not a qualitative property.

[jkowalleck]

Now my question: Is this a valid SBOM ?

It should be schema-valid, yes.

[finger42]

So I need to deduplicate the SBOM generated with the cyclonedx-javascript-library.
I will check that and report again.
Thx for the help so far.

[jkowalleck]

I try to enhance a given SBOM, which is generated with
https://github.com/CycloneDX/cyclonedx-javascript-library

@finger42 , may I ask for some background?

Are you using one of the many open-source SBOM generators running on nodej? Which one?
Or are you writing your own tool? Can I take a look at it?

Thank you in advance.
(I am the author and maintainer of that javascript library you were using. It would help to understand your downstream use cases. The usual thing with open source is: you never hear about success stories, you only see complaints and issue reports when something does not work as expected. Learning about downstream use cases helps to understand what is good and what needs to be improved.)

[finger42]

@jkowalleck

I try to build SBOMs with different tools. Some of these tools are using javascirpt for webapps, some of them the nuget lib, others the python lib. Or even other SBOM generators ( yocto. buildroot, ... ) I currently dont know, but they exist ;-)

All in common is the final goal to import SBOMs in self hosted https://dependencytrack.org/ to monitor the components for different firmware versions (also for old devices).
Most of the Sec. Issue matching is done with CPEs ( NIST )
Some of other databases are using PURL

https://github.com/CycloneDX/cyclonedx-javascript-library

I will ask my webdevs why the double components are inside the given SBOM.

I think that the CRA is the reason why this topic ( SBOM ) is getting more attention.

And yes, I understand that it is hard to get feedback from closed source to open source projects.
I am really happy with the features of the cyclonedx libs in general and all the work the contributes and maintainers do.

Thank you !

[jkowalleck]

a workaround: use properties to make each component unique.

[finger42]

@jkowalleck I need to wait till next year for my team an test the properties as a workaround.
Maybe we could find out the root cause for the double entries.

[jkowalleck]

a thing could be the nature of not-deduplicated node modules - read https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/docs/component_deduplication.md

and see examples here:
https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/demo/*/example-results/bare/bom.*.json versus https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/demo/*/example-results/flat/bom.*.json

[finger42]

Some background info:

1. Our main project is a monorepository with several packages that are consumed by apps.
2. We use yarn version 4.2.2 as package manager.
3. In each app we list the required dependencies specifically, e.g. "@mui/material": "5.14.0". We don’t hoist them to the root package/ don’t list them as peerDependencies.

SBOM generation:
We use "@cyclonedx/yarn-plugin-cyclonedx": "^1.0.1" (GitHub - CycloneDX/cyclonedx-node-yarn: Create CycloneDX Software Bill of Materials (SBOM) from Node.js Yarn projects.) to generate the SBOM.
It is installed as devDependency and currently run with this command:

yarn exec cyclonedx-yarn --output-reproducible --mc-type application --output-file sbom.cdx.json --production --spec-version 1.5

Results and Issues:
Some packages like @mui/styled-engine are listed more than once (as mentioned above):

"bom-ref": "@mui/styled-engine@npm:5.16.8 [00f08]",
"bom-ref": "@mui/styled-engine@npm:5.16.8 [6dc2f]",

This seems strange because our yarn.lock lists it only once:

"@mui/styled-engine@npm:^5.16.8":
  version: 5.16.8
  resolution: "@mui/styled-engine@npm:5.16.8"
  dependencies:
    "@babel/runtime": "npm:^7.23.9"
    "@emotion/cache": "npm:^11.11.0"
    csstype: "npm:^3.1.3"
    prop-types: "npm:^15.8.1"
  peerDependencies:
    "@emotion/react": ^11.4.1
    "@emotion/styled": ^11.3.0
    react: ^17.0.0 || ^18.0.0 || ^19.0.0
  peerDependenciesMeta:
    "@emotion/react":
      optional: true
    "@emotion/styled":
      optional: true
  checksum: 10c0/9c1866cf94fed61013324458be623f2de0c0cb6be065730c12441e798172a08389524b4b011c9e5a120bcc50c07cc7a835d365bd75a8b60f9a553d1de9c8af20
  languageName: node
  linkType: hard

but the results of running “yarn why @mui/styled-engine” correspond with the SBOM:

├─ @mui/system@npm:5.16.8
│  └─ @mui/styled-engine@npm:5.16.8 (via npm:^5.16.8)
│
├─ @mui/system@npm:5.16.8 [4db1c]
│  └─ @mui/styled-engine@npm:5.16.8 [00f08] (via npm:^5.16.8 [00f08])
│
└─ @mui/system@npm:5.16.8 [603e4]
   └─ @mui/styled-engine@npm:5.16.8 [6dc2f] (via npm:^5.16.8 [6dc2f])

[jkowalleck]

Ah, the usual non-deduplicated node modules, that is not an issue. That should be no issue.

Anyway, I see the root cause to be, as you suggested,
that the components are equal(bom-ref is not part of equality right now) and duplicates are therefore dropped.
A short term solution would be to add a unique property to the components generated by the yarn plugin, but for this I would need a reproducible testbed(similar to the ones in https://github.com/CycloneDX/cyclonedx-node-yarn/tree/main/tests/_data/testbeds)

I guess there is only one long-term solution to it:
bom-refs MUST become of equality-checks

I will work on a fix - a breaking change that should be released with the next major version, soon.

[jkowalleck]

this bug caused #753

[finger42]

Hello,
I tested locally the branch:
https://github.com/CycloneDX/cyclonedx-python-lib/tree/feat/BomRef-affects-comparissons-equality
and it works really nice.

Thx for your help @jkowalleck and also thx @wkoot !
You helped us a lot to monitor big nested dep. trees with https://dependencytrack.org/

Can you tell me when the version 9.x will be released ?

[jkowalleck]

the next release will take a while, as it will be composed of several interlinked features.

you can speed this up by contributing to them.
you may comment on #754 which part you will take over, and pullrequest the needed changes to the target branch feat/BomRef-affects-comparissons-equality.

but before all of this, i would really love to get #755 done. this one will make #754 easier to review.
Here goes the same: If you want to speed things up, please drop a note which part you want to work on, and just pullrequest that.

for all of this it is the same: pullrequest early (draft/wip) to indicate your intention & dont try to be perfect. "it works" is enough, we can improve over time.

[jkowalleck]

A preview of the fix/feature is available via https://github.com/CycloneDX/cyclonedx-python-lib/releases/tag/v9.0.1-rc.1

[finger42]

A preview of the fix/feature is available via https://github.com/CycloneDX/cyclonedx-python-lib/releases/tag/v9.0.1-rc.1

Thank you.
I have tested the branch and it works nicely !