Skip to content

feat(server): support passing fetchLicense #1286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
+40 −26
Draft

feat(server): support passing fetchLicense #1286

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f6a83ab
feat(server): add fetchLicense arg
setchy Jul 31, 2024
5d6b390
feat(server): add fetchLicense arg
setchy Aug 1, 2024
074b4d3
feat(server): add fetchLicense arg
setchy Aug 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/SERVER.md
View file
Original file line number Diff line number Diff line change
@@ -52,6 +52,7 @@ Arguments can be passed either via the query string or as a JSON body. The follo
| only | Include components only containing this word in purl. Useful to generate BOM with first party components alone. Multiple values allowed. [array] |
| autoCompositions | Automatically set compositions when the BOM was filtered. [boolean] [default: true] |
| gitBranch | Git branch used when cloning the repository. If not specified will use the default branch assigned to the repository. |
| fetchLicense | Automatically query public registries such as maven, npm, or nuget to resolve the package licenses. This is a time-consuming operation and is disabled by default. |

## Ways to use server mode

8 changes: 4 additions & 4 deletions index.js
View file
Original file line number Diff line number Diff line change
@@ -32,7 +32,6 @@ import {
CARGO_CMD,
CLJ_CMD,
DEBUG_MODE,
FETCH_LICENSE,
LEIN_CMD,
MAX_BUFFER,
PREFER_MAVEN_DEPS_TREE,
@@ -135,6 +134,7 @@ import {
parseSwiftResolved,
parseYarnLock,
readZipEntry,
shouldFetchLicense,
splitOutputByGradleProjects,
} from "./utils.js";
let url = import.meta.url;
@@ -3100,7 +3100,7 @@ export async function createPythonBom(path, options) {
if (tempDir?.startsWith(tmpdir()) && rmSync) {
rmSync(tempDir, { recursive: true, force: true });
}
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
pkgList = await getPyMetadata(pkgList, false);
}
return buildBomNSData(options, pkgList, "pypi", {
@@ -4280,7 +4280,7 @@ export async function createSwiftBom(path, options) {
}
}
}
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
pkgList = await getSwiftPackageMetadata(pkgList);
}
return buildBomNSData(options, pkgList, "swift", {
@@ -5209,7 +5209,7 @@ export async function createCsharpBom(path, options) {
dependsOn: Array.from(parentDependsOn),
});
}
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
const retMap = await getNugetMetadata(pkgList, dependencies);
if (retMap.dependencies?.length) {
dependencies = mergeDependencies(
4 changes: 4 additions & 0 deletions server.js
View file
Original file line number Diff line number Diff line change
@@ -92,6 +92,7 @@ const parseQueryString = (q, body, options = {}) => {
"includeFormulation",
"includeCrypto",
"standard",
"fetchLicense",
];

for (const param of queryParams) {
@@ -115,6 +116,9 @@ const parseQueryString = (q, body, options = {}) => {
if (options.profile) {
applyProfileOptions(options);
}
if (options.fetchLicense) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since node is a single process event driven model, one request will affect multiple responses. We need to refactor the codebase and make the fetch license operate with an options argument instead of only relying on env variables

process.env.FETCH_LICENSE = options.fetchLicense;
}
return options;
};

53 changes: 31 additions & 22 deletions utils.js
View file
Original file line number Diff line number Diff line change
@@ -128,9 +128,12 @@ export const PREFER_MAVEN_DEPS_TREE =
["true", "1"].includes(process.env.PREFER_MAVEN_DEPS_TREE);

// Whether license information should be fetched
export const FETCH_LICENSE =
process.env.FETCH_LICENSE &&
["true", "1"].includes(process.env.FETCH_LICENSE);
export function shouldFetchLicense() {
return (
process.env.FETCH_LICENSE &&
["true", "1"].includes(process.env.FETCH_LICENSE)
);
}

// Whether search.maven.org will be used to identify jars without maven metadata; default, if unset shall be 'true'
export const SEARCH_MAVEN_ORG =
@@ -765,12 +768,18 @@ export async function getNpmMetadata(pkgList) {
let body = {};
if (metadata_cache[key]) {
body = metadata_cache[key];
if (DEBUG_MODE) {
console.log(`npm metadata: using ${key} metadata from local cache`);
}
} else {
const res = await cdxgenAgent.get(NPM_URL + key, {
responseType: "json",
});
body = res.body;
metadata_cache[key] = body;
if (DEBUG_MODE) {
console.log(`npm metadata: looking up ${key} metadata on npm`);
}
}
p.description =
body.versions?.[p.version]?.description || body.description;
@@ -872,7 +881,7 @@ export async function parsePkgJson(pkgJsonFile, simple = false) {
// continue regardless of error
}
}
if (!simple && FETCH_LICENSE && pkgList && pkgList.length) {
if (!simple && shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parsePkgJson`,
@@ -1215,7 +1224,7 @@ export async function parsePkgLock(pkgLockFile, options = {}) {
options,
));

if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parsePkgLock`,
@@ -1532,7 +1541,7 @@ export async function parseYarnLock(yarnLockFile) {
}
});
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parseYarnLock`,
@@ -1609,7 +1618,7 @@ export async function parseNodeShrinkwrap(swFile) {
}
}
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parseNodeShrinkwrap`,
@@ -2043,7 +2052,7 @@ export async function parsePnpmLock(pnpmLock, parentComponent = null) {
});
}
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parsePnpmLock`,
@@ -2102,7 +2111,7 @@ export async function parseBowerJson(bowerJsonFile) {
// continue regardless of error
}
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parseBowerJson`,
@@ -2187,7 +2196,7 @@ export async function parseMinJs(minJsFile) {
// continue regardless of error
}
}
if (FETCH_LICENSE && pkgList && pkgList.length) {
if (shouldFetchLicense() && pkgList && pkgList.length) {
if (DEBUG_MODE) {
console.log(
`About to fetch license information for ${pkgList.length} packages in parseMinJs`,
@@ -3176,7 +3185,7 @@ export async function getMvnMetadata(pkgList, jarNSMapping = {}) {
if (!pkgList || !pkgList.length) {
return pkgList;
}
if (DEBUG_MODE && FETCH_LICENSE) {
if (DEBUG_MODE && shouldFetchLicense()) {
console.log(`About to query maven for ${pkgList.length} packages`);
}
for (const p of pkgList) {
@@ -3212,7 +3221,7 @@ export async function getMvnMetadata(pkgList, jarNSMapping = {}) {
}
const group = p.group || "";
// If the package already has key metadata skip querying maven
if (group && p.name && p.version && !FETCH_LICENSE) {
if (group && p.name && p.version && !shouldFetchLicense()) {
cdepList.push(p);
continue;
}
@@ -3454,7 +3463,7 @@ export function guessPypiMatchingVersion(versionsList, versionSpecifiers) {
* @param {Boolean} fetchDepsInfo Fetch dependencies info from pypi
*/
export async function getPyMetadata(pkgList, fetchDepsInfo) {
if (!FETCH_LICENSE && !fetchDepsInfo) {
if (!shouldFetchLicense() && !fetchDepsInfo) {
return pkgList;
}
const PYPI_URL = process.env.PYPI_URL || "https://pypi.org/pypi/";
@@ -4368,7 +4377,7 @@ export async function getGoPkgLicense(repoMetadata) {
export async function getGoPkgComponent(group, name, version, hash) {
let pkg = {};
let license = undefined;
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
if (DEBUG_MODE) {
console.log(
`About to fetch go package license information for ${group}:${name}`,
@@ -4697,7 +4706,7 @@ export async function parseGosumData(gosumData) {
const version = tmpA[1].replace("/go.mod", "");
const hash = tmpA[tmpA.length - 1].replace("h1:", "sha256-");
let license = undefined;
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
if (DEBUG_MODE) {
console.log(
`About to fetch go package license information for ${name}`,
@@ -4749,7 +4758,7 @@ export async function parseGopkgData(gopkgData) {
case "name":
pkg.group = "";
pkg.name = value;
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
pkg.license = await getGoPkgLicense({
group: pkg.group,
name: pkg.name,
@@ -4959,7 +4968,7 @@ export async function parseGemspecData(gemspecData) {
}
});
pkgList = [pkg];
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
return await getRubyGemsMetadata(pkgList);
}
return pkgList;
@@ -5211,7 +5220,7 @@ export async function parseGemfileLockData(gemLockData, lockFile) {
dependsOn: Array.from(dependenciesMap[k]),
});
}
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
pkgList = await getRubyGemsMetadata(pkgList);
return { pkgList, dependenciesList };
}
@@ -5569,7 +5578,7 @@ export async function parseCargoTomlData(
if (pkg) {
addPackageToList(pkgList, pkg, { packageMode, simple });
}
if (!simple && FETCH_LICENSE) {
if (!simple && shouldFetchLicense()) {
return await getCratesMetadata(pkgList);
}
return pkgList;
@@ -5709,7 +5718,7 @@ export async function parseCargoData(
if (pkg) {
addPackageToList(pkgList, pkg, { simple });
}
if (FETCH_LICENSE && !simple) {
if (!simple && shouldFetchLicense()) {
return await getCratesMetadata(pkgList);
}
return pkgList;
@@ -5875,7 +5884,7 @@ export async function parseCargoAuditableData(cargoData) {
});
}
});
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
return await getCratesMetadata(pkgList);
}
return pkgList;
@@ -5914,7 +5923,7 @@ export async function parsePubLockData(pubLockData) {
}
}
});
if (FETCH_LICENSE) {
if (shouldFetchLicense()) {
return await getDartMetadata(pkgList);
}
return pkgList;