Skip to content

Enable liquid batch testing #229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
delphine-demeulenaere wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Enable liquid batch testing #229

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
500ce98
adjust version number
delphine-demeulenaere Nov 6, 2025
c376a9c
update CLI terminal
delphine-demeulenaere Nov 6, 2025
67103f0
update liquidTestRunner
delphine-demeulenaere Nov 6, 2025
1acc60c
make sure error message in terminal refers to correct code line
delphine-demeulenaere Nov 6, 2025
cc3edb9
adjust naming from 'batch' to 'pattern' (review comment)
delphine-demeulenaere Dec 12, 2025
fdc8386
adjust printed formatting to bullet points (review comment)
delphine-demeulenaere Dec 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions bin/cli.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -459,24 +459,30 @@ program
.option("--html-preview", "Get a static html of the export-view of the template generated with the Liquid Test data (optional)", false)
.option("--preview-only", "Skip the checking of the results of the Liquid Test in case you only want to generate a preview template (optional)", false)
.option("--status", "Only return the status of the test runs as PASSED/FAILED (optional)", false)
.option("-p, --pattern <pattern>", "Run all tests that match this pattern (optional)", "")

.action((options) => {
if (!options.handle && !options.accountTemplate) {
consola.error("You need to specify either a reconciliation handle or an account template");
process.exit(1);
}

if (options.test && options.pattern) {
consola.error("You cannot use both --test and --pattern options at the same time");
process.exit(1);
}

const templateType = options.handle ? "reconciliationText" : "accountTemplate";
const templateName = options.handle ? options.handle : options.accountTemplate;

if (options.status) {
liquidTestRunner.runTestsStatusOnly(options.firm, templateType, templateName, options.test);
liquidTestRunner.runTestsStatusOnly(options.firm, templateType, templateName, options.test, options.pattern);
} else {
if (options.previewOnly && !options.htmlInput && !options.htmlPreview) {
consola.info(`When using "--preview-only" you need to specify at least one of the following options: "--html-input", "--html-preview"`);
process.exit(1);
}
liquidTestRunner.runTestsWithOutput(options.firm, templateType, templateName, options.test, options.previewOnly, options.htmlInput, options.htmlPreview);
liquidTestRunner.runTestsWithOutput(options.firm, templateType, templateName, options.test, options.previewOnly, options.htmlInput, options.htmlPreview, options.pattern);
}
});

Expand Down Expand Up @@ -682,20 +688,26 @@ program
.option("-t, --test <test-name>", `Specify the name of the test to be run (optional). It has to be used together with "--handle"`, "")
.option("--html", `Get a html file of the template's input-view generated with the Liquid Test information (optional). It has to be used together with "--handle"`, false)
.option("--yes", "Skip the prompt confirmation (optional)")
.option("-b, --pattern <pattern>", `Run all tests that match this pattern (optional). It has to be used together with "--handle" or "--account-template"`, "")
Copy link
Contributor

@michieldegezelle michieldegezelle Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you change the -b into -p?

.action((options) => {
cliUtils.checkDefaultFirm(options.firm, firmIdDefault);
cliUtils.checkUniqueOption(["handle", "updateTemplates", "accountTemplate"], options);

if (options.test && options.pattern) {
consola.error("You cannot use both --test and --pattern options at the same time");
process.exit(1);
}

if (options.updateTemplates && !options.yes) {
cliUtils.promptConfirmation();
}

if (options.accountTemplate) {
devMode.watchLiquidTest(options.firm, options.accountTemplate, options.test, options.html, "accountTemplate");
devMode.watchLiquidTest(options.firm, options.accountTemplate, options.test, options.html, "accountTemplate", options.pattern);
}

if (options.handle) {
devMode.watchLiquidTest(options.firm, options.handle, options.test, options.html, "reconciliationText");
devMode.watchLiquidTest(options.firm, options.handle, options.test, options.html, "reconciliationText", options.pattern);
}
Comment on lines +691 to 711
Copy link

@coderabbitai coderabbitai bot Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Consider consistent short option for --pattern across commands (-p vs -b).

run-test uses -p (Line 462) but development-mode uses -b (Line 691) for the same concept. If there’s no conflict, aligning them reduces muscle-memory friction; otherwise, a short note in help text/docs would help.

🤖 Prompt for AI Agents 
In bin/cli.js around lines 691 to 711 the short flag for --pattern is -b which
is inconsistent with the run-test command using -p (line ~462); change the short
option to -p to align both commands if no other option currently uses -p, and if
-p is already taken either choose a consistent alternative across commands or
add an explicit note in the command help text documenting the difference; after
changing, run a quick scan of the CLI options to ensure no flag collisions and
update any usage/help strings or tests that reference -b to use -p.

if (options.updateTemplates) {
devMode.watchLiquidFiles(options.firm);
Expand Down
7 changes: 4 additions & 3 deletions lib/cli/devMode.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,9 @@ const chokidar = require("chokidar");
* @param {String} testName - Test name (empty string to run all tests)
* @param {boolean} renderInput - Open browser and show the HTML from input view
* @param {String} templateType - Template type (reconciliationText, accountTemplate)
* @param {String} pattern - Pattern to match test names (empty string to run all tests)
*/
async function watchLiquidTest(firmId, handle, testName, renderInput, templateType) {
async function watchLiquidTest(firmId, handle, testName, renderInput, templateType, pattern = "") {
if (templateType !== "reconciliationText" && templateType !== "accountTemplate") {
consola.error(`Template type is missing or invalid`);
process.exit(1);
Expand All @@ -33,15 +34,15 @@ async function watchLiquidTest(firmId, handle, testName, renderInput, templateTy
// Watch YAML
chokidar.watch(filePath).on("change", async () => {
// Run test
await liquidTestRunner.runTestsWithOutput(firmId, templateType, handle, testName, false, renderInput);
await liquidTestRunner.runTestsWithOutput(firmId, templateType, handle, testName, false, renderInput, false, pattern);
});

// Watch liquid files
const liquidFiles = fsUtils.listExistingRelatedLiquidFiles(firmId, handle, templateType);
for (const filePath of liquidFiles) {
chokidar.watch(filePath).on("change", async () => {
// Run test
await liquidTestRunner.runTestsWithOutput(firmId, templateType, handle, testName, false, renderInput);
await liquidTestRunner.runTestsWithOutput(firmId, templateType, handle, testName, false, renderInput, false, pattern);
});
}
}
Expand Down
149 changes: 128 additions & 21 deletions lib/liquidTestRunner.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,76 @@ function findTestRows(testContent) {
return indexes;
}

function buildTestParams(firmId, templateType, handle, testName = "", renderMode) {
function filterTestsByPattern(testContent, pattern, testIndexes) {
const indexes = testIndexes || findTestRows(testContent);
const matchingTests = Object.keys(indexes).filter((testName) => testName.includes(pattern));

if (matchingTests.length === 0) {
return { filteredContent: "", matchingTests: [], lineAdjustments: {} };
}

const testRows = testContent.split("\n");

const orderedTests = Object.entries(indexes)
.map(([name, index]) => ({ name, index }))
.sort((a, b) => a.index - b.index);

const matchingSet = new Set(matchingTests);
const segments = [];

orderedTests.forEach((test, idx) => {
if (!matchingSet.has(test.name)) {
return;
}

let start = test.index;

while (start > 0) {
const previousLine = testRows[start - 1];
const trimmedPrevious = previousLine.trim();
if (trimmedPrevious === "" || trimmedPrevious.startsWith("#")) {
start -= 1;
} else {
break;
}
}

let end = testRows.length;
for (let nextIdx = idx + 1; nextIdx < orderedTests.length; nextIdx++) {
const nextTest = orderedTests[nextIdx];
if (nextTest.index > test.index) {
end = nextTest.index;
break;
}
}

const segment = testRows.slice(start, end).join("\n").trimEnd();
segments.push(segment);
});

const filteredContent = segments.join("\n\n").trim();
const orderedMatchingTests = orderedTests.filter((test) => matchingSet.has(test.name)).map((test) => test.name);

const lineAdjustments = {};
if (filteredContent) {
const filteredIndexes = findTestRows(filteredContent);
orderedMatchingTests.forEach((testName) => {
const originalIndex = indexes[testName];
const filteredIndex = filteredIndexes[testName];
if (typeof originalIndex === "number" && typeof filteredIndex === "number") {
lineAdjustments[testName] = originalIndex - filteredIndex;
}
});
}

return {
filteredContent,
matchingTests: orderedMatchingTests,
lineAdjustments,
};
}

function buildTestParams(firmId, templateType, handle, testName = "", renderMode, pattern = "") {
let relativePath = `./reconciliation_texts/${handle}`;

if (templateType === "accountTemplate") {
Expand Down Expand Up @@ -63,6 +132,29 @@ function buildTestParams(firmId, templateType, handle, testName = "", renderMode
return false;
}

const testIndexes = findTestRows(testContent);

let finalTests = testContent;
let lineAdjustments = {};

if (pattern) {
const { filteredContent, matchingTests, lineAdjustments: patternLineAdjustments } = filterTestsByPattern(testContent, pattern, testIndexes);

if (!matchingTests.length) {
consola.error(`No tests found containing "${pattern}" in their name`);
process.exit(1);
}

finalTests = filteredContent;
lineAdjustments = patternLineAdjustments;
consola.info(
`Running ${matchingTests.length} test${matchingTests.length === 1 ? "" : "s"} matching pattern "${pattern}":`
);
matchingTests.forEach((testName) => {
consola.log(` • ${testName}`);
});
}

let templateContent;

if (templateType === "accountTemplate") {
Expand All @@ -88,20 +180,19 @@ function buildTestParams(firmId, templateType, handle, testName = "", renderMode

const testParams = {
template: templateContent,
tests: testContent,
tests: finalTests,
mode: renderMode,
};

// Include only one test
if (testName) {
const indexes = findTestRows(testContent);
if (!Object.keys(indexes).includes(testName)) {
if (!Object.keys(testIndexes).includes(testName)) {
consola.error(`Test ${testName} not found in YAML`);
process.exit(1);
}
testParams.test_line = indexes[testName] + 1;
testParams.test_line = testIndexes[testName] + 1;
}
return testParams;
return { testParams, metadata: { lineAdjustments } };
}

async function fetchResult(firmId, testRunId, templateType) {
Expand Down Expand Up @@ -129,12 +220,14 @@ async function fetchResult(firmId, testRunId, templateType) {
return testRun;
}

function listErrors(items, type) {
function listErrors(items, type, lineAdjustment = 0) {
const itemsKeys = Object.keys(items);
consola.log(chalk.red(`${itemsKeys.length} ${type} expectation${itemsKeys.length > 1 ? "s" : ""} failed`));
itemsKeys.forEach((itemName) => {
const itemDetails = items[itemName];
consola.log(`At line number ${itemDetails.line_number}`);
if (typeof itemDetails.line_number === "number") {
consola.log(`At line number ${itemDetails.line_number + lineAdjustment}`);
}
let gotDataType = typeof itemDetails.got;
let expectedDataType = typeof itemDetails.expected;
let displayedGot = itemDetails.got;
Expand Down Expand Up @@ -215,7 +308,7 @@ function checkTestErrorsPresent(testName, testsFeedback) {
return errorsPresent;
}

function processTestRunResponse(testRun, previewOnly) {
function processTestRunResponse(testRun, previewOnly, lineAdjustments = {}) {
// Possible status: started, completed, test_error, internal_error
let errorsPresent;
switch (testRun.status) {
Expand Down Expand Up @@ -249,6 +342,7 @@ function processTestRunResponse(testRun, previewOnly) {
consola.log(chalk.bold(testName));

const testElements = testRun.tests[testName];
const lineAdjustment = lineAdjustments[testName] || 0;

// Display success messages of test
if (testElements.reconciled === null) {
Expand All @@ -268,19 +362,21 @@ function processTestRunResponse(testRun, previewOnly) {
// Reconciled
if (testElements.reconciled !== null) {
consola.log(chalk.red("Reconciliation expectation failed"));
consola.log(`At line number ${testElements.reconciled.line_number}`);
if (typeof testElements.reconciled.line_number === "number") {
consola.log(`At line number ${testElements.reconciled.line_number + lineAdjustment}`);
}
consola.log(`got ${chalk.blue.bold(testElements.reconciled.got)} but expected ${chalk.blue.bold(testElements.reconciled.expected)}`);
consola.log("");
}

// Results
if (Object.keys(testElements.results).length > 0) {
listErrors(testElements.results, "result");
listErrors(testElements.results, "result", lineAdjustment);
}

// Rollforwards
if (Object.keys(testElements.rollforwards).length > 0) {
listErrors(testElements.rollforwards, "rollforward");
listErrors(testElements.rollforwards, "rollforward", lineAdjustment);
}
});
break;
Expand Down Expand Up @@ -365,16 +461,18 @@ async function handleHTMLfiles(testName = "", testRun, renderMode) {
}

// Used by VSCode Extension
async function runTests(firmId, templateType, handle, testName = "", previewOnly = false, renderMode = "none") {
async function runTests(firmId, templateType, handle, testName = "", previewOnly = false, renderMode = "none", pattern = "") {
try {
if (templateType !== "reconciliationText" && templateType !== "accountTemplate") {
consola.error(`Template type is missing or invalid`);
process.exit(1);
}

const testParams = buildTestParams(firmId, templateType, handle, testName, renderMode);
const buildResult = buildTestParams(firmId, templateType, handle, testName, renderMode, pattern);

if (!buildResult) return;

if (!testParams) return;
const { testParams, metadata } = buildResult;

let testRun = null;
let previewRun = null;
Expand All @@ -392,24 +490,33 @@ async function runTests(firmId, templateType, handle, testName = "", previewOnly
testRun = await fetchResult(firmId, testRunId, templateType);
}

return { testRun, previewRun };
return { testRun, previewRun, metadata };
} catch (error) {
errorUtils.errorHandler(error);
}
}

async function runTestsWithOutput(firmId, templateType, handle, testName = "", previewOnly = false, htmlInput = false, htmlPreview = false) {
async function runTestsWithOutput(
firmId,
templateType,
handle,
testName = "",
previewOnly = false,
htmlInput = false,
htmlPreview = false,
pattern = ""
) {
try {
if (templateType !== "reconciliationText" && templateType !== "accountTemplate") {
consola.error(`Template type is missing or invalid`);
process.exit(1);
}

const renderMode = runTestUtils.checkRenderMode(htmlInput, htmlPreview);
const testsRun = await runTests(firmId, templateType, handle, testName, previewOnly, renderMode);
const testsRun = await runTests(firmId, templateType, handle, testName, previewOnly, renderMode, pattern);
if (!testsRun) return;

processTestRunResponse(testsRun?.testRun || testsRun?.previewRun, previewOnly);
processTestRunResponse(testsRun?.testRun || testsRun?.previewRun, previewOnly, testsRun?.metadata?.lineAdjustments || {});

if (testsRun.previewRun && testsRun.previewRun.status !== "test_error" && renderMode !== "none") {
handleHTMLfiles(testName, testsRun.previewRun, renderMode);
Expand All @@ -421,14 +528,14 @@ async function runTestsWithOutput(firmId, templateType, handle, testName = "", p

// RETURN (AND LOG) ONLY PASSED OR FAILED
// CAN BE USED BY GITHUB ACTIONS
async function runTestsStatusOnly(firmId, templateType, handle, testName = "") {
async function runTestsStatusOnly(firmId, templateType, handle, testName = "", pattern = "") {
if (templateType !== "reconciliationText" && templateType !== "accountTemplate") {
consola.error(`Template type is missing or invalid`);
process.exit(1);
}

let status = "FAILED";
const testResult = await runTests(firmId, templateType, handle, testName, false, "none");
const testResult = await runTests(firmId, templateType, handle, testName, false, "none", pattern);

if (!testResult) {
status = "PASSED";
Expand Down
Loading
Loading