diff --git a/.github/workflows/auto-format.yml b/.github/workflows/auto-format.yml new file mode 100644 index 000000000..195d87aef --- /dev/null +++ b/.github/workflows/auto-format.yml @@ -0,0 +1,45 @@ +--- +name: Auto-format PRs + +on: + pull_request: + types: [opened, synchronize, reopened] + +permissions: + contents: write + +jobs: + auto-format: + name: Auto format with pre-commit + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 + persist-credentials: true + + - name: Set up Python + uses: actions/setup-python@v4 + with: + python-version: '3.x' + + - name: Install pre-commit + run: python -m pip install --upgrade pip pre-commit + + - name: Run pre-commit and apply fixes + id: precommit + run: | + # Run all hooks and allow modifications + pre-commit run --all-files || true + # If pre-commit modified files, commit and push them back to the PR branch + if [ -n "$(git status --porcelain)" ]; then + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add -A + git commit -m "chore(format): apply pre-commit formatting" + # Try to push; if this action does not have permission the push will fail. + git push || (echo "Push failed: this action cannot push to the PR branch. If this PR came from a fork, enable 'Allow edits from maintainers' or run 'pre-commit install' and 'pre-commit run --all-files' locally, then push." && exit 0) + else + echo "No formatting changes needed" + fi diff --git a/.github/workflows/pre-commit-check.yml b/.github/workflows/pre-commit-check.yml new file mode 100644 index 000000000..288553b34 --- /dev/null +++ b/.github/workflows/pre-commit-check.yml @@ -0,0 +1,31 @@ +--- +name: pre-commit checks + +on: + push: + pull_request: + +permissions: + contents: read + +jobs: + pre-commit: + name: Run pre-commit checks + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Set up Python + uses: actions/setup-python@v4 + with: + python-version: "3.x" + + - name: Install pre-commit + run: python -m pip install --upgrade pip pre-commit + + - name: Run pre-commit (checks) + # This will exit non-zero if any hook fails (so the PR will be blocked) + run: pre-commit run --all-files \ No newline at end of file diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 000000000..b0fa604e6 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,16 @@ +exclude: | + ^application/static/| + ^application/frontend/www/| + ^application/static/assets/| + ^node_modules/| + ^dist/| + ^build/| + ^\.cache/| + +repos: + - repo: https://github.com/psf/black + rev: 24.10.0 + hooks: + - id: black + language_version: python3 + diff --git a/application/utils/spreadsheet_parsers.py b/application/utils/spreadsheet_parsers.py index be36be719..867b0f42e 100644 --- a/application/utils/spreadsheet_parsers.py +++ b/application/utils/spreadsheet_parsers.py @@ -333,7 +333,7 @@ def update_cre_in_links( def parse_hierarchical_export_format( - cre_file: List[Dict[str, str]] + cre_file: List[Dict[str, str]], ) -> Dict[str, List[defs.Document]]: """parses the main OpenCRE csv and creates a list of standards in it diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 7fdc26f9a..922860b06 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -6,8 +6,7 @@ The following is a set of guidelines for contributing. These are mostly guidelin #### Table Of Contents -- [Contributing to OpenCRE](#contributing-to-opencre) - - [Table Of Contents](#table-of-contents) +- [Contributing to OpenCRE](#contributing-to-opencre) - [Table Of Contents](#table-of-contents) - [Code of Conduct](#code-of-conduct) - [I don't want to read this whole thing I just have a question!!!](#i-dont-want-to-read-this-whole-thing-i-just-have-a-question) - [How Can I Contribute?](#how-can-i-contribute) @@ -20,7 +19,6 @@ The following is a set of guidelines for contributing. These are mostly guidelin - [Pull Requests](#pull-requests) - [Styleguides](#styleguides) - [Git Commit Messages](#git-commit-messages) - ## Code of Conduct @@ -38,7 +36,6 @@ or send a message to rob.vanderveer@owasp.org The "Issues" page lists a number of features we would like to implement, we have tagged the ones we believe are easy to pick up with the tag `good first issue` and/or `beginner`. Alternatively you can contribute content (see below) or request features or mappings by opening an Issue. - ### How can I contribute content (a standard mapping or changes to the CRE catalog)? Adding a mapping to OpenCRE for a new standard X means that sections in X are each assigned to the corresponding ‘Common Requirement’ (or CRE number) at opencre.org. @@ -46,12 +43,13 @@ For example, the section 613-Insufficien Session expiration in the CWE standard The result is that when you go to the overview page of that requirement(CRE), users will see a link to CWE 613: https://www.opencre.org/cre/065-782 How to: + 1. Get the OpenCRE standard mapping template spreadsheet -2. For every section in the standard, find the corresponding Common Requirement (CRE number) at OpenCRE and enter in that row the details of that section in the right columns: name, id, and hyperlink +2. For every section in the standard, find the corresponding Common Requirement (CRE number) at OpenCRE and enter in that row the details of that section in the right columns: name, id, and hyperlink 3. In case you identify opportunities to add Common Requirements: add those to the spreadsheet 4. Send the mapping template file by creating a new github issue and add the file. That way, the community can see it, and we can use that issue to further communicate. Another option is to send the file to rob.vanderveer@owasp.org. You can also use that mail address for any questions. -ad. 1 +ad. 1 The mapping spreadsheet (Excel) can be obtained [here](https://github.com/OWASP/OpenCRE/raw/refs/heads/main/docs/CREmappingtemplate.xls). Note, it contains one example, where it links the CRE for development processes to a section in NIST 800-53. @@ -71,12 +69,11 @@ In general, this will be rare. ad.4 OpenCRE has an importing interface in case you run your own myOpenCRE, but for the public opencre.org we first perform some checks before we add a standard to it - hence the request to send the mapping to us in email. +#### Alternative, automated method of linking: -#### Alternative, automated method of linking: -NOTE: there is an alternative method of linking a standard to OpenCRE, and it's the preferred way because it is self-maintaining. It requires editing the orginal standard text. What you do is you add to the standard a hyperlink reference to the relevant common requirement (e.g. [Restrict XML parsing](https://www.opencre.org/cre/764-507)) in a section on that topic. And then you specify to us at OpenCRE how we can read your source (e.g. Github) automatically to extract those hyperlinks and to construct links to that section in the document, and parse the name and id. That way, the links between the standard and OpenCRE will stay intact if links move, or change. It's easy and no maintenance required. +NOTE: there is an alternative method of linking a standard to OpenCRE, and it's the preferred way because it is self-maintaining. It requires editing the orginal standard text. What you do is you add to the standard a hyperlink reference to the relevant common requirement (e.g. [Restrict XML parsing](https://www.opencre.org/cre/764-507)) in a section on that topic. And then you specify to us at OpenCRE how we can read your source (e.g. Github) automatically to extract those hyperlinks and to construct links to that section in the document, and parse the name and id. That way, the links between the standard and OpenCRE will stay intact if links move, or change. It's easy and no maintenance required. Again, this only works if you have control over the content of the standard. Contact us if you want to set this up. We'd be happy to help. - ### Reporting Bugs When you are creating a bug report, please [include as many details as possible](#how-do-i-submit-a-good-bug-report). Fill out [the required template](https://github.com/common-requirement-enumeration/.github/blob/main/.github/ISSUE_TEMPLATE.md), the information it asks for helps us resolve issues faster. @@ -89,11 +86,11 @@ Bugs are tracked as [GitHub issues](https://guides.github.com/features/issues/). Explain the problem and include additional details to help maintainers reproduce the problem: -* **Use a clear and descriptive title** for the issue to identify the problem. -* **Describe the exact steps which reproduce the problem** in as many details as possible. -* **Provide specific examples to demonstrate the steps**. Include links to files or GitHub projects, or copy/pasteable snippets, which you use in those examples. If you're providing snippets in the issue, use [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines). -* **Describe the behavior you observed after following the steps** and point out what exactly is the problem with that behavior. -* **Explain which behavior you expected to see instead and why.** +- **Use a clear and descriptive title** for the issue to identify the problem. +- **Describe the exact steps which reproduce the problem** in as many details as possible. +- **Provide specific examples to demonstrate the steps**. Include links to files or GitHub projects, or copy/pasteable snippets, which you use in those examples. If you're providing snippets in the issue, use [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines). +- **Describe the behavior you observed after following the steps** and point out what exactly is the problem with that behavior. +- **Explain which behavior you expected to see instead and why.** ### Suggesting Enhancements @@ -105,19 +102,19 @@ When you are creating an enhancement suggestion, please [include as many details Enhancement suggestions are tracked as [GitHub issues](https://guides.github.com/features/issues/). Create an issue on that repository and provide the following information: -* **Use a clear and descriptive title** for the issue to identify the suggestion. -* **Provide a step-by-step description of the suggested enhancement** in as many details as possible. -* **Provide specific examples to demonstrate the steps**. Include copy/pasteable snippets which you use in those examples, as [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines). -* **Describe the current behavior** and **explain which behavior you expected to see instead** and why. -* **Explain why this enhancement would be useful**. +- **Use a clear and descriptive title** for the issue to identify the suggestion. +- **Provide a step-by-step description of the suggested enhancement** in as many details as possible. +- **Provide specific examples to demonstrate the steps**. Include copy/pasteable snippets which you use in those examples, as [Markdown code blocks](https://help.github.com/articles/markdown-basics/#multiple-lines). +- **Describe the current behavior** and **explain which behavior you expected to see instead** and why. +- **Explain why this enhancement would be useful**. ### Your First Code Contribution Unsure where to begin contributing? You can start by looking through these `beginner`, `good first issue` and `help-wanted` issues: -* Beginner issues - issues which should only require a few lines of code, and a test or two. -* Good first issue - issues which should require more substantial changes but can be done in an afternoon or two. -* Help wanted issues - issues which should be a bit more involved than `beginner` issues. +- Beginner issues - issues which should only require a few lines of code, and a test or two. +- Good first issue - issues which should require more substantial changes but can be done in an afternoon or two. +- Help wanted issues - issues which should be a bit more involved than `beginner` issues. #### Pull Requests @@ -130,12 +127,11 @@ We use eslint and black to enforce style. `make lint` should fix most style prob ### Git Commit Messages -* Use the present tense ("Add feature" not "Added feature") -* Use the imperative mood ("Move cursor to..." not "Moves cursor to...") -* Limit the first line to 72 characters or less -* Reference issues and pull requests liberally after the first line -* When only changing documentation, include `[ci skip]` in the commit title. - +- Use the present tense ("Add feature" not "Added feature") +- Use the imperative mood ("Move cursor to..." not "Moves cursor to...") +- Limit the first line to 72 characters or less +- Reference issues and pull requests liberally after the first line +- When only changing documentation, include `[ci skip]` in the commit title. ## Working on the frontend @@ -143,8 +139,62 @@ You can start the frontend in a debuggable instance separate from the backend us After you have installed the application following the README. You can start the backend server with: + 
make dev-flask
-Then in a separate terminal + +Then in a separate terminal + 
cd application/frontend
+ and finally start the frontend in a debug session: + 
yarn start
+ +--- + +## Code Formatting & CI Consistency + +OpenCRE enforces consistent formatting and basic linting using **pre-commit**.\ +This is done to prevent recurring CI failures (for example, `PYTHON_BLACK` errors) and to ensure a smooth review process for **all contributors**, including fork-based pull requests. + +### What this means for contributors + +- Python formatting is enforced using **Black** +- These checks run both **locally (recommended)** and **in CI** + +If formatting or linting rules are violated, CI will fail until the issue is fixed. + +### Running checks locally (strongly recommended) + +Installing and running pre-commit locally helps you catch issues **before pushing**. + +``` +pip install pre-commit +pre-commit install +pre-commit run --all-files + +``` + +### **CI behavior** + +- All pull requests (including forks) run pre-commit checks in CI + +- Python formatting is handled by **pre-commit**, not Super-Linter + +- Super-Linter continues to run for non-Python checks + +### **Auto-formatting** + +Some pull requests may receive **automatic formatting commits** via GitHub Actions when possible: + +- PRs from the main repository + +- Forked PRs with **"Allow edits from maintainers"** enabled + +If automatic fixes cannot be pushed, CI logs will explain how to fix issues locally. + +### **Forks & maintainer edits** + +Contributors using forks are encouraged to enable **"Allow edits from maintainers"** when opening a PR. + +This allows maintainers to quickly apply formatting fixes and reduce review friction. diff --git a/migrations/versions/7a17989aa1e3_first_migration.py b/migrations/versions/7a17989aa1e3_first_migration.py index d0e269829..c5ed7d586 100644 --- a/migrations/versions/7a17989aa1e3_first_migration.py +++ b/migrations/versions/7a17989aa1e3_first_migration.py @@ -1,7 +1,7 @@ """First Migration Revision ID: 7a17989aa1e3 -Revises: +Revises: Create Date: 2021-08-31 19:23:49.227719 """