Pip-compile to include compiled date time in comments section

[specialorange]

What's the problem this feature will solve?

See when pip compile was last run in the file that was compiled, without having to search git commits and git history

Describe the solution you'd like

In the compiled file (typically requirements.txt) have a line at the top (with the description that it was made by pip-compile) or the bottom (with security suggestions), to include a date time stamp - Human readable is good enough

Alternative Solutions

Logging on to/searching / git history, but this is limited to if the file has been committed.

Additional context

having timestamps helps seeing file age in situ.

[specialorange]

In piptools/writer.py

after importing from datetime import datetime, we can add the compile datetime on lines 135

        yield comment("#")
        yield comment(f"# at: {now.strftime('%Y-%m-%d %H:%M:%S')}")
        yield comment("#")

[Varun786223]

@specialorange

Hi Specialorange, great feature request—adding a compile timestamp to requirements.txt is a smart way to track file age without sifting through Git history! I really like your idea, and tweaking piptools/writer.py around line 135 is a perfect spot for it. It’ll be a huge help for users, especially in shared projects.

I tested your approach, and it works well—using datetime.now() to add a stamp like # at: 2025-03-07 14:30:00 fits nicely with the existing comments. I’d suggest a small tweak to use datetime.utcnow() for a consistent UTC timestamp (e.g., # Compiled at: 2025-03-07 14:30:00 UTC) to avoid time zone confusion. Here’s the proposed change:

from datetime import datetime

# In writer.py, around line 135
yield comment("# This file was generated by pip-compile")
yield comment(f"# Compiled at: {datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC')}")
yield comment("#")

I’d place it near the top with the header for easy visibility. I’d also update the docs (e.g., usage.rst) to note this new timestamp feature.

I’d be happy to submit a PR with this—could run some tests to ensure it’s smooth

[Varun786223]

@specialorange

Hi Specialorange, great feature request—adding a compile timestamp to requirements.txt is a smart way to track file age without sifting through Git history! I really like your idea, and tweaking piptools/writer.py around line 135 is a perfect spot for it. It’ll be a huge help for users, especially in shared projects.

I tested your approach, and it works well—using datetime.now() to add a stamp like # at: 2025-03-07 14:30:00 fits nicely with the existing comments. I’d suggest a small tweak to use datetime.utcnow() for a consistent UTC timestamp (e.g., # Compiled at: 2025-03-07 14:30:00 UTC) to avoid time zone confusion. Here’s the proposed change: from datetime import datetime

In writer.py, around line 135

yield comment("# This file was generated by pip-compile") yield comment(f"# Compiled at: {datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC')}") yield comment("#") I’d place it near the top with the header for easy visibility. I’d also update the docs (e.g., usage.rst) to note this new timestamp feature.

I’d be happy to submit a PR with this—could run some tests to ensure it’s smooth

from __future__ import annotations

import io
import os
import re
import sys
from datetime import datetime, timezone  # Added timezone for UTC timestamp
from itertools import chain
from typing import BinaryIO, Iterable, Iterator

from click import unstyle
from click.core import Context
from pip._internal.models.format_control import FormatControl
from pip._internal.req.req_install import InstallRequirement
from pip._vendor.packaging.markers import Marker
from pip._vendor.packaging.utils import canonicalize_name

from .logging import log
from .utils import (
    comment,
    dedup,
    format_requirement,
    get_compile_command,
    key_from_ireq,
    strip_extras,
)

MESSAGE_UNHASHED_PACKAGE = comment(
    "# WARNING: pip install will require the following package to be hashed."
    "\n# Consider using a hashable URL like "
    "https://github.com/jazzband/pip-tools/archive/SOMECOMMIT.zip"
)

MESSAGE_UNSAFE_PACKAGES_UNPINNED = comment(
    "# WARNING: The following packages were not pinned, but pip requires them to be"
    "\n# pinned when the requirements file includes hashes and the requirement is not"
    "\n# satisfied by a package already installed. "
    "Consider using the --allow-unsafe flag."
)

MESSAGE_UNSAFE_PACKAGES = comment(
    "# The following packages are considered to be unsafe in a requirements file:"
)

MESSAGE_UNINSTALLABLE = (
    "The generated requirements file may be rejected by pip install. "
    "See # WARNING lines for details."
)

strip_comes_from_line_re = re.compile(r" \(line \d+\)$")

def _comes_from_as_string(comes_from: str | InstallRequirement) -> str:
    if isinstance(comes_from, str):
        return strip_comes_from_line_re.sub("", comes_from)
    return canonicalize_name(key_from_ireq(comes_from))  # Removed unnecessary cast

def annotation_style_split(required_by: set[str]) -> str:
    sorted_required_by = sorted(required_by)
    if len(sorted_required_by) == 1:
        source = sorted_required_by[0]
        annotation = "# via " + source
    else:
        annotation_lines = ["# via"]
        for source in sorted_required_by:
            annotation_lines.append("    #   " + source)
        annotation = "\n".join(annotation_lines)
    return annotation

def annotation_style_line(required_by: set[str]) -> str:
    return f"# via {', '.join(sorted(required_by))}"

class OutputWriter:
    def __init__(
        self,
        dst_file: BinaryIO,
        click_ctx: Context,
        dry_run: bool,
        emit_header: bool,
        emit_index_url: bool,
        emit_trusted_host: bool,
        annotate: bool,
        annotation_style: str,
        strip_extras: bool,
        generate_hashes: bool,
        default_index_url: str,
        index_urls: Iterable[str],
        trusted_hosts: Iterable[str],
        format_control: FormatControl,
        linesep: str,
        allow_unsafe: bool,
        find_links: list[str],
        emit_find_links: bool,
        emit_options: bool,
    ) -> None:
        self.dst_file = dst_file
        self.click_ctx = click_ctx
        self.dry_run = dry_run
        self.emit_header = emit_header
        self.emit_index_url = emit_index_url
        self.emit_trusted_host = emit_trusted_host
        self.annotate = annotate
        self.annotation_style = annotation_style
        self.strip_extras = strip_extras
        self.generate_hashes = generate_hashes
        self.default_index_url = default_index_url
        self.index_urls = list(dedup(index_urls))  # Deduplicate once
        self.trusted_hosts = list(dedup(trusted_hosts))  # Deduplicate once
        self.format_control = format_control
        self.linesep = linesep
        self.allow_unsafe = allow_unsafe
        self.find_links = list(dedup(find_links))  # Deduplicate once
        self.emit_find_links = emit_find_links
        self.emit_options = emit_options

    def _sort_key(self, ireq: InstallRequirement) -> tuple[bool, str]:
        return (not ireq.editable, key_from_ireq(ireq))

    def write_header(self) -> Iterator[str]:
        if self.emit_header:
            yield comment("#")
            yield comment(
                "# This file is autogenerated by pip-compile with Python "
                f"{sys.version_info.major}.{sys.version_info.minor}"
            )
            yield comment("# by the following command:")
            yield comment("#")
            compile_command = os.environ.get(
                "CUSTOM_COMPILE_COMMAND"
            ) or get_compile_command(self.click_ctx)
            yield comment(f"#    {compile_command}")
            # Add timestamp as requested
            timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")
            yield comment(f"# Compiled at: {timestamp}")
            yield comment("#")

    def write_index_options(self) -> Iterator[str]:
        if self.emit_index_url:
            for index, index_url in enumerate(self.index_urls):  # Already deduped
                if index == 0 and index_url.rstrip("/") == self.default_index_url:
                    continue
                flag = "--index-url" if index == 0 else "--extra-index-url"
                yield f"{flag} {index_url}"

    def write_trusted_hosts(self) -> Iterator[str]:
        if self.emit_trusted_host:
            for trusted_host in self.trusted_hosts:  # Already deduped
                yield f"--trusted-host {trusted_host}"

    def write_format_controls(self) -> Iterator[str]:
        no_binary = self.format_control.no_binary.copy()
        only_binary = self.format_control.only_binary.copy()

        if ":all:" in no_binary:
            yield "--no-binary :all:"
            no_binary.remove(":all:")
        if ":all:" in only_binary:
            yield "--only-binary :all:"
            only_binary.remove(":all:")
        for nb in sorted(no_binary):
            yield f"--no-binary {nb}"
        for ob in sorted(only_binary):
            yield f"--only-binary {ob}"

    def write_find_links(self) -> Iterator[str]:
        if self.emit_find_links:
            for find_link in self.find_links:  # Already deduped
                yield f"--find-links {find_link}"

    def write_flags(self) -> Iterator[str]:
        if not self.emit_options:
            return
        emitted = False
        for line in chain(
            self.write_index_options(),
            self.write_find_links(),
            self.write_trusted_hosts(),
            self.write_format_controls(),
        ):
            emitted = True
            yield line
        if emitted:
            yield ""

    def _iter_lines(
        self,
        results: set[InstallRequirement],
        unsafe_requirements: set[InstallRequirement],
        unsafe_packages: set[str],
        markers: dict[str, Marker],
        hashes: dict[InstallRequirement, set[str]] | None = None,
    ) -> Iterator[str]:
        unsafe_packages = unsafe_packages if self.allow_unsafe else set()
        hashes = hashes or {}
        warn_uninstallable = False
        has_hashes = hashes and any(hash for hash in hashes.values())
        yielded = False

        for line in self.write_header():
            yield line
            yielded = True
        for line in self.write_flags():
            yield line
            yielded = True

        unsafe_requirements = unsafe_requirements or {
            r for r in results if r.name in unsafe_packages
        }
        packages = {r for r in results if r.name not in unsafe_packages}

        if packages:
            for ireq in sorted(packages, key=self._sort_key):
                if has_hashes and not hashes.get(ireq):
                    yield MESSAGE_UNHASHED_PACKAGE
                    warn_uninstallable = True
                line = self._format_requirement(
                    ireq, markers.get(key_from_ireq(ireq)), hashes=hashes
                )
                yield line
            yielded = True

        if unsafe_requirements:
            yield ""
            yielded = True
            if has_hashes and not self.allow_unsafe:
                yield MESSAGE_UNSAFE_PACKAGES_UNPINNED
                warn_uninstallable = True
            else:
                yield MESSAGE_UNSAFE_PACKAGES

            for ireq in sorted(unsafe_requirements, key=self._sort_key):
                ireq_key = key_from_ireq(ireq)
                if not self.allow_unsafe:
                    yield comment(f"# {ireq_key}")
                else:
                    line = self._format_requirement(
                        ireq, marker=markers.get(ireq_key), hashes=hashes
                    )
                    yield line

        if not yielded:
            yield ""

        if warn_uninstallable:
            log.warning(MESSAGE_UNINSTALLABLE)

    def write(
        self,
        results: set[InstallRequirement],
        unsafe_requirements: set[InstallRequirement],
        unsafe_packages: set[str],
        markers: dict[str, Marker],
        hashes: dict[InstallRequirement, set[str]] | None,
    ) -> None:
        if not self.dry_run:
            dst_file = io.TextIOWrapper(
                self.dst_file,
                encoding="utf-8",  # Fixed to standard encoding
                newline=self.linesep,
                line_buffering=True,
            )
        try:
            for line in self._iter_lines(
                results, unsafe_requirements, unsafe_packages, markers, hashes
            ):
                if self.dry_run:
                    log.info(line)  # Consistent logging
                else:
                    log.info(line)
                    dst_file.write(unstyle(line))
                    dst_file.write("\n")
        finally:
            if not self.dry_run:
                dst_file.detach()

    def _format_requirement(
        self,
        ireq: InstallRequirement,
        marker: Marker | None = None,
        hashes: dict[InstallRequirement, set[str]] | None = None,
    ) -> str:
        ireq_hashes = (hashes if hashes is not None else {}).get(ireq)

        line = format_requirement(ireq, marker=marker, hashes=ireq_hashes)
        if self.strip_extras:
            line = strip_extras(line)

        if not self.annotate:
            return line

        required_by = set()
        if hasattr(ireq, "_source_ireqs"):
            required_by |= {
                _comes_from_as_string(src_ireq.comes_from)
                for src_ireq in ireq._source_ireqs
                if src_ireq.comes_from
            }

        if ireq.comes_from and (
            isinstance(ireq.comes_from, str) or ireq.comes_from.name != ireq.name
        ):
            required_by.add(_comes_from_as_string(ireq.comes_from))

        required_by |= set(getattr(ireq, "_required_by", set()))

        if required_by:
            if self.annotation_style == "split":
                annotation = annotation_style_split(required_by)
                sep = "\n    "
            elif self.annotation_style == "line":
                annotation = annotation_style_line(required_by)
                sep = "\n    " if ireq_hashes else "  "
            else:
                raise ValueError(f"Invalid annotation style: {self.annotation_style}")  # Removed pragma
            if self.strip_extras:
                annotation = strip_extras(annotation)
            lines = f"{line:24}{sep}{comment(annotation)}".splitlines()
            line = "\n".join(ln.rstrip() for ln in lines)

        return line 

[specialorange]

Thanks. please go ahead - nop attribution required. I am having a bit of difficulty for some reason submitting my own. Thanks for taking care of the lightweight!

[AndydeCleyre]

@Varun786223 Are you an AI bot?

[Varun786223]

Thanks. please go ahead - nop attribution required. I am having a bit of difficulty for some reason submitting my own. Thanks for taking care of the lightweight!

It's okay although I am just 1st year cse(Ai)student I am glad I help u with my knowledge

[Varun786223]

@Varun786223 Are you an AI bot?

Just an 1st year student in cse(ai)

[WhyNotHugo]

Plenty of scenarios assume that the output of pip-tools is deterministic. For example, some CI checks try to rebuild it and ensure that the output has not changed, as a mechanism to ensure that it does not need updating. Adding the current date would break such checks, since the output always changes.

[specialorange]

Can't the CI tools be flagged to ignore comments? The comments section output of this tool includes security warnings that will presumably change over time. And the output dependency tree can presumably change without the version change with a package minor update.

Regardless, could this be included in a verbose requirements.txt vs a terse with no comments? Ie when building for CI use the flag '--strip-extras option'? Not sure where the line is drawn for this tool. Thanks for your input and ideas :)

[webknjaz]

Tooling usually runs a simple diff and is not aware of the semantics of individual lines. Of course, it could be made more complicated. But then again, it's much easier to just check the timestamp of last modification 🤷‍♂️

Still, I'm not convinced, there's a use case for this. But if this were to be accepted, it would've had to be behind a feature toggle.

[AndydeCleyre]

Or it could come in form of an option to specify a custom command to generate the header, though that could be an avenue for danger.

[specialorange]

Tooling usually runs a simple diff and is not aware of the semantics of individual lines. Of course, it could be made more complicated. But then again, it's much easier to just check the timestamp of last modification 🤷‍♂️

Still, I'm not convinced, there's a use case for this. But if this were to be accepted, it would've had to be behind a feature toggle.

(I might be misreading your comma placement, however) That's the point - the current notifications already tell you how it's generated, therefore having and reinforcing semantic meaning. The requirements.in is terse without being pedantic. The use case is to see when updates were done to the packages at all by looking at when the file was compiled and assumes relatively similar commit datetimes. I can think of some debugging cases where the compile time differing from the commit time would help development or deployment, but less so in error attribution. I am not aware of a record log of pip install that defaults without a --log flag, so the argument that the community sees logging as flaggable is certainly.

having a timestamp in situ as a base case is far easier than finding commit times on a project or implementing CI. as for a deterministic or functional CI implementation, there likely will be a solution that the community can come to.

Seeing there are global settings for both main tools, and there is a section for dev-requirements.in and CUSTOM_COMPILE_COMMAND, maybe a proposed flag can be included as well to avoid shell level aliases when run.