Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@ Signal processing primitives for the [ezmsg](https://www.ezmsg.org) message-pass
* **Windowing** - Sliding windows and buffering utilities
* **Math operations** - Arithmetic, log, abs, difference, and more
* **Signal generation** - Synthetic signal generators
* More! Brows the API documentation for more details.

All modules use `AxisArray` as the primary data structure for passing signals between components.
All modules use [`AxisArray`](https://www.ezmsg.org/ezmsg/reference/API/axisarray.html) as the primary data structure for passing signals between components. The default data backend is NumPy, but other backends are supported via the Array API such as CuPy and MLX.

## Installation

Expand Down
1 change: 1 addition & 0 deletions docs/source/_templates/autosummary/module.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{{ fullname | escape | underline}}

.. automodule:: {{ fullname }}
:no-members:

{% block attributes %}
{% if attributes %}
Expand Down
3 changes: 2 additions & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
"sphinx.ext.intersphinx",
"sphinx.ext.viewcode",
"sphinx.ext.duration",
"sphinx.ext.graphviz",
# "sphinx_autodoc_typehints", # Disabled due to compatibility issue
"sphinx_copybutton",
"myst_parser", # For markdown files
Expand Down Expand Up @@ -70,7 +71,7 @@
intersphinx_mapping = {
"python": ("https://docs.python.org/3/", None),
"numpy": ("https://numpy.org/doc/stable/", None),
"scipy": ("https://scipy.org/doc/scipy/", None),
"scipy": ("https://docs.scipy.org/doc/scipy/", None),
"ezmsg": ("https://www.ezmsg.org/ezmsg/", None),
"ezmsg.baseproc": ("https://www.ezmsg.org/ezmsg-baseproc/", None),
"ezmsg.learn": ("https://www.ezmsg.org/ezmsg-learn/", None),
Expand Down
8 changes: 4 additions & 4 deletions docs/source/guides/HybridBuffer.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
## HybridBuffer
# HybridBuffer

The HybridBuffer is a stateful, FIFO buffer that combines a deque for fast appends with a contiguous circular buffer for efficient, advancing reads. The synchronization between the deque and the circular buffer can be immediate, upon threshold reaching, or on demand, allowing for flexible data management strategies.

This buffer is designed to be agnostic to the array library used (e.g., NumPy, CuPy, PyTorch) via the Python Array API standard.

### Basic Reading and Writing Behaviour
## Basic Reading and Writing Behaviour

The following diagram illustrates the states of the HybridBuffer across data writes and reads when `update_strategy="on_demand"`:

Expand Down Expand Up @@ -39,7 +39,7 @@ H. We then `read(4)` again. This time, a `flush()` is not triggered because we h

Note: `peek(n)` and `seek(n)`, where `n` > `n_available` will raise an error. However, `peek(None)` will return all available samples without error, and `seek(None)` will advance the tail to the end of the available data.

### Overflow Behaviour
## Overflow Behaviour

The criteria to trigger an overflow are as follows:
* the deque has more data than there is space in the circular buffer, where space is the combination of previously read samples and unwritten samples in the circular buffer.
Expand Down Expand Up @@ -70,7 +70,7 @@ There are a few mitigations to defer flushing to help prevent overflows:
* Be cautious relying on repeated calls to `peek_at(k, allow_flush=False)` as it scans over the items in the deque which can be slow.
* When calling `read(n)`, if a flush is necessary, and it will cause an overflow, and the overflow could be prevented with a pre-emptive read up to `n`, then it will do the read in 2 parts. First it will call `peek(n_unread_in_buffer)` and `seek(n_unread_in_buffer)` to read the unread samples in the circular buffer. Second, it will call `peek(n_remaining)` and `seek(n_remaining)` to trigger a flush -- which should no longer cause an overflow -- then read the remaining requested samples and stitch them together.

### Advanced Pointer Manipulation
## Advanced Pointer Manipulation

The previous section describes how `read`, `peek`, `seek`, and `peek_at` function in normal use cases. It is also possible to call `seek` with a negative value, which will attempt to move the tail pointer backwards over previously-read (or previously sought-over) data by that many samples. `seek` returns the number of samples that were actually moved, which may be less than the requested value if there was insufficient room. Negative seeks can only rewind into previously read data, and positive seeks can only advance into unread data, possibly including data that gets flushed from the deque.

Expand Down
24 changes: 10 additions & 14 deletions docs/source/guides/explanations/sigproc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ It also comes with a collection of pre-built signal processing classes and relev
A list of available signal processors and ezmsg Units can be found in the (TBD) `ezmsg-sigproc reference <http://sigproc.ezmsg.org>`_.


|ezmsg_logo_small| Rationale For Implementation
Rationale For Implementation
********************************************************

Providing a flexible and extensible framework for signal processing tasks makes it
Expand All @@ -20,7 +20,7 @@ Providing a flexible and extensible framework for signal processing tasks makes
- allows standalone use outside of an ezmsg context


|ezmsg_logo_small| How to decide which processor template to use?
How to decide which processor template to use?
******************************************************************

We use the term "processor" to refer to any class that processes signals. We then separate processors into types based on whether or not they receive input messages (typically signal data), send output messages, or both:
Expand Down Expand Up @@ -94,7 +94,7 @@ The decision tree for this classification is as follows:

The leaf nodes in yellow are abstract base classes provided in `ezmsg.sigproc.base` for implementing standalone processors. The table below summarizes these base classes.

|ezmsg_logo_small| Abstract implementations (Base Classes) for standalone processors
Abstract implementations (Base Classes) for standalone processors
***************************************************************************************


Expand Down Expand Up @@ -242,7 +242,7 @@ NOTES:
do not inherit from ``BaseStatefulProcessor`` and ``BaseStatefulProducer``. They accomplish statefulness by inheriting from the mixin abstract base class ``CompositeStateful``, which implements the state related methods: ``get_state_type``, ``state.setter``, ``state.getter``, ``_hash_message``, ``_reset_state``, and ``stateful_op`` (as well as composite processor chain related methods). However, ``BaseStatefulProcessor``, ``BaseStatefulProducer`` implement ``stateful_op`` method for a single processor in an incompatible way to what is required for composite chains of processors.


|ezmsg_logo_small| Implementing a custom standalone processor
Implementing a custom standalone processor
****************************************************************

1. Create a new settings dataclass: ``class MySettings(ez.Settings):``
Expand All @@ -268,7 +268,7 @@ do not inherit from ``BaseStatefulProcessor`` and ``BaseStatefulProducer``. They
* ``ClockProducer`` overrides ``__call__`` in order to provide a synchronous call bypassing the default async behaviour.


|ezmsg_logo_small| Abstract implementations (Base Classes) for ezmsg Units using processors
Abstract implementations (Base Classes) for ezmsg Units using processors
**********************************************************************************************

Generic TypeVars for ezmsg Units
Expand Down Expand Up @@ -314,7 +314,7 @@ Base Classes for ezmsg processor Units:

Note, it is strongly recommended to use `BaseConsumerUnit`, `BaseTransformerUnit`, or `BaseAdaptiveTransformerUnit` for implementing concrete subclasses rather than `BaseProcessorUnit`.

|ezmsg_logo_small| How to implement a custom ezmsg Unit from a standalone processor
How to implement a custom ezmsg Unit from a standalone processor
=====================================================================================

1. Create and test custom standalone processor as above.
Expand Down Expand Up @@ -353,13 +353,9 @@ Often, all that is required is the following (e.g., for a custom transformer):

.. note:: The type of ProcessorUnit is based on the internal processor and not the input or output of the unit. Input streams are allowed in ProducerUnits and output streams in ConsumerUnits. For an example of such a use case, see ``BaseCounterFirstProducerUnit`` and its subclasses. ``BaseCounterFirstProducerUnit`` has an input stream that receives a flag signal from a clock that triggers a call to the internal producer.

|ezmsg_logo_small| See Also
See Also
********************************

1. `Signal Processor Documentation <sigproc_processor_documentation>`_
#. `Signal Processing Tutorial <../../tutorials/signalprocessing.html>`_
#. `Signal Processing HOW TOs <../../how-tos/signalprocessing/main.html>`_

.. |ezmsg_logo_small| image:: ../_static/_images/ezmsg_logo.png
:width: 40
:alt: ezmsg logo
#. :doc:`Signal Processing Tutorial <../tutorials/signalprocessing>` — end-to-end walkthrough building a processor and Unit
#. `ezmsg-baseproc documentation <https://www.ezmsg.org/ezmsg-baseproc/>`_ — base class API reference and implementation guide
#. :doc:`ezmsg-sigproc API reference <../../api/index>` — autogenerated reference for all sigproc modules
4 changes: 0 additions & 4 deletions docs/source/guides/how-tos/signalprocessing/adaptive.rst

This file was deleted.

4 changes: 0 additions & 4 deletions docs/source/guides/how-tos/signalprocessing/checkpoint.rst

This file was deleted.

6 changes: 0 additions & 6 deletions docs/source/guides/how-tos/signalprocessing/composite.rst

This file was deleted.

This file was deleted.

8 changes: 0 additions & 8 deletions docs/source/guides/how-tos/signalprocessing/processor.rst

This file was deleted.

4 changes: 0 additions & 4 deletions docs/source/guides/how-tos/signalprocessing/standalone.rst

This file was deleted.

8 changes: 0 additions & 8 deletions docs/source/guides/how-tos/signalprocessing/stateful.rst

This file was deleted.

41 changes: 0 additions & 41 deletions docs/source/guides/how-tos/signalprocessing/unit.rst

This file was deleted.

5 changes: 3 additions & 2 deletions docs/source/guides/sigproc/architecture.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,10 @@ Key Modules
Where to Learn Next
-------------------

* Study the :doc:`base` page to master the processor architecture.
* Read the :doc:`../explanations/sigproc` explainer to master the processor architecture.
* Follow the :doc:`../tutorials/signalprocessing` to build a processor and Unit from scratch.
* See the `ezmsg-baseproc documentation <https://www.ezmsg.org/ezmsg-baseproc/>`_ for the base class API and implementation details.
* Explore unit tests in the repository for hands-on examples of composing processors and Units.
* Review the `ezmsg framework <https://www.ezmsg.org>`_ to understand the surrounding ecosystem.
* Experiment with the code—try running processors standalone and then integrate them into a small pipeline to observe the trade-offs firsthand.

This approach equips newcomers to choose the right level of abstraction—raw processor, Unit wrapper, or full pipeline—based on the demands of their analysis or streaming application.
19 changes: 0 additions & 19 deletions docs/source/guides/sigproc/base.rst

This file was deleted.

18 changes: 1 addition & 17 deletions docs/source/guides/sigproc/content-sigproc.rst
Original file line number Diff line number Diff line change
@@ -1,25 +1,9 @@
ezmsg-sigproc
===============

Timeseries signal processing implementations in ezmsg, leveraging numpy and scipy.
Most of the methods and classes in this extension are intended to be used in building signal processing pipelines.
They use :class:`ezmsg.util.messages.axisarray.AxisArray` as the primary data structure for passing signals between components.
The message's data are typically NumPy arrays, though many transformers support the
:doc:`Array API standard <../explanations/array_api>` for use with CuPy, PyTorch, and other backends.

.. note:: Some generators might yield valid :class:`AxisArray` messages with ``.data`` size of 0.
This may occur when the generator receives inadequate data to produce a valid output, such as when windowing or buffering.

`ezmsg-sigproc` contains two types of modules:

- base processors and units that provide fundamental building blocks for signal processing pipelines
- in-built signal processing modules that implement common signal processing techniques

.. toctree::
:maxdepth: 1

architecture
base
units
processors
../explanations/sigproc
../explanations/array_api
Loading