Add CMAES optimizer from nevergrad and refactor existing code

.pre-commit-config.yaml

@@ -29,6 +29,7 @@ repos:
         additional_dependencies:
           - hatchling
           - ruff
+          - iminuit
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v5.0.0
     hooks:

.tools/envs/testenv-linux.yml

@@ -33,6 +33,7 @@ dependencies:
       - DFO-LS>=1.5.3  # dev, tests
       - Py-BOBYQA  # dev, tests
       - fides==0.7.4  # dev, tests
+      - nevergrad  # dev, tests
       - kaleido  # dev, tests
       - pandas-stubs  # dev, tests
       - types-cffi  # dev, tests

.tools/envs/testenv-numpy.yml

@@ -31,6 +31,7 @@ dependencies:
       - DFO-LS>=1.5.3  # dev, tests
       - Py-BOBYQA  # dev, tests
       - fides==0.7.4  # dev, tests
+      - nevergrad  # dev, tests
       - kaleido  # dev, tests
       - types-cffi  # dev, tests
       - types-openpyxl  # dev, tests

.tools/envs/testenv-others.yml

@@ -31,6 +31,7 @@ dependencies:
       - DFO-LS>=1.5.3  # dev, tests
       - Py-BOBYQA  # dev, tests
       - fides==0.7.4  # dev, tests
+      - nevergrad  # dev, tests
       - kaleido  # dev, tests
       - pandas-stubs  # dev, tests
       - types-cffi  # dev, tests

.tools/envs/testenv-pandas.yml

@@ -31,6 +31,7 @@ dependencies:
       - DFO-LS>=1.5.3  # dev, tests
       - Py-BOBYQA  # dev, tests
       - fides==0.7.4  # dev, tests
+      - nevergrad  # dev, tests
       - kaleido  # dev, tests
       - types-cffi  # dev, tests
       - types-openpyxl  # dev, tests

docs/source/algorithms.md

@@ -3984,8 +3984,6 @@ iminuit).
       - Values greater than 1 specify the maximum number of restart attempts.  
 ```
 
-(nevergrad-algorithms)=
-
 ## Nevergrad Optimizers
 
 optimagic supports some algorithms from the
@@ -4041,6 +4039,138 @@ these optimizers, you need to have
       for speed. Default is False.
     - **special_speed_quasi_opp_init** (bool): Whether to use special quasi-opposition
       initialization for speed. Default is False.
+
+```
+
+```{eval-rst}
+.. dropdown::  nevegrad_cmaes
+
+    .. code-block::
+
+        "nevergrad_cmaes"
+
+    Minimize a scalar function using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) algorithm.
+
+    The Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is a stochastic derivative-free numerical optimization algorithm for difficult (non-convex, ill-conditioned, multi-modal, rugged, noisy) optimization problems in continuous search spaces.
+
+    The version available through nevergrad wraps an external implementation `pycma <https://github.com/CMA-ES/pycma>`_.
+
+    The original method can be found in
+    :cite:`Hansen2023`.
+
+    The fast implementation relies on fcmaes which can be installed with `pip install fcmaes`.
+
+    - **scale** (float): Scale of the search.  
+      - Default: `1.0`
+
+    - **elitist** (bool): Whether to switch to elitist mode (μ,λ)-CMA-ES, where the best point in the population is always retained.  
+      - Default: `False`
+
+    - **population_size** (Optional[int]): Population size. If None, it will be set to:
+
+      .. math::
+          \max\left(\text{num_workers}, 4 + \left\lfloor 3 \cdot \log(\text{dimension}) \right\rfloor\right)
+
+    - **diagonal** (bool): Use the diagonal version of CMA, which is more efficient for high-dimensional problems.  
+      - Default: `False`
+
+    - **high_speed** (bool): Use a metamodel for recommendation to speed up optimization.  
+      - Default: `False`
+
+    - **fast_cmaes** (bool): Use the fast CMA-ES implementation.  
+      - Cannot be used with `diagonal=True`.  
+      - Default: `False`
+
+    - **random_init** (bool): If True, initialize the optimizer with random parameters.  
+      - Default: `False`
+
+    - **n_cores** (int): Number of cores to use for parallel function evaluation.  
+      - Default: `1`
+
+    - **step_size_adaptive** (bool | str): Whether to adapt the step size. Can be a boolean or a string specifying the adaptation strategy.  
+      - Default: `True`
+
+    - **step_size_damping_factor** (float): Damping factor for step size adaptation.  
+      - Default: `1.0`
+
+    - **step_size_damping_rate** (float): Damping rate for step size adaptation.  
+      - Default: `0.1`
+
+    - **step_size_update_squared** (bool): Whether to use squared step sizes in updates.  
+      - Default: `False`
+
+    - **learning_rate_cov_mat_update** (float): Learning rate for the covariance matrix update.  
+      - Default: `1.0`
+
+    - **learning_rate_rank_one_update** (float): Multiplier for the rank-one update learning rate of the covariance matrix.  
+      - Default: `1.0`
+
+    - **learning_rate_rank_mu_update** (float): Multiplier for the rank-mu update learning rate of the covariance matrix.  
+      - Default: `1.0`
+
+    - **learning_rate_mean_update** (float): Learning rate for the mean update.  
+      - Default: `1.0`
+
+    - **learning_rate_diagonal_update** (float): Learning rate for the diagonal update.  
+      - Default: `0.0`
+
+    - **num_parents** (Optional[int]): Number of parents (μ) for recombination.  
+      - Default: `None` (automatically determined)
+
+    - **negative_update** (bool): Whether to use negative updates for the covariance matrix.  
+      - Default: `True`
+
+    - **mirror_sampling_strategy** (int): Strategy for mirror sampling.  
+      - `0`: Unconditional mirroring  
+      - `1`: Selective mirroring  
+      - `2`: Selective mirroring with delay (default)
+
+    - **normalize_cov_trace** (bool | str): How to normalize the trace of the covariance matrix.  
+      - `False`: No normalization  
+      - `True`: Normalize to 1  
+      - `"arithm"`: Arithmetic mean normalization  
+      - `"geom"`: Geometric mean normalization  
+      - `"aeig"`: Arithmetic mean of eigenvalues  
+      - `"geig"`: Geometric mean of eigenvalues  
+      - Default: `False`
+
+    - **diag_covariance_iters** (int | bool): Number of iterations to use diagonal covariance matrix before switching to full matrix.  
+      - If `False`, always use full matrix.  
+      - Default: `False`
+
+    - **stopping_maxfun** (int): Maximum number of function evaluations before termination.  
+      - Default: `STOPPING_MAXFUN_GLOBAL`
+
+    - **stopping_maxiter** (int): Maximum number of iterations before termination.  
+      - Default: `STOPPING_MAXITER`
+
+    - **stopping_timeout** (float): Maximum time in seconds before termination.  
+      - Default: `float("inf")`
+
+    - **stopping_cov_mat_cond** (float): Maximum condition number of the covariance matrix before termination.  
+      - Default: `1e14`
+
+    - **convergence_ftol_abs** (float): Absolute tolerance on function value changes for convergence.  
+      - Default: `1e-11`
+
+    - **convergence_ftol_rel** (float): Relative tolerance on function value changes for convergence.  
+      - Default: `0.0`
+
+    - **convergence_xtol_abs** (float): Absolute tolerance on parameter changes for convergence.  
+      - Default: `1e-11`
+
+    - **convergence_iter_noimprove** (Optional[int]): Number of iterations without improvement before termination.  
+      - Default: `None`
+
+    - **invariant_path** (bool): Whether evolution path (pc) should be invariant to transformations.  
+      - Default: `False`
+
+    - **eval_final_mean** (bool): Whether to evaluate the final mean solution.  
+      - Default: `True`
+
+    - **seed** (Optional[int]): Seed used by the internal random number generator for reproducibility.  
+      - Default: `None`
+
 ```
 
 ## References

docs/source/refs.bib

@@ -906,6 +906,17 @@ @article{JAMES1975343
 author = {F. James and M. Roos}
 }
 
+
+@misc{Hansen2023,
+title={The CMA Evolution Strategy: A Tutorial}, 
+author={Nikolaus Hansen},
+year={2023},
+eprint={1604.00772},
+archivePrefix={arXiv},
+primaryClass={cs.LG},
+url={https://arxiv.org/abs/1604.00772}, 
+}
+
 @InProceedings{Kennedy1995,
   author={Kennedy, J. and Eberhart, R.},
   booktitle={Proceedings of ICNN'95 - International Conference on Neural Networks}, 

environment.yml

@@ -43,6 +43,7 @@ dependencies:
       - DFO-LS>=1.5.3  # dev, tests
       - Py-BOBYQA  # dev, tests
       - fides==0.7.4  # dev, tests
+      - nevergrad  # dev, tests
       - kaleido  # dev, tests
       - pre-commit>=4  # dev
       - -e .  # dev

pyproject.toml

@@ -17,6 +17,7 @@ dependencies = [
     "annotated-types",
     "typing-extensions",
     "iminuit",
+    "nevergrad",
 ]
 dynamic = ["version"]
 keywords = [
@@ -350,6 +351,7 @@ module = [
     "cyipopt",
     "nlopt",
     "bokeh",
+    "nevergrad",
     "bokeh.layouts",
     "bokeh.models",
     "bokeh.plotting",