warmstart doc

Author: statmlben

doc/source/tutorials.rst

@@ -22,9 +22,6 @@ See `Manual ReHLine Formulation`_ documentation for more details and examples on
 
 Moreover, the following specific classes of formulations can be directly solved by `ReHLine`.
 
-- **Empirical Risk Minimization** (ERM) with various loss functions, see `ReHLine: Empirical Risk Minimization`_.
-- **Matrix Factorization** (MF) with with various loss functions, see `ReHLine: Matrix Factorization`_.
-
 List of Tutorials
 =================
 
@@ -40,14 +37,14 @@ List of Tutorials
    - | `ReHLine <./autoapi/rehline/index.html#rehline.ReHLine>`_
    - | ReHLine minimization with manual parameter settings.
 
- * - `ReHLine: Ridge Composite Quantile Regression <./examples/CQR.ipynb>`_
-   - | `CQR_Ridge <./autoapi/rehline/index.html#rehline.CQR_Ridge>`_
-   - | Composite Quantile Regression (CQR) with a ridge penalty.
-
  * - `ReHLine: Empirical Risk Minimization <./tutorials/ReHLine_ERM.rst>`_
    - | `plqERM_Ridge <./autoapi/rehline/index.html#rehline.plqERM_Ridge>`_
    - | Empirical Risk Minimization (ERM) with a piecewise linear-quadratic (PLQ) objective with a ridge penalty.
 
+ * - `ReHLine: Ridge Composite Quantile Regression <./examples/CQR.ipynb>`_
+   - | `CQR_Ridge <./autoapi/rehline/index.html#rehline.CQR_Ridge>`_
+   - | Composite Quantile Regression (CQR) with a ridge penalty.
+
  * - `ReHLine: Matrix Factorization <./tutorials/ReHLine_MF.rst>`_
    - | `plqMF_Ridge <./autoapi/rehline/index.html#rehline.plqERM_Ridge>`_
    - | Matrix Factorization (MF) with a piecewise linear-quadratic (PLQ) objective with a ridge penalty.
@@ -60,4 +57,5 @@ List of Tutorials
    ./tutorials/ReHLine_ERM
    ./tutorials/loss
    ./tutorials/constraint
+   ./tutorials/warmstart
 

doc/source/tutorials/warmstart.rst

@@ -0,0 +1,94 @@
+Warm-Starting with ReHLine
+==========================
+
+This tutorial explains how to use warm-starting with ReHLine, a Python library for regression with hinge loss, to enhance the efficiency of solving similar optimization problems.
+
+Introduction
+------------
+
+Warm-starting is a technique used to accelerate the convergence of optimization algorithms by initializing them with a solution from a previous run. This is particularly beneficial when you have a sequence of related problems to solve.
+
+Setup
+-----
+
+Before you begin, ensure you have the necessary packages installed. You need the `rehline` library, which is used for regression with hinge loss, and `numpy` for numerical operations. Install these packages using pip if you haven't already:
+
+.. code-block:: bash
+
+    pip install rehline numpy
+
+Simulating the Dataset
+----------------------
+
+We first create a random dataset for classification:
+
+.. code-block:: python
+
+    import numpy as np
+
+    n, d, C = 1000, 3, 0.5
+    X = np.random.randn(n, d)
+    beta0 = np.random.randn(d)
+    y = np.sign(X.dot(beta0) + np.random.randn(n))
+
+- **n** is the number of samples.
+- **d** is the number of features.
+- **C** is a regularization parameter.
+- **X** is the feature matrix.
+- **y** is the target vector, generated as a sign function of a linear combination of features plus some noise.
+
+Using ReHLine Solver
+--------------------
+
+The `ReHLine_solver` is tested first with a cold start and then with a warm start:
+
+.. code-block:: python
+
+    from rehline._base import ReHLine_solver
+
+    U = -(C*y).reshape(1,-1)
+    V = (C*np.array(np.ones(n))).reshape(1,-1)
+    res = ReHLine_solver(X, U, V)  # Cold start
+    res_ws = ReHLine_solver(X, U, V, Lambda=res.Lambda)  # Warm start
+
+- **Cold Start**: The solver starts from scratch without any prior information.
+- **Warm Start**: The solver uses the solution from the cold start (`res.Lambda`) as the initial point for the next run.
+
+Using ReHLine Class
+-------------------
+
+The `ReHLine` class is used to fit a model:
+
+.. code-block:: python
+
+    from rehline import ReHLine
+
+    clf = ReHLine(verbose=1)
+    clf.C = C
+    clf.U = -y.reshape(1,-1)
+    clf.V = np.array(np.ones(n)).reshape(1,-1)
+    clf.fit(X)  # Cold start
+
+    clf.C = 2*C
+    clf.warm_start = 1
+    clf.fit(X)  # Warm start
+
+- **Cold Start**: The class is fitted with the initial data.
+- **Warm Start**: The class is fitted again with a different regularization parameter (`2*C`), using the previous solution as a starting point.
+
+Using plqERM_Ridge
+------------------
+
+Finally, the `plqERM_Ridge` class is tested similarly:
+
+.. code-block:: python
+
+    from rehline import plqERM_Ridge
+
+    clf = plqERM_Ridge(loss={'name': 'svm'}, C=C, verbose=1)
+    clf.fit(X=X, y=y)  # Cold start
+
+    clf.C = 2*C
+    clf.warm_start = 1
+    clf.fit(X=X, y=y)  # Warm start
+

to-do.md

@@ -1,7 +1,8 @@
 # To-do list
 
 ## src
-- [ ] warmstarting GridSearchCV
+- [x] warmstarting 
+- [ ] GridSearchCV
 
 ## Class
 - [ ] sklearn Classifier and Regressor Estimator
@@ -10,6 +11,7 @@
 ## Loss
 - [ ] MAE
 - [ ] TV
+- [ ] Hinge_squared
 
 ## Constraint
 - [ ] box constraints