Full_SVM 🛠️

This class implements a Support Vector Machine (SVM) from scratch, focusing on finding optimal hyperplanes (weights w and bias b) using a closed-form solution with entropy minimization. The implementation supports One-vs-One (OvO) multi-class strategies, making it flexible for various classification tasks.

Features ✨

• Closed-form solution for weight vector calculation
• Entropy-based bias selection for optimal decision boundaries
• Multi-class support with One-vs-One strategies
• Robust error handling and input validation
• Threshold filtering for bias calculation on projected data
• Support Two voting: Choose between confidence-weighted voting (using decision values) and pure majority voting.

Research Objectives 🎯

1. Develop a computationally efficient SVM
2. Explore entropy minimization as an alternative to margin maximization for bias selection
3. Create an educational implementation that demonstrates core SVM concepts

Mathematical Approach 📐

Weight Vector Calculation The weight vector w is computed as the normalized difference between class means: $w = (mean(X_{pos}) - mean(X_{neg})) / norm$

Bias Selection The optimal bias b is found by projecting all data points onto w and selecting the threshold T that minimizes the number of projected data points for finding the optimal b:

$H = -p_{pos} * log_2(p_{pos}) - p_{neg} * log_2(p_{neg})$

$b = -T \text{ where } T \text{ minimizes } (n_{left}*H_{left} + n_{right}*H_{right})/n_{total}$

Iterative Refinement Each iteration keeps only the points closest to the current decision boundary, focusing the classifier on potential support vectors.

Libraries 📚

• NumPy: Used for array manipulations, mean calculations, projections, and sorting operations.
• bisect: Used to efficiently find insertion points for threshold values when calculating the optimal bias, ensuring O(log n) complexity for sorted array operations.
• itertools: Used for generating combinations of class pairs in the One-vs-One strategy.

Installation 💻

Clone the repository:

!git clone https://github.com/Wissam2001/Full_SVM.git
cd Full_SVM
import Full_SVM

Run the code:

...
f_svm = Full_SVM.Full_SVM(num_iterations=5, threshold=0.5)
f_svm.fit(X_train,y_train)

# To predict with confidence-weighted voting (using decision values)
y_pridect = f_svm.predict(X_test)
#... and for Pure majority voting
y_pridect = f_svm.predict_with_voting((X_test)
...

Important⚠️ Make sure to install the used libraries for this project

Contributing 🤝

Further work can be done to:

• Add cross-validation support
• Optimize entropy calculation for large datasets

Contact ✉️

• Email: wissambadia4@gmail.com

• LinkedIn: Badia Ouissam Lakas