How to convert npy file?

[minminjao]

I wonder how you did it with the configured code when you changed the dataset to npy file. Looking forward to other people's help and advice.

[minminjao]

about this repo 'https://github.com/qiboteam/qibo/tree/master/examples/anomaly_detection'

[alecandido]

It seems pretty clear how the file is being loaded

qibo/examples/anomaly_detection/train.py

Lines 114 to 115 in d10c1e3

	file_dataset = LOCAL_FOLDER / "data" / "standard_data.npy"
	dataset_np = np.load(file_dataset)

It is a NumPy-specific format, but NumPy is used widely enough to be a reasonably assumed dependency in Python code (and somewhat possible to load even beyond Python itself).

I'm possibly missing the question

[minminjao]

I wonder how you changed the csv file, which is a dataset, to the npy file. I'm trying to apply the below dataset here, but it doesn't work.

'https://zenodo.org/records/8202802'
@alecandido

[alecandido]

Converting a CSV file to NumPy format is simple with Python, and the only dependency you need is NumPy itself.

Essentially:

import numpy as np

np.save("ciao.npy", np.loadtxt("ciao.csv"))

However, you do not have to do that. What you need is often an array in memory. If your original dataset is a CSV file, you can also keep it as it is, and just load with np.loadtxt() (or Pandas, or whatever else you prefer).

Just aim for the np.ndarray object :)

[alecandido]

@minminjao I converted your question as a discussion, since it's not an issue strictly related to the Qibo software itself. However, no problem in continuing discussing it here :)

[minminjao]

I'm asking if you mean to use the csv file as it is, to change the part where the file path is called in the qibo-anomaly detection part. If possible, I'd like to ask if I can get an example and send you a dataset.

ex)

import pandas as pd
df = pd.read_csv("ciao.csv")

[alecandido]

So, if you have to load it by yourself, np.loadtxt() has exactly the signature you're looking for: reading a CSV file into a np.ndarray.

The advice is to just change the name of the file in the code (possibly the full path). And if you do not have a .npy file, you just need to change also the method, from np.load() (supposed to load .npy files) to np.loadtxt() (tailored for CSV-like)

If you want to use the Pandas reader, you just have to extract the values of the underlying array

array = pd.read_csv("ciao.csv").values

[minminjao]

Or do I just need to change it through the code you gave me? When I researched the data, I asked you to confirm that you need to apply 'MinMaxScaler' to introduce it to qibo.

Or I ask if you mean to do as below as you advised.

file_dataset = LOCAL_FOLDER / "data" / "standard_data.csv"
dataset_np = np.loadtxt(file_dataset, delimiter=',')

@alecandido

[minminjao]

Should I operate it based on the changed code like the one below?

# Other hyperparameters
n_qubits = 6
q_compression = 3

# Load and pre-process data
file_dataset = LOCAL_FOLDER / "data" / "standard_data.csv"
dataset_np = np.loadtxt(file_dataset, delimiter=',')
dataset = tf.convert_to_tensor(dataset_np)
train = dataset[0:train_size]

# Initialize random parameters
n_params = (n_layers * n_qubits + q_compression) * 3
params = tf.Variable(tf.random.normal((n_params,)))

[minminjao]

As shown in the picture above, the csv file is configured.

@alecandido

[alecandido]

Should I operate it based on the changed code like the one below?

Yes, that's what I expect. Though you may have to change the amount of qubits and parameters according to your dataset (or define a certain mapping). Refer to https://github.com/qiboteam/qibo/tree/d10c1e384de9740511c13d900967c35e705bb787/examples/anomaly_detection#how-to-run-an-example

As shown in the picture above, the csv file is configured.

Ok, I fear the example is expecting a purely numerical data set. So, you may have to take into account a categorical or mapping (or some other one) by yourself, in order to reapply the rest of the steps.

[minminjao]

I'm asking if I can do it like the code below.
@alecandido

import pandas as pd
from sklearn.preprocessing import LabelEncoder, StandardScaler


csv_path = LOCAL_FOLDER / "data" / "standard_data.csv"
df = pd.read_csv(csv_path)


for col in ['src_ip', 'dst_ip', 'protocol']:
    le = LabelEncoder()
    df[col] = le.fit_transform(df[col])


scaler = StandardScaler()
dataset_np = scaler.fit_transform(df)  # numpy array


dataset = tf.convert_to_tensor(dataset_np, dtype=tf.float32)
train = dataset[:train_size]

[alecandido]

The best way to check if you can do is to run your code on your file, and compare the result (train) with what you would get from the original code on the original file (just loading the .npy and truncating it).

If you obtain two homogeneous arrays (not same numbers, of course - but same number of dimensions, for example), that is fine (almost for sure).

(irrelevant advice over your code: if you keep fitting it, and you use just once every time, you may not need to instantiate more than one LabelEncoder, so you could lift the constructor out of the loop)