Overfitting when training lstm model!

@cspampin

Hi @cspampin, thanks for your great work firstly. Currently, I've trained the model (lstm) in your dataset. But after training for 100 epochs, the model was overfitting, TrL was decreasing while VL and TeL were increasing, and TeA at max VA = 0.1457 at epoch 28. I wonder how can I fix the problem to achieve good performance (as result in the paper).

Hi
I also have the same problem and same result.

Hi, the results in the paper are affected by slow frequency in EEG data. When filtering properly the data, the results for lstm are about 15-20% as you obtained.

Thanks for your response.
For the second part of the paper 2000 images are needed.
The ImageNet dataset is so large and I can’t download it to separate the images.
Did you uploaded these images in any repository?

We cannot release ImageNet data for licence issues. What we can do is to share the class names. Sorry.

Thanks for your response.
Yo reported above 80 % accuracy for test and validation sets.
How is that possible?
You said here he results for LSTM are about 15-20%

If you run the code on the raw data you will get 80% of accuracy, if you filter the data from 1Hz to 45Hz you obtain the 20% of accuracy.

Thanks a lot
I will test it.

@cspampin

@cspampin So, is the accuracy reported in the paper for CVPR and TPAMI using raw eeg signals?

@cspampin

@cspampin Is the file named "eeg_signals_raw_with_mean_std.pth" the raw data you mentioned? I still can not reproduce the results. @dara1400 what's your process? Thanks!

@YutLan

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper.

It will be helpful if you crosscheck this from your end.

@prajwalsingh

@prajwalsingh Thank you very much! I have downloaded the dataset. Does the raw data mean the file "eeg_signals_raw_with_mean_std.pth"?

@YutLan

@YutLan , Yes.

@prajwalsingh

@prajwalsingh Thank you! It will be helpfu! Maybe my implementation misused the early stop. I will crosscheck this from my end!

@prajwalsingh

By the way, Nice to meet you! @prajwalsingh I've actually read your team's eeg2image paper before. What a coincidence!

Thank you 😊, good to know that.

@YutLan

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper.
It will be helpful if you crosscheck this from your end.

Yeah, I found it works on raw data but not preprocessed data either. And I wonder why dose this phenomenon happen, I mean, the raw data looks too prefect (than other eeg dataset such as THINGS-eeg) isn't it?

In the PAMI paper, they address the problem with raw data that leads to overfitting (might be some noise/signal pattern is easy to identify be neural network) as well as there are some overlaps in the train and test data. Therefore authors proposed a method for processing filter 5-95Hz in the PAMI paper. You can preprocess the raw data from your end with a new train-val-test split without overlapping samples.

You can also check one of our works recently accepted in WACV24: Learning Robust Deep Visual Representations from EEG Brain Recordings [https://arxiv.org/abs/2310.16532]

Oh! Thank you for your guidance. I realized I was reading an older version of the paper. Does the new version propose corrections to the original experiments? I will read it to understand. I will also review your work. If there are areas I don't understand, I hope to seek your guidance. :)
Thanks again!

@YutLan

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper.
It will be helpful if you crosscheck this from your end.

Yeah, I found it works on raw data but not preprocessed data either. And I wonder why dose this phenomenon happen, I mean, the raw data looks too prefect (than other eeg dataset such as THINGS-eeg) isn't it?

In the PAMI paper, they address the problem with raw data that leads to overfitting (might be some noise/signal pattern is easy to identify be neural network) as well as there are some overlaps in the train and test data. Therefore authors proposed a method for processing filter 5-95Hz in the PAMI paper. You can preprocess the raw data from your end with a new train-val-test split without overlapping samples.

You can also check one of our works recently accepted in WACV24: Learning Robust Deep Visual Representations from EEG Brain Recordings [https://arxiv.org/abs/2310.16532]

I find something interesting 🤣 . In another TPAMI paper The Perils and Pitfalls of Block Design for EEG Classification Experiments, it said that "We take our correspondence with the authors to imply that no filtering was applied during acquisition, no filtering was applied prior to production of either the Python or Matlab format released data". And I have found some more papers which think this dataset has some flaws. Maybe I should change the dataset 😢

@YutLan

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper.
It will be helpful if you crosscheck this from your end.

Yeah, I found it works on raw data but not preprocessed data either. And I wonder why dose this phenomenon happen, I mean, the raw data looks too prefect (than other eeg dataset such as THINGS-eeg) isn't it?

In the PAMI paper, they address the problem with raw data that leads to overfitting (might be some noise/signal pattern is easy to identify be neural network) as well as there are some overlaps in the train and test data. Therefore authors proposed a method for processing filter 5-95Hz in the PAMI paper. You can preprocess the raw data from your end with a new train-val-test split without overlapping samples.
You can also check one of our works recently accepted in WACV24: Learning Robust Deep Visual Representations from EEG Brain Recordings [https://arxiv.org/abs/2310.16532]

I find something interesting 🤣 . In another TPAMI paper The Perils and Pitfalls of Block Design for EEG Classification Experiments, it said that "We take our correspondence with the authors to imply that no filtering was applied during acquisition, no filtering was applied prior to production of either the Python or Matlab format released data". And I have found some more papers which think this dataset has some flaws. Maybe I should change the dataset 😢

Interesting, thanks for sharing the details. You can check the ThoughtViz paper and one more object EEG dataset is there. I think there are few papers in CVPR21 that proposed the EEG dataset.

@YutLan

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/singh_prajwal_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

Hi, could you please share the code again? The link is not available. Many thanks!

@YutLan

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/singh_prajwal_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

Hi, could you please share the code again? The link is not available. Many thanks!

Hi, I have updated the [link].

@YutLan

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/singh_prajwal_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

Hi, could you please share the code again? The link is not available. Many thanks!

Hi, I have updated the [link].

Wow, thanks for your quick reply, I will try this code!

@YutLan

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/singh_prajwal_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

Hi, could you please share the code again? The link is not available. Many thanks!

Hi, I have updated the [link].

Hello, could you please share your opinion about dataset splitting? I noticed that the paper mentioned, "Our dataset was split into training, validation, and test sets, with respective fractions 80% (1600 images), 10% (200 images), 10% (200 images)." Does this mean that the final size of the training set is 16006=9600, the validation set is 2006=1200, and the test set is 200*6=1200? However, in the file block_splits_by_image_single.pth, the lengths of the three are 669:167:164, and in block_splits_by_image_all.pth, the lengths are 7970:1998:1997. These are not in an 8:1:1 ratio. What is the reason for this? Many thanks!

@cspampin

@cspampin Is the file named "eeg_signals_raw_with_mean_std.pth" the raw data you mentioned? I still can not reproduce the results. @dara1400 what's your process? Thanks!

Hi YutLan, I read your "Seeing through the Brain" paper before. I'm trying to download the dataset from OneDrive now, but I'm finding that the download is unusually slow in mainland China, and I've been unable to get it to download successfully. Could you please share a download link with me privately? Thank you very much!

Bring Prajwals' modifications to train on raw eeg data

change lstm depth to 4

@prajwalsingh

我已经用原始数据搜索了学习率、种子。在有效数据集和测试数据集上的最佳结果约为 40%。您能否告诉我您的实验细节？ @prajwalsingh 我从这里下载了数据。谢谢！

这是在哪里下载的，我是一个小白，实在没有找到，希望您的帮助！

@makabakayyy

Hi @makabakayyy, you can find the dataset here. If it is not opening for you, in that case, use VPN service.

@YutLan You can try this code. I have used this got good ACC: https://iitgnacin-my.sharepoint.com/:f:/g/personal/singh_prajwal_iitgn_ac_in/Ep1YBpxAjthFr1V7_uHFLy8BGYfbTx94xiKxJ1VPSojzMA?e=HGtwLV Let me know if it works.

Hi, could you please share the code again? The link is not available. Many thanks!

Hi, I have updated the [link].

Great Indian.❤️

@YutLan What I observe is results are reproducible with raw file data but not with 5_95 hz filter one. I have cross-checked with all other methods in different papers and reached the same conclusion. We tried reproducing it with 5_95 Hz for a long time but when I saw this repo then I realise that it is possible with Raw data. Now only thing I am not sure about is the results of the PAMI paper.
It will be helpful if you crosscheck this from your end.

Yeah, I found it works on raw data but not preprocessed data either. And I wonder why dose this phenomenon happen, I mean, the raw data looks too prefect (than other eeg dataset such as THINGS-eeg) isn't it?

In the PAMI paper, they address the problem with raw data that leads to overfitting (might be some noise/signal pattern is easy to identify be neural network) as well as there are some overlaps in the train and test data. Therefore authors proposed a method for processing filter 5-95Hz in the PAMI paper. You can preprocess the raw data from your end with a new train-val-test split without overlapping samples.
You can also check one of our works recently accepted in WACV24: Learning Robust Deep Visual Representations from EEG Brain Recordings [https://arxiv.org/abs/2310.16532]

I find something interesting 🤣 . In another TPAMI paper The Perils and Pitfalls of Block Design for EEG Classification Experiments, it said that "We take our correspondence with the authors to imply that no filtering was applied during acquisition, no filtering was applied prior to production of either the Python or Matlab format released data". And I have found some more papers which think this dataset has some flaws. Maybe I should change the dataset 😢

Great Chinese. TPAMI paper The Perils and Pitfalls of Block Design for EEG Classification Experiments you shared helps a lot!

After my experiment. I find the accuracy really depends on the learning rate. Which set at 1e4 only get 0.03, and 2e3 get nealy 0.7! I believe high lr somehow may benefit for performance. That really surprise me, and hope this helps you.

Navigation Menu

Repository navigation

Overfitting when training lstm model! #1

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Overfitting when training lstm model! #1

Description

Activity

dara1400 commented on Feb 8, 2022

cspampin commented on Feb 9, 2022

dara1400 commented on Feb 9, 2022

cspampin commented on Feb 9, 2022

dara1400 commented on Feb 9, 2022

cspampin commented on Feb 9, 2022

dara1400 commented on Feb 9, 2022

prajwalsingh commented on Apr 3, 2023

YutLan commented on May 23, 2023

prajwalsingh commented on May 23, 2023

YutLan commented on May 23, 2023

prajwalsingh commented on May 23, 2023

YutLan commented on May 23, 2023

YutLan commented on May 23, 2023

prajwalsingh commented on May 23, 2023

12 remaining items

Zoe-Wan commented on Dec 11, 2023

Zoe-Wan commented on Dec 11, 2023

prajwalsingh commented on Dec 11, 2023

bettermanc commented on Apr 3, 2024

prajwalsingh commented on Apr 3, 2024

bettermanc commented on Apr 3, 2024

bettermanc commented on Apr 4, 2024

woldier commented on May 11, 2024

makabakayyy commented on Jan 11, 2025

prajwalsingh commented on Jan 12, 2025

5f8a1b7d commented on Jan 31, 2025

5f8a1b7d commented on Jan 31, 2025

ZiyiTsang commented on Mar 27, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions