[Archive] : Choosing the best structure for archiving gazettes

[yasandu0505]

We can do following below options to archive gazettes.

• Creating a folder structure for gazettes

gazette-archive/
└── 2010/
├── 2010_archive_log.csv # Success log
├── 2010_failed_log.csv # Failed downloads log
├── 01/
│ └── 15/
│ └── G-2010-001/
│ └── G-2010-001_english.pdf
└── 02/
└── 10/
└── G-2010-002/
└── G-2010-002_sinhala.pdf

• HDFS (Hadoop Distributed File System)
  Files are broken into blocks and distributed across DataNodes, while the NameNode manages the file system namespace and regulates access. This design allows HDFS to store and manage massive datasets across a cluster of machines, optimizing for high throughput data access.

For now we are using the first option. looking forward to switch to the HDFS after the testing and validating results of the base program.