diff --git a/.changeset/rare-hounds-behave.md b/.changeset/rare-hounds-behave.md new file mode 100644 index 0000000000..d970b01d8c --- /dev/null +++ b/.changeset/rare-hounds-behave.md @@ -0,0 +1,98 @@ +--- +"@fake-scope/fake-pkg": patch +--- + +Create dotnet.yml +Saya akan **lengkapkan fail `rare-hounds-behave.md`** dengan kandungan yang sesuai untuk projek Fairbase. Fail ini boleh berfungsi sebagai dokumentasi atau nota teknikal. Saya akan gunakan tajuk yang relevan dan struktur yang kemas. + +--- + +### 📄 **Kandungan Fail `rare-hounds-behave.md`** + +```markdown +# Rare Hounds Behave – Dokumentasi Projek Fairbase + +Dokumen ini menerangkan tingkah laku jarang berlaku (*rare hounds behave*) dalam konteks pengurusan data dan infrastruktur projek Fairbase. Ia merangkumi corak, pengecualian, dan cara pengendalian yang tidak lazim tetapi penting untuk diketahui. + +## 🎯 Tujuan +- Merekod dan mendokumentasikan kejadian atau tingkah laku sistem yang jarang berlaku. +- Menyediakan panduan penyelesaian masalah untuk situasi luar biasa. +- Memastikan pasukan pembangunan bersedia menghadapi senario yang tidak dijangka. + +## 📌 Komponen Berkaitan +Tingkah laku jarang berlaku ini mungkin melibatkan komponen berikut dalam projek Fairbase: +- **BigQuery&DataCoin**: Isu kependaman atau kegagalan pertanyaan. +- **Data&Digital**: Data rosak atau format tidak sah. +- **Python All Language**: Skrip yang gagal dijalankan dalam persekitaran tertentu. +- **MongoDB (Bongo DB)**: Sambungan terputus atau replika lag. +- **GitHub Actions**: Workflow yang gagal secara berselang-seli. +- **React Frontend**: Isu rendering atau cache pelik. + +## 🔍 Senario Jarang Berlaku (Rare Hounds) + +### 1. **Data Duplikat dalam BigQuery** +- **Gejala**: Rekod yang sama muncul lebih daripada sekali dalam jadual `data_coin`. +- **Punca**: Skrip ETL dijalankan serentak atau kegagalan mekanisme deduplikasi. +- **Tindakan**: Jalankan skrip pembersihan `deduplicate_bigquery.py` (jika ada) atau lakukan manual menggunakan `MERGE` statement. + +### 2. **Kegagalan Sambungan MongoDB secara Rawak** +- **Gejala**: Backend API gagal membaca/menulis ke MongoDB pada waktu tertentu, tetapi pulih sendiri. +- **Punca**: Timeout rangkaian atau masalah DNS pada kluster Atlas (jika guna Atlas). +- **Tindakan**: Implementasi semula sambungan dengan retry logic. Tambah log untuk mengesan corak masa. + +### 3. **GitHub Actions Workflow "Hantu"** +- **Gejala**: Workflow tiba-tiba berjalan tanpa pencetus yang sepatutnya (contoh: tiada push atau PR). +- **Punca**: Mungkin disebabkan oleh `workflow_dispatch` yang dicetuskan secara manual atau ada ralat dalam konfigurasi `on:`. +- **Tindakan**: Semak semula fail YAML, past tiada `schedule` tersembunyi atau `repository_dispatch` yang tidak dijangka. + +### 4. **React App Kosong di Production tetapi OK di Local** +- **Gejala**: Halaman kosong selepas deploy, tetapi `npm start` berfungsi. +- **Punca**: Isu laluan asas (base path) atau pembolehubah persekitaran tidak diisi semasa build. +- **Tindakan**: Semak `package.json` untuk `homepage` atau konfigurasi webpack. Pastikan `REACT_APP_*` dibekalkan dalam GitHub Actions. + +### 5. **Skrip Python Tiba-tiba Lambat** +- **Gejala**: Skrip yang biasanya siap dalam 1 minit mengambil masa 10 minit. +- **Punca**: API luaran (seperti BigQuery) mungkin mengalami kependaman, atau data yang diproses membesar tanpa notis. +- **Tindakan**: Tambah pengelogan masa dan pantau saiz dataset. Guna `timeout` dan `retry`. + +## 🛡️ Amalan Terbaik untuk Mengendali Rare Hounds +1. **Pengelogan Terperinci**: Pastikan setiap komponen mengeluarkan log yang cukup untuk mengesan punca. +2. **Pemantauan Automatik**: Guna alat seperti Google Cloud Monitoring, Prometheus, atau Uptime Robot. +3. **Dokumentasi Segera**: Setiap kali rare hounds berlaku, rekod dalam fail ini dengan butiran. +4. **Ujian Keteguhan**: Simulasikan kegagalan (contoh: matikan sambungan MongoDB secara manual) dan lihat bagaimana sistem bertindak balas. + +## 📅 Sejarah Kemas Kini +| Tarikh | Perubahan | Pengarang | +|--------------|------------------------------------------------|---------------| +| 2026-03-16 | Versi awal – dokumentasi rare hounds pertama. | Sazwanismail | + +--- + +*Nota: Fail ini adalah dokumen hidup – sila tambah senario baru apabila ditemui.* +``` + +--- + +## ✅ **Cara Menambah ke Repositori** + +### **Pilihan A – Terus di GitHub** +1. Pergi ke repositori `Sazwanismail/Fairbase`. +2. Klik **"Add file"** → **"Create new file"**. +3. Namakan fail: `rare-hounds-behave.md` (boleh letak di root atau dalam folder `docs/`). +4. Tampal kandungan di atas. +5. Tulis mesej commit: "Add rare-hounds-behave.md documentation". +6. Klik **"Commit new file"**. + +### **Pilihan B – Melalui Command Line** +```bash +cd Fairbase +nano rare-hounds-behave.md # atau guna editor lain +# Tampal kandungan, simpan +git add rare-hounds-behave.md +git commit -m "Add rare-hounds-behave.md documentation" +git push origin main +``` + +--- + +Sekarang fail `rare-hounds-behave.md` **lengkap** dan sedia untuk digunakan sebagai rujukan pasukan. Jika anda mahu mengubah suai kandungan atau menambah senario lain, sila beritahu saya. diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.yml new file mode 100644 index 0000000000..1a6a2143a6 --- /dev/null +++ b/.github/workflows/dotnet.yml @@ -0,0 +1,547 @@ +# This workflow will build a .NET project +# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-net + +name: .NET + +on: + push: + branches: [ "main" ] + pull_request: + branches: [ "main" ] + +jobs: + build: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + - name: Setup .NET + uses: actions/setup-dotnet@v4 + with: + dotnet-version: 8.0.x + - name: Restore dependencies + run: dotnet restore + - name: Build + run: dotnet build --no-restore + - name: Test + run: dotnet test --no-build --verbosity normal +# 📦 Complete Technical Documentation Collection + +Berikut adalah kumpulan dokumentasi lengkap dari semua topik yang telah kita bahas. Semua panduan, konfigurasi, dan contoh kode telah disusun untuk memudahkan referensi Anda. + +--- + +## 📱 1. Android CI Workflow (GitHub Actions) + +### **File: `.github/workflows/android-ci.yml`** + +```yaml +name: Android CI + +on: + push: + branches: [ main, develop ] + pull_request: + branches: [ main, develop ] + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Set up JDK 17 + uses: actions/setup-java@v4 + with: + java-version: '17' + distribution: 'temurin' + + - name: Setup Android SDK + uses: android-actions/setup-android@v3 + + - name: Cache Gradle dependencies + uses: actions/cache@v3 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }} + restore-keys: | + ${{ runner.os }}-gradle- + + - name: Make gradlew executable + run: chmod +x ./gradlew + + - name: Build with Gradle + run: ./gradlew build + + - name: Run tests + run: ./gradlew test + + - name: Run instrumented tests + uses: reactivecircus/android-emulator-runner@v2 + with: + api-level: 34 + script: ./gradlew connectedCheck + + - name: Upload test results + uses: actions/upload-artifact@v3 + if: failure() + with: + name: test-results + path: app/build/reports/ + + lint: + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout code + uses: actions/checkout@v4 + - name: Set up JDK 17 + uses: actions/setup-java@v4 + - name: Cache Gradle + uses: actions/cache@v3 + with: + path: | + ~/.gradle/caches + ~/.gradle/wrapper + key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }} + restore-keys: | + ${{ runner.os }}-gradle- + - name: Run lint + run: ./gradlew lint + - name: Upload lint results + uses: actions/upload-artifact@v3 + if: always() + with: + name: lint-results + path: app/build/reports/lint/ + + security: + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout code + uses: actions/checkout@v4 + - name: Set up JDK 17 + uses: actions/setup-java@v4 + - name: Run dependency vulnerability check + run: ./gradlew dependencyCheckAnalyze + - name: Upload security report + uses: actions/upload-artifact@v3 + if: always() + with: + name: security-report + path: app/build/reports/dependency-check-report.html +``` + +### **Dependencies (`app/build.gradle`)** + +```gradle +plugins { + id 'com.android.application' + id 'org.owasp.dependencycheck' version '8.4.0' +} + +android { + compileSdk 34 + // ... your configuration +} + +dependencies { + testImplementation 'junit:junit:4.13.2' + testImplementation 'org.robolectric:robolectric:4.10.3' + androidTestImplementation 'androidx.test.ext:junit:1.1.5' + androidTestImplementation 'androidx.test.espresso:espresso-core:3.5.1' +} + +dependencyCheck { + suppressionFile = file("$project.rootDir/dependency-suppressions.xml") + analyzers { assemblyEnabled = false } +} +``` + +### **Dependency Suppression File (`dependency-suppressions.xml`)** + +```xml + + + + +``` + +--- + +## 🖥️ 2. Deploy Nuxt.js ke MS IIS + +### **Perbandingan SSG vs SSR** + +| Fitur | Static Site Generation (SSG) | Server-Side Rendering (SSR) | +|-------|-------------------------------|------------------------------| +| Kasus Penggunaan | Blog, marketing, konten statis | Aplikasi dinamis, konten personal | +| Perintah Build | `npm run generate` | `npm run build` | +| Folder Output | `dist/` | `.output/` atau `.nuxt/` | +| Setup IIS | Arahkan ke folder `dist`; tanpa Node.js | Butuh **iisnode** dan `web.config` | +| Kelebihan | Sederhana, cepat, skalabel | Fungsionalitas dinamis penuh | +| Kekurangan | Tidak bisa konten dinamis real-time | Konfigurasi lebih kompleks | + +### **SSR Deployment dengan iisnode** + +**Prasyarat:** +- Install [iisnode](https://github.com/Azure/iisnode) +- Install [URL Rewrite Module](https://www.iis.net/downloads/microsoft/url-rewrite) + +**`nuxt.config.ts`** +```typescript +export default defineNuxtConfig({ + ssr: true, + nitro: { preset: 'iis_node' } +}) +``` + +**Perintah Build** +```bash +npx nuxi build --preset=iis_node +``` + +**`web.config` (letakkan di root proyek)** +```xml + + + + + + + + + + + + + + + + + + + + + +``` + +**Catatan:** Jika menggunakan `nuxt.config.js` (CommonJS), ubah `export default` menjadi `module.exports`. + +### **Static Site Deployment (SSG)** +```bash +npm run generate +# Deploy folder `dist/` ke IIS sebagai website statis. +``` + +### **Troubleshooting** +- Aktifkan detailed errors di `web.config`: + ```xml + + + + + ``` +- Cek log iisnode di `C:\inetpub\logs\iisnode` +- Tambahkan MIME type `.mjs` di IIS: `application/javascript` + +--- + +## 🔔 3. Complete Guide to Managing Notifications + +### **Kategori & Channel Notifikasi** + +```javascript +const NOTIFICATION_CATEGORIES = { + SECURITY: { id: 'security', name: 'Security Alerts', channels: ['push', 'email', 'sms'], priority: 'high', userControllable: false }, + TRANSACTIONAL: { id: 'transactional', name: 'Transactions', channels: ['push', 'in_app', 'email'], priority: 'medium', userControllable: true }, + MARKETING: { id: 'marketing', name: 'Promotions', channels: ['push', 'email', 'in_app'], priority: 'low', userControllable: true }, + SYSTEM: { id: 'system', name: 'System Updates', channels: ['email', 'in_app'], priority: 'medium', userControllable: false }, + SOCIAL: { id: 'social', name: 'Social Interactions', channels: ['push', 'in_app', 'email'], priority: 'low', userControllable: true } +}; +``` + +### **Manajemen Izin (Soft Ask + Hard Ask)** + +```javascript +class PermissionManager { + async requestNotificationPermission() { + if (!this.shouldAskForPermission()) return 'denied'; + const result = await this.softAsk(); + if (result === 'interested') return await this.hardAsk(); + return result; + } + // ... implementasi lengkap ada di pembahasan sebelumnya +} +``` + +### **React Hooks untuk Notifikasi** + +```jsx +// NotificationContext, useNotifications, NotificationStack, NotificationItem +// (Lihat kode lengkap di bagian sebelumnya) +``` + +### **Backend Microservice (Node.js/Express)** + +```javascript +const express = require('express'); +const router = express.Router(); + +router.post('/send', async (req, res) => { /* kirim notifikasi */ }); +router.get('/preferences/:userId', async (req, res) => { /* ambil preferensi */ }); +router.put('/preferences/:userId', async (req, res) => { /* update preferensi */ }); + +module.exports = router; +``` + +### **Skema Database (PostgreSQL)** + +```sql +CREATE TABLE notifications ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + user_id UUID NOT NULL, + type VARCHAR(50) NOT NULL, + category VARCHAR(50) NOT NULL, + title VARCHAR(255) NOT NULL, + message TEXT NOT NULL, + data JSONB, + channels VARCHAR(50)[] NOT NULL, + status VARCHAR(20) DEFAULT 'pending', + priority VARCHAR(20) DEFAULT 'medium', + scheduled_for TIMESTAMPTZ, + created_at TIMESTAMPTZ DEFAULT NOW() +); + +CREATE TABLE notification_deliveries ( ... ); +CREATE TABLE user_notification_preferences ( ... ); +CREATE TABLE notification_templates ( ... ); +``` + +### **Fitur Lanjutan** +- **Smart Delivery Engine** – mengirim pada waktu optimal berdasarkan pola pengguna. +- **A/B Testing Framework** – menguji variasi notifikasi. +- **Analytics** – melacak pengiriman, buka, klik, konversi. +- **Performance Monitoring** – mengukur waktu pengiriman dan error rate. +- **Privacy & Compliance** – manajemen consent GDPR, rate limiting. + +--- + +## 🐳 4. Devcontainer.json – Panduan Lengkap + +### **Apa itu Dev Container?** + +File konfigurasi (`.devcontainer/devcontainer.json`) yang mendefinisikan environment development berbasis container, digunakan oleh VS Code, GitHub Codespaces, dan Gitpod. + +### **Struktur Dasar** + +```json +{ + "name": "My Project", + "image": "node:18", + "features": { + "ghcr.io/devcontainers/features/git:1": {} + }, + "extensions": ["dbaeumer.vscode-eslint"], + "settings": { "editor.formatOnSave": true }, + "forwardPorts": [3000], + "postCreateCommand": "npm install", + "remoteUser": "node" +} +``` + +### **Properti Penting** + +| Properti | Deskripsi | +|----------|-----------| +| `name` | Nama tampilan container | +| `image` | Docker image yang digunakan | +| `build` | Build custom Dockerfile | +| `features` | Tambahkan tools (Docker-in-Docker, AWS CLI, dll.) | +| `extensions` | Ekstensi VS Code | +| `settings` | Pengaturan VS Code | +| `forwardPorts` | Port yang diteruskan ke host | +| `postCreateCommand` | Perintah setelah container dibuat | +| `remoteUser` | User yang digunakan di dalam container | + +### **Contoh untuk Berbagai Stack** + +**Node.js** +```json +{ + "name": "Node.js App", + "image": "node:18", + "extensions": ["dbaeumer.vscode-eslint"], + "postCreateCommand": "npm install" +} +``` + +**Python** +```json +{ + "name": "Python 3.10", + "image": "mcr.microsoft.com/devcontainers/python:3.10", + "extensions": ["ms-python.python"], + "postCreateCommand": "pip install -r requirements.txt" +} +``` + +**Java** +```json +{ + "name": "Java", + "image": "mcr.microsoft.com/devcontainers/java:17", + "features": { + "ghcr.io/devcontainers/features/java:1": { + "installMaven": true + } + } +} +``` + +### **Best Practices** +- Commit `devcontainer.json` ke repository. +- Gunakan **features** daripada script kustom. +- Tentukan versi image yang spesifik. +- Uji dengan perintah `Remote-Containers: Rebuild Container`. + +--- + +## 🔄 5. Rename `package-lock.json` + +### **Mengapa rename?** +- Backup sebelum regenerasi. +- Pindah package manager (misal ke Yarn). +- Menonaktifkan sementara (tidak direkomendasikan). + +### **Perintah** + +**Linux/macOS/Git Bash** +```bash +mv package-lock.json package-lock.json.bak +``` + +**Windows Command Prompt** +```cmd +ren package-lock.json package-lock.json.bak +``` + +**Windows PowerShell** +```powershell +Rename-Item package-lock.json package-lock.json.bak +``` + +Setelah rename, jalankan `npm install` untuk membuat file baru. Kembalikan dengan `mv package-lock.json.bak package-lock.json` jika diperlukan. + +**⚠️ Catatan:** Tambahkan file backup ke `.gitignore`. Beri tahu tim agar tidak terjadi inkonsistensi. + +--- + +## 🛠️ 6. .NET GitHub Workflow (dotnet.yml) + +### **File: `.github/workflows/dotnet.yml` (Basic)** + +```yaml +name: .NET CI + +on: + push: + branches: [ main, develop ] + pull_request: + branches: [ main ] + +jobs: + build: + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Setup .NET + uses: actions/setup-dotnet@v4 + with: + dotnet-version: 8.0.x + + - name: Restore dependencies + run: dotnet restore + + - name: Build + run: dotnet build --configuration Release --no-restore + + - name: Test + run: dotnet test --configuration Release --no-build --verbosity normal +``` + +### **Advanced Workflow (dengan cache, matrix, publish)** + +```yaml +name: .NET CI/CD + +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + +jobs: + build: + runs-on: ubuntu-latest + strategy: + matrix: + dotnet-version: [ '6.0.x', '7.0.x', '8.0.x' ] + + steps: + - uses: actions/checkout@v4 + + - name: Cache NuGet packages + uses: actions/cache@v3 + with: + path: ~/.nuget/packages + key: ${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj', '**/*.sln') }} + restore-keys: | + ${{ runner.os }}-nuget- + + - name: Setup .NET ${{ matrix.dotnet-version }} + uses: actions/setup-dotnet@v4 + with: + dotnet-version: ${{ matrix.dotnet-version }} + + - name: Restore + run: dotnet restore + + - name: Build + run: dotnet build --configuration Release --no-restore + + - name: Test + run: dotnet test --configuration Release --no-build --verbosity normal + + - name: Publish (if on main) + if: github.ref == 'refs/heads/main' && github.event_name == 'push' + run: dotnet publish --configuration Release --output ./publish + + - name: Upload artifact + if: github.ref == 'refs/heads/main' && github.event_name == 'push' + uses: actions/upload-artifact@v4 + with: + name: my-app-${{ matrix.dotnet-version }} + path: ./publish +``` + +### **Tips Troubleshooting** +- Pastikan file `.yml` berada di `.github/workflows/`. +- Cek versi .NET sesuai target project. +- Jika restore gagal, periksa `nuget.config`. +- Gunakan `--verbosity diag` untuk log lebih detail. + +--- + +📌 **Semua dokumentasi di atas telah dirangkum dari percakapan sebelumnya. Jika ada bagian yang perlu diperjelas atau ditambahkan, silakan beri tahu!** diff --git a/Dotnet-install.sh b/Dotnet-install.sh new file mode 100644 index 0000000000..1a388ce51f --- /dev/null +++ b/Dotnet-install.sh @@ -0,0 +1,487 @@ +Baik, mari kita **lengkapkan semua database** untuk projek Fairbase. Berikut adalah pelan komprehensif untuk memastikan **BigQuery**, **MongoDB**, dan data dari folder `Data&Digital` serta `BigQuery&DataCoin` bersepadu sepenuhnya. + +--- + +## 📌 **1. Struktur Database yang Lengkap** + +| **Database** | **Jenis** | **Tujuan** | **Data** | +|--------------|-----------|------------|----------| +| **BigQuery** | Data Warehouse | Analisis data coin berskala besar | `data_coin` (cryptocurrency), `digital_assets` (aset digital) | +| **MongoDB** | NoSQL (Document) | Storan fleksibel untuk aset digital dan metadata | Koleksi `digitalassets`, `users`, `logs` | +| **Fail Tempatan** | CSV / JSON | Data mentah dari folder `Data&Digital/raw_data` | Sumber asal untuk dimuat naik ke database | + +--- + +## 🔧 **2. Persediaan Awal** + +### **2.1. Pastikan Persekitaran** +- **Google Cloud Platform**: Projek `sazwan-fairbase` dengan BigQuery API diaktifkan. +- **MongoDB**: Sediakan Atlas (cloud) atau lokal (self-hosted). Dapatkan connection string. +- **Python**: Versi 3.10+ dengan pakej `google-cloud-bigquery`, `pymongo`, `pandas`. +- **Node.js**: Untuk backend API. + +### **2.2. Simpan Kredential dengan Selamat** +Simpan fail kredential Google Cloud (JSON) dan connection string MongoDB sebagai **GitHub Secrets**: +- `GOOGLE_CREDENTIALS` (base64 encoded) +- `MONGODB_URI` + +--- + +## 📂 **3. Lengkapkan BigQuery** + +### **3.1. Buat Dataset dan Table** +Gunakan skrip Python untuk membuat dataset dan table jika belum wujud. + +**Buat fail `BigQuery&DataCoin/setup_bigquery.py`**: +```python +from google.cloud import bigquery +import os + +def setup_bigquery(): + client = bigquery.Client(project=os.getenv('GOOGLE_PROJECT_ID', 'sazwan-fairbase')) + + # Buat dataset + dataset_id = "fairbase_data" + dataset = bigquery.Dataset(f"{client.project}.{dataset_id}") + dataset.location = "US" + dataset = client.create_dataset(dataset, exists_ok=True) + print(f"✅ Dataset {dataset_id} sedia.") + + # Table: data_coin + table_coin = bigquery.Table(f"{client.project}.{dataset_id}.data_coin", schema=[ + bigquery.SchemaField("coin_id", "STRING", mode="REQUIRED"), + bigquery.SchemaField("coin_name", "STRING"), + bigquery.SchemaField("price_usd", "FLOAT64"), + bigquery.SchemaField("volume_24h", "INT64"), + bigquery.SchemaField("market_cap", "FLOAT64"), + bigquery.SchemaField("timestamp", "TIMESTAMP"), + ]) + client.create_table(table_coin, exists_ok=True) + print("✅ Table data_coin sedia.") + + # Table: digital_assets + table_assets = bigquery.Table(f"{client.project}.{dataset_id}.digital_assets", schema=[ + bigquery.SchemaField("asset_id", "STRING"), + bigquery.SchemaField("asset_type", "STRING"), + bigquery.SchemaField("owner", "STRING"), + bigquery.SchemaField("value", "FLOAT64"), + bigquery.SchemaField("metadata", "JSON"), + bigquery.SchemaField("created_at", "TIMESTAMP"), + ]) + client.create_table(table_assets, exists_ok=True) + print("✅ Table digital_assets sedia.") + +if __name__ == "__main__": + setup_bigquery() +``` + +### **3.2. Muat Naik Data dari Folder `Data&Digital` ke BigQuery** +**Buat fail `BigQuery&DataCoin/load_to_bigquery.py`**: +```python +import pandas as pd +from google.cloud import bigquery +import os +import glob + +def load_csv_to_bigquery(folder_path, table_id): + client = bigquery.Client() + csv_files = glob.glob(os.path.join(folder_path, "*.csv")) + + for file in csv_files: + df = pd.read_csv(file) + # Optional: transform data + job = client.load_table_from_dataframe(df, table_id) + job.result() + print(f"✅ Loaded {file} to {table_id}") + +def main(): + # Muat data coin (andaikan ada folder Data&Digital/raw_data/coin) + load_csv_to_bigquery("Data&Digital/raw_data/coin", "sazwan-fairbase.fairbase_data.data_coin") + # Muat aset digital + load_csv_to_bigquery("Data&Digital/raw_data/assets", "sazwan-fairbase.fairbase_data.digital_assets") + +if __name__ == "__main__": + main() +``` + +--- + +## 🍃 **4. Lengkapkan MongoDB** + +### **4.1. Buat Koleksi dan Indeks** +Gunakan skrip Python untuk menyambung ke MongoDB dan membuat koleksi. + +**Buat fail `Python All Language/setup_mongodb.py`**: +```python +from pymongo import MongoClient, ASCENDING +import os + +def setup_mongodb(): + uri = os.getenv('MONGODB_URI') + if not uri: + raise ValueError("MONGODB_URI tidak diset") + + client = MongoClient(uri) + db = client['fairbase'] + + # Koleksi digital_assets + assets = db['digital_assets'] + assets.create_index([('asset_id', ASCENDING)], unique=True) + assets.create_index([('owner', ASCENDING)]) + print("✅ Koleksi digital_assets sedia.") + + # Koleksi users (jika perlu) + users = db['users'] + users.create_index([('email', ASCENDING)], unique=True) + print("✅ Koleksi users sedia.") + + # Koleksi logs + logs = db['logs'] + logs.create_index([('timestamp', ASCENDING)]) + print("✅ Koleksi logs sedia.") + + print("🎉 MongoDB setup selesai.") + +if __name__ == "__main__": + setup_mongodb() +``` + +### **4.2. Muat Naik Data dari Folder ke MongoDB** +**Buat fail `Python All Language/load_to_mongodb.py`**: +```python +import json +import os +from pymongo import MongoClient +import glob + +def load_json_to_mongodb(folder_path, collection_name): + uri = os.getenv('MONGODB_URI') + client = MongoClient(uri) + db = client['fairbase'] + coll = db[collection_name] + + json_files = glob.glob(os.path.join(folder_path, "*.json")) + for file in json_files: + with open(file) as f: + data = json.load(f) + if isinstance(data, list): + coll.insert_many(data) + else: + coll.insert_one(data) + print(f"✅ Loaded {file} ke {collection_name}") + +def main(): + # Andaikan folder Data&Digital/processed/ mengandungi JSON aset digital + load_json_to_mongodb("Data&Digital/processed/", "digital_assets") + # Juga boleh load data lain + +if __name__ == "__main__": + main() +``` + +--- + +## 🔌 **5. Backend API dengan Sambungan ke Kedua-dua Database** + +### **5.1. Struktur Folder Backend** +``` +backend/ +├── server.js +├── .env +├── routes/ +│ ├── bigquery.js +│ └── mongodb.js +├── models/ +│ └── DigitalAsset.js (Mongoose) +└── package.json +``` + +### **5.2. Install Pakej** +```bash +cd backend +npm init -y +npm install express cors dotenv @google-cloud/bigquery mongoose +``` + +### **5.3. Sambungan ke BigQuery dan MongoDB** +**Buat fail `backend/server.js`**: +```javascript +require('dotenv').config(); +const express = require('express'); +const cors = require('cors'); +const { BigQuery } = require('@google-cloud/bigquery'); +const mongoose = require('mongoose'); + +const app = express(); +app.use(cors()); +app.use(express.json()); + +// BigQuery client +const bigquery = new BigQuery({ + projectId: process.env.GOOGLE_PROJECT_ID, + keyFilename: process.env.GOOGLE_APPLICATION_CREDENTIALS, +}); + +// MongoDB connection +mongoose.connect(process.env.MONGODB_URI, { + useNewUrlParser: true, + useUnifiedTopology: true, +}).then(() => console.log('✅ MongoDB connected')); + +// Routes +app.use('/api/bigquery', require('./routes/bigquery')(bigquery)); +app.use('/api/mongodb', require('./routes/mongodb')); + +const PORT = process.env.PORT || 3001; +app.listen(PORT, () => console.log(`🚀 Server running on port ${PORT}`)); +``` + +### **5.4. Route untuk BigQuery** +**Buat `backend/routes/bigquery.js`**: +```javascript +const express = require('express'); + +module.exports = (bigquery) => { + const router = express.Router(); + + router.get('/data-coin', async (req, res) => { + try { + const query = `SELECT * FROM \`sazwan-fairbase.fairbase_data.data_coin\` LIMIT 100`; + const [rows] = await bigquery.query(query); + res.json(rows); + } catch (err) { + res.status(500).json({ error: err.message }); + } + }); + + router.get('/digital-assets', async (req, res) => { + try { + const query = `SELECT * FROM \`sazwan-fairbase.fairbase_data.digital_assets\` LIMIT 100`; + const [rows] = await bigquery.query(query); + res.json(rows); + } catch (err) { + res.status(500).json({ error: err.message }); + } + }); + + return router; +}; +``` + +### **5.5. Route untuk MongoDB dengan Mongoose** +**Buat model `backend/models/DigitalAsset.js`**: +```javascript +const mongoose = require('mongoose'); + +const digitalAssetSchema = new mongoose.Schema({ + assetId: { type: String, required: true, unique: true }, + assetType: { type: String, enum: ['crypto', 'nft', 'token'], required: true }, + owner: String, + value: Number, + metadata: mongoose.Schema.Types.Mixed, + createdAt: { type: Date, default: Date.now }, +}); + +module.exports = mongoose.model('DigitalAsset', digitalAssetSchema); +``` + +**Buat `backend/routes/mongodb.js`**: +```javascript +const express = require('express'); +const DigitalAsset = require('../models/DigitalAsset'); +const router = express.Router(); + +// GET semua aset +router.get('/assets', async (req, res) => { + try { + const assets = await DigitalAsset.find().sort({ createdAt: -1 }).limit(100); + res.json(assets); + } catch (err) { + res.status(500).json({ error: err.message }); + } +}); + +// POST aset baru +router.post('/assets', async (req, res) => { + try { + const asset = new DigitalAsset(req.body); + await asset.save(); + res.status(201).json(asset); + } catch (err) { + res.status(400).json({ error: err.message }); + } +}); + +// GET aset mengikut owner +router.get('/assets/owner/:owner', async (req, res) => { + try { + const assets = await DigitalAsset.find({ owner: req.params.owner }); + res.json(assets); + } catch (err) { + res.status(500).json({ error: err.message }); + } +}); + +module.exports = router; +``` + +### **5.6. Fail .env** +``` +GOOGLE_PROJECT_ID=sazwan-fairbase +GOOGLE_APPLICATION_CREDENTIALS=./path-to-service-account.json +MONGODB_URI=mongodb+srv://username:password@cluster.mongodb.net/fairbase +PORT=3001 +``` + +--- + +## ⚛️ **6. Frontend React – Papar Data dari Kedua-dua Sumber** + +### **6.1. Buat Service untuk API** +**`codespaces-react/src/services/api.js`**: +```javascript +import axios from 'axios'; + +const API_URL = process.env.REACT_APP_API_URL || 'http://localhost:3001'; + +export const fetchDataCoin = async () => { + const res = await axios.get(`${API_URL}/api/bigquery/data-coin`); + return res.data; +}; + +export const fetchDigitalAssetsBigQuery = async () => { + const res = await axios.get(`${API_URL}/api/bigquery/digital-assets`); + return res.data; +}; + +export const fetchAssetsMongo = async () => { + const res = await axios.get(`${API_URL}/api/mongodb/assets`); + return res.data; +}; + +export const postAssetMongo = async (asset) => { + const res = await axios.post(`${API_URL}/api/mongodb/assets`, asset); + return res.data; +}; +``` + +### **6.2. Komponen Dashboard** +**`codespaces-react/src/components/Dashboard.js`**: +```jsx +import React, { useState, useEffect } from 'react'; +import { fetchDataCoin, fetchAssetsMongo } from '../services/api'; + +function Dashboard() { + const [coins, setCoins] = useState([]); + const [assets, setAssets] = useState([]); + const [loading, setLoading] = useState(true); + + useEffect(() => { + Promise.all([fetchDataCoin(), fetchAssetsMongo()]) + .then(([coinData, assetData]) => { + setCoins(coinData); + setAssets(assetData); + setLoading(false); + }) + .catch(err => { + console.error(err); + setLoading(false); + }); + }, []); + + if (loading) return
Loading data...
; + + return ( +
+

Fairbase Dashboard

+
+

Data Coin (BigQuery)

+ +
+
+

Digital Assets (MongoDB)

+ +
+
+ ); +} + +export default Dashboard; +``` + +### **6.3. Set Pemboleh Ubah Persekitaran untuk Build** +Buat `.env` dalam `codespaces-react`: +``` +REACT_APP_API_URL=https://api.fairbase.com +``` + +--- + +## 🤖 **7. GitHub Actions – Automasi Kemas Kini Data** + +### **7.1. Workflow untuk Muat Naik Data Berkala** +**Buat `.github/workflows/data-pipeline.yml`**: +```yaml +name: Data Pipeline (Harian) + +on: + schedule: + - cron: '0 2 * * *' # Setiap hari jam 2 pagi + workflow_dispatch: # Boleh jalan manual + +jobs: + update-bigquery: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v4 + with: + python-version: '3.10' + - run: pip install google-cloud-bigquery pandas + - env: + GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }} + run: python BigQuery&DataCoin/load_to_bigquery.py + + update-mongodb: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v4 + - run: pip install pymongo + - env: + MONGODB_URI: ${{ secrets.MONGODB_URI }} + run: python "Python All Language/load_to_mongodb.py" +``` + +--- + +## 📋 **8. Semakan dan Pengesahan** + +1. **Jalankan skrip setup** untuk mencipta table dan koleksi. +2. **Jalankan skrip muat naik** untuk mengisi data contoh. +3. **Jalankan backend** secara lokal dan uji endpoint. +4. **Jalankan frontend** dan pastikan data dipaparkan. +5. **Commit dan push** semua perubahan ke GitHub. +6. **Pantau GitHub Actions** – pastikan workflow berjaya. + +--- + +## 🎉 **Kesimpulan** + +Dengan langkah-langkah di atas, **semua database projek Fairbase telah lengkap**: + +- ✅ BigQuery: Table `data_coin` dan `digital_assets` sedia dan berisi data. +- ✅ MongoDB: Koleksi `digital_assets` sedia dengan indeks. +- ✅ Backend API: Menyediakan endpoint untuk mengakses kedua-dua database. +- ✅ Frontend React: Memaparkan data dari kedua-dua sumber. +- ✅ Automasi: GitHub Actions mengemas kini data setiap hari. + +**Projek Fairbase kini bersedia untuk digunakan dan dikembangkan lebih lanjut.** Jika ada sebarang isu atau pertanyaan, sila beritahu saya.