Update READMEs for better onboarding process

README.md

@@ -1,12 +1,69 @@
 # assignment-snapshots
 
-TODO link to backups readme and react on rails readme
+IRB Protocol ID: 2025-03-18384
 
-## Continuous Integration
+- A command-line interface (CLI) to retrieve [OkPy](https://okpy.org) backup data (the "backups CLI") and store it in a local SQLite database
+- A full-stack web application (the "web app") to display the OkPy backup data to help TAs give qualitative feedback on a student's code
+- Exploratory data analysis (EDA) of OkPy backups to examine students' working habits in UC Berkeley's introductory computing courses, CS 61A and DATA C88C, from Fall 2023 - Fall 2025
 
-This repository uses [GitHub Actions](https://docs.github.com/en/actions) to run [black](https://black.readthedocs.io/en/stable/index.html) and [pylint](https://www.pylint.org/). See `.github/workflows`.
+## Setup
 
-## Pre-Commit Hooks
+1. Clone the repository with SSH:
+```sh
+git clone git@github.com:berkeley-cdss/assignment-snapshots.git
+```
+2. Follow the instructions in the [Backups CLI README](src/backups/README.md)'s **Setup** section. You **may skip step 4 (retrieving your OkPy backup token)** if you don't intend to actually request any backups.
+    1. You only need step 4 if you intend to run the `request` command of the Backups CLI. This also requires that you are added as a course staff member for the course you are requesting the backups from.
+3. Follow the instructions in the [Web App README](src/snapshots-app/README.md) (read and do all the sections; only do the AWS section once you have [AWS access](https://technology.berkeley.edu/bcloud-aws-central-faq)).
+    1. AWS access is only granted to internal UC Berkeley contributors. See internal [onboarding documentation](https://docs.google.com/document/d/1KhpRW0GYBY-HRSRG8b6z3EbRSFQJaqVPPUsha_puY2I/edit?tab=t.0).
+    2. When you run the `./bin/dev` command, you should only be able to view the home page with the Login button. If you click the button, it will error until you have done the next step.
+4. To get started with some toy development data **if you are on the IRB protocol** (internal contributors only):
+    1. Replace the `data` directory with the contents of this zip file (unzip it first): [data.zip](https://drive.google.com/file/d/11O9hZ4Fvh8JcTbUuDdtaF5Jv11ZJUWrX/view?usp=drive_link)
+    2. Follow the steps in the "[Dumping the database](https://github.com/berkeley-cdss/assignment-snapshots/tree/main/src/backups#dumping-database-from-okpy-backups-cli-into-rails-database)" section of the Backups CLI README to pipe the data from your local data directory into the snapshots Rails development database
+
+## Repo Structure
+
+All source code is located in the [`src`](src/) directory. All OkPy backup data is located in the [`data`](data/) directory (most of which is ignored by git for privacy reasons).
+
+### Backups CLI
+
+- All code related to the backups CLI is located in the [`src/backups`](src/backups/) directory.
+- The [`main.py`](src/backups/main.py) file contains the CLI command functions, and then other functions implement specific parts of the backups CLI such as database functions or functions that request or parse the raw OkPy backup data.
+
+### Web App
+
+- All code related to the assignment snapshots interface/web app is located in the [`src/snapshots-app`](src/snapshots-app/) directory.
+- All **frontend** code (React components) is located in the [`src/snapshots-app/client`](src/snapshots-app/client/) directory (specifically you define components in the [`src/snapshots-app/client/bundles`](src/snapshots-app/client/bundles/) directory)
+- All **backend** code (Ruby on Rails) is located in other folders. These are the main ones you will need to edit:
+    - [`src/snapshots-app/app`](src/snapshots-app/app/) contains the code for models, views, and controllers
+    - [`src/snapshots-app/config/routes.rb`](src/snapshots-app/config/routes.rb) contains the backend API routes
+    - [`src/snapshots-app/db/schema.rb`](src/snapshots-app/db/schema.rb) contains the database schemas
+    - [`src/snapshots-app/db/seeds.rb`](src/snapshots-app/db/seeds.rb) contains the code to seed the development database (e.g. set the initial values of the database programmatically)
+
+### EDA
+
+- All code related to the assignment snapshots EDA is located in the [`src/notebooks`](src/notebooks/) directory.
+
+### Continuous Integration
+
+This repository uses [GitHub Actions](https://docs.github.com/en/actions) to run various unit tests and linters. See [`.github/workflows`](.github/workflows/).
 
 This repository uses the [pre-commit](https://pre-commit.com/) package to automatically run
-the black Python formatter upon running `git commit`. Hooks can be added to `.pre-commit-config.yaml`.
+the black Python formatter upon running `git commit`. Hooks can be added to [`.pre-commit-config.yaml`](.pre-commit-config.yaml).
+
+## Contributors
+
+- Faculty advisors
+    - [Lisa Yan](yanlisa@berkeley.edu) (Spring 2025 - present)
+    - [Michael Ball](ball@berkeley.edu) (Spring 2025 - present)
+    - [Kay Ousterhout](keo@eecs.berkeley.edu) (Spring 2026 - present)
+- Students
+    - [Rebecca Dang](mailto:rdang@berkeley.edu) (Spring 2025 - present) (5th Year MS EECS)
+    - [Richard Villagomez](mailto:richard.villagomez@berkeley.edu) (Spring 2026 - present)
+    - [Jasmine Sov](mailto:jasminesov@berkeley.edu) (Spring 2026 - present)
+
+If you are affiliated with UC Berkeley and are interested in working on this project long-term, please reach out to us!
+
+If you are external to UC Berkeley and/or only wish to make code changes, please feel free to fork the repo
+and create a pull request to contribute code to the repository. Before requesting a review and merging,
+ensure that all status checks are passing.

src/backups/README.md

@@ -5,11 +5,11 @@
 1. Install [uv](https://docs.astral.sh/uv/getting-started/installation/)
 2. `cd` into this directory (`src/backups`)
 3. Run `uv sync` to install dependencies and create the virtual environment (`.venv/`)
-4. Create a `.env` file (follow the `.env-template`):
+4. **Optional**, only if you wish to run the `request` command to request OkPy backups: Create** a `.env` file (follow the `.env-template`):
     - Update the `.env` file with your [OkPy access token](https://okpy.github.io/documentation/ok-api.html#ok-server-api-authentication). You must have staff permissions for the course you want to query and you must periodically re-request your token.
 
 > [!NOTE]
-> In VS Code, you may need to [select your Python interpreter manually](https://github.com/astral-sh/uv/issues/8706#issuecomment-3070696514) to get language features in your editor.
+> In VS Code, you may need to [select your Python interpreter manually](https://github.com/astral-sh/uv/issues/8706#issuecomment-3070696514) to get language features in your editor. Alternatively, open the [`src/backups`](.) directory in its own VS Code window so that VS Code can detect the correct environment (`.venv`).
 
 ## Commands
 
@@ -53,30 +53,37 @@ Run `--help` with any of the commands for more information.
 Create a configuration file to save yourself the effort of typing a bunch of CLI arguments.
 An example can be found in [src/backups/backup_config.json](src/backups/backup_config.json).
 All fields are required (e.g. they must either be provided in the config or via the CLI).
-CLI arguments will override anything in the config.
+If you provide both a config and CLI arguments, the CLI arguments will override anything in the config.
 
 ## Dumping database from OkPy Backups CLI into Rails database
 
-1. Run backups CLI command(s) to create or update `$OUTPUT_DB_NAME.db`
-2. Create `.sql` dump of output `.db` file. Replace `$OUTPUT_DB_NAME` and `$OUTPUT_DUMP_NAME` with values of your choice:
+1. **Optional if not done already:** Run backups CLI command(s) in this directory to create or update `$OUTPUT_DB_NAME.db`. If you are an internal contributor working with the toy data from `data.zip`, skip this step.
+2. Create `.sql` dump of output `.db` file. Replace `$OUTPUT_DB_NAME` and `$OUTPUT_DUMP_NAME` with values of your choice **in the root directory** of the repository:
 ```sh
-cd $REPO_ROOT
 sqlite3 data/private/$OUTPUT_DB_NAME.db .dump > data/private/$OUTPUT_DUMP_NAME.sql
 ```
-3. Update `.sql` dump:
-    1. Remove `../../data/private/` prefix from paths
-    2. Remove/comment out `CREATE TABLE` statements since that will interfere with the Rails database migrations
-4. Optional if not done already: [Generate corresponding Rails model(s)](https://guides.rubyonrails.org/command_line.html#generating-models) as needed, e.g. `rails generate model <model_name> <column_name:data_type> ...`
+3. Update `data/private/$OUTPUT_DUMP_NAME.sql`:
+    1. Remove `../../data/private/` prefix from paths. **IMPORTANT:** Make sure you are removing the trailing `/`.
+    2. Remove/comment out `CREATE TABLE` statements since that will interfere with the Rails database migrations (Rails will already handle table creation on its own end, so if you have a duplicate `CREATE TABLE` statement Rails will error).
+4. **Optional if not done already:** [Generate corresponding Rails model(s)](https://guides.rubyonrails.org/command_line.html#generating-models) **in the `src/snapshots-app` directory** by running the following command. If you are an internal contributor working with the toy data from `data.zip`, skip this step.
+```sh
+rails generate model <model_name> <column_name:data_type> ...
+```
 > [!CAUTION]
 > THE FOLLOWING STEP WILL RESET (e.g. delete everything) AND RE-MIGRATE THE RAILS DB. BE CAREFUL!
-5. Run `cd src/snapshots-app` and `rails db:migrate:reset`.
-6. Run command below to execute commands from output `.sql` dump into the Rails app `development.sqlite3` database. Replace `$OUTPUT_DUMP_NAME` with the same value from steps 1 and 2:
+5. Run the following command **in the `src/snapshots-app` directory**:
+```sh
+rails db:migrate:reset
+```
+6. Run the following command **in the root directory** of the repository to execute commands from output `.sql` dump into the Rails app `development.sqlite3` database. Replace `$OUTPUT_DUMP_NAME` with the same value from steps 1 and 2:
 ```sh
-cd $REPO_ROOT
 sqlite3 src/snapshots-app/storage/development.sqlite3 < data/private/$OUTPUT_DUMP_NAME.sql
 ```
-7. Run `rails db:seed` to seed the Rails database with "hardcoded" seed data
-8. Verify that your data has been loaded properly in the rails console by using [the `ActiveRecord` query interface](https://guides.rubyonrails.org/active_record_querying.html#retrieving-objects-from-the-database) for example:
+7. Run the following command **in the `src/snapshots-app` directory** to seed the Rails database with hardcoded initial data:
+```sh
+rails db:seed
+```
+8. Verify that your data has been loaded properly in the Rails console by using [the `ActiveRecord` query interface](https://guides.rubyonrails.org/active_record_querying.html#retrieving-objects-from-the-database). For example, run the following commands **in the `src/snapshots-app` directory**:
 ```sh
 $ rails c
 snapshots-app(dev)> BackupMetadatum.all # view data and check it looks correct, press q to exit long list

src/snapshots-app/README.md

@@ -2,43 +2,87 @@
 
 ## Requirements
 
-* [NodeJS v22](https://nodejs.org/en/download)
-* [Ruby v3.3.0](https://rvm.io/)
+- NodeJS v22
+- Ruby v3.3.0
 
+We recommend installing the above using `nvm` and `rvm`, respectively, to manage your virtual environment.
+**Run all commands in the current directory.**
+
+1. [Install NodeJS for your OS using `nvm` with `npm`](https://nodejs.org/en/download) (use the dropdowns to generate the correct terminal commands and then run them)
+2. Tell `nvm` you want to use that version now:
 ```sh
 nvm use 22
+```
+3. Install [`rvm`](https://rvm.io/):
+```sh
+curl -sSL https://get.rvm.io | bash -s stable
+```
+4. Install Ruby v3.3.0:
+```sh
+rvm install 3.3.0
+```
+> [!WARNING]
+> If the command above results in some kind of error related to OpenSSL,
+> you can try running the steps under [Troubleshooting](#troubleshooting)
+> to fix your installation.
+5. Tell `rvm` you want to use that version now:
+```sh
 rvm use 3.3.0
 ```
 
-## Installation
+### Troubleshooting
+
+If you are on MacOS and have [OpenSSL](https://www.openssl.org/) installed via [Homebrew](https://brew.sh)
+and are running into an error related to those things when running step 3 above, it is likely
+because of [this issue](https://stackoverflow.com/questions/77874851/trouble-installing-specific-version-of-ruby-openssl-issue).
+Try these commands to fix it:
 
+1. Reinstall OpenSSL v3
 ```sh
-# install ruby on rails dependencies
-bundle install
+brew install openssl@3
+```
+2. Clear `rvm`'s cache
+```sh
+rvm cleanup
+```
+3. Install Ruby v3.3.0 with the path to the OpenSSL v3 installation:
+```sh
+rvm install ruby-3.3.0 --with-openssl-dir="$(brew --prefix openssl@3)"
+```
 
-# install node js dependencies
-npm install
+## Installation
 
-# needed to run startup script
+1. Install frontend dependencies:
+```sh
+npm install
+```
+2. Install backend dependencies:
+```sh
+bundle install
+```
+1. Install [`overmind`](https://github.com/DarthSim/overmind) which is needed to run the [Usage](#usage) scripts (this assumes you are on MacOS and have already installed [Homebrew](https://brew.sh)):
+```sh
 brew install overmind
 ```
 
-## Startup
+## Usage
+
+To preview the web app locally, run either of the following commands (we recommend the first when developing):
 
 ```sh
 ./bin/dev # For HMR (hot module reloading)
 # or
 ./bin/dev static # Without HMR, statically creating the bundles
 ```
 
-Visit [http://localhost:3000](http://localhost:3000) and see your React On Rails app running!
+Visit [http://localhost:3000](http://localhost:3000) and see your React On Rails app running on your local development server!
 
-To quit the server press `Ctrl + C`. You may need to do this twice (not sure why).
+To quit the server press `Ctrl + C`. **You may need to do this twice (not sure why)**.
 
-## Common Commands
+## Common Development Commands
 
 ```sh
-# Run tests
+# Run unit tests
 rake
 
 # Run brakeman (scan for common Rails vulnerabilities)
@@ -51,6 +95,13 @@ rubocop --autocorrect
 npx prettier client/ --write
 ```
 
+For more information on these tools, see their documentation:
+
+- [Rake](https://guides.rubyonrails.org/v4.1/command_line.html#rake)
+- [Brakeman](https://brakemanscanner.org/docs/)
+- [Rubocop](https://docs.rubocop.org/rubocop/index.html)
+- [Prettier](https://prettier.io/docs/)
+
 ## Directory Structure
 
 This is a [React on Rails](https://www.shakacode.com/react-on-rails/docs/) app. The Rails part (backend) is under the `app/` directory and the React frontend is under the `client/` directory.
@@ -59,7 +110,7 @@ This is a [React on Rails](https://www.shakacode.com/react-on-rails/docs/) app.
 
 [S3](https://aws.amazon.com/s3/) is used to store assignment snapshot files. To access S3, you will need to:
 
-1. Request `PowerUserAccess` through [UC Berkeley's managed AWS instance](https://technology.berkeley.edu/bcloud-aws-central-faq). (You should contact [Lisa Yan](mailto:yanlisa@berkeley.edu) and [Michael Ball](ball@berkeley.edu) to get added properly since you will need access to the correct S3 buckets.)
+1. Request `PowerUserAccess` through [UC Berkeley's managed AWS instance](https://technology.berkeley.edu/bcloud-aws-central-faq). (You should contact [Lisa Yan](mailto:yanlisa@berkeley.edu) and [Michael Ball](ball@berkeley.edu) to get added properly since you will need access to the correct S3 buckets. See also: [internal onboarding docs](https://docs.google.com/document/d/1KhpRW0GYBY-HRSRG8b6z3EbRSFQJaqVPPUsha_puY2I/edit?tab=t.0).)
 2. Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
 3. For local development, [configure logging in with SSO](https://docs.aws.amazon.com/sdkref/latest/guide/access-sso.html) with these configuration values and use the `AWSPowerUserAccess` role:
 ```sh