Merge in developmental branch

.gitignore

@@ -0,0 +1,3 @@
+config/bash/env.sh
+**/.ipynb*
+tools/

README.md

@@ -1,2 +1,65 @@
-# insight-crystal-project
-Crystal Database
+# Crystal-Base
+# Table of Contents
+1. [Protein Crystallization Challenges](README.md#Protein-Crystallization-Challenges)
+2. [Dataset](README.md#Dataset)
+3. [Architecture](README.md#Architecture)
+4. [Web App](README.md#Web-App)
+
+## Protein Crystallization Challenges
+
+Crystal-Base is an image classification pipeline that reports whether or not an image contains a protein crystal. Crystal-Base caters towards both academic and industrial researchers who are running large scale HTS protein crystallization projects who do not want to spend time on the mundane task of identifying possible protein crystals from their crystallization screens.
+
+![Image of Protein Crystal Screen](images/Crystal-Screen.png)
+
+## Dataset
+
+All protein crystal data was obtained from the [Marco Database](https://marco.ccr.buffalo.edu/)
+
+## Architecture
+![Image of Pipeline](images/Pipeline.png)
+
+### Setting up AWS
+
+Crystal-base uses **pegasus** to setup AWS clusters with configurations in **yaml** files.
+
+Run `./main.sh --setup-pegasus` to install pegasus.
+
+Run `./main.sh --setup-config` to setup the bash environment
+
+Run `./main.sh --setup-database` to setup a Postgres database.
+
+Run `./main.sh --setup-hadoop` to setup a hadoop cluster.
+
+Run `./main.sh --setup-spark` to setup a spark cluster
+
+Run `./main.sh --setup-web-server` to setup a web server.
+
+### Ingestion
+
+Crystal base ingests files from the [Marco Database](https://marco.ccr.buffalo.edu/) using **bash** and an EC2 instance to an S3 bucket.
+
+Run `source src/bash/ingestMarcoFiles.sh && ingestMarcosFiles` to ingest files
+
+### Training
+
+Crystal-base uses transfer learning [inceptionv3](https://www.tensorflow.org/tutorials/images/image_recognition) training model to identify protein drop crystals from the [Marco Database](https://marco.ccr.buffalo.edu/).
+
+Run `python3 src/python/classifyImagesTrainer.py` to train the image classifier and write to a Postgres Database.
+
+### Distributed Image Classification
+
+Data is ingested with Spark from S3 buckets and batch processedon a distributed tensorflow cluster using executors running their own tensorflow instances.
+
+Run `./main.sh --classify-images simple` to use the simple test classifier. Results are expected to output to a Postgres database.
+
+## Web App
+
+Crystal-base has a web interface that runs its own instance of the trained tensorflow model.
+
+![Image of Web App](images/crystal-base-web-app.png)
+
+Run `./main.sh --run-webs-server` to run this web-server instance.
+
+### Try it out!
+
+Upload protein crystal jpeg images at [Crystal-Base](http://www.crystal-base.com)

config/database-cluster/master.yml

@@ -0,0 +1,10 @@
+purchase_type: on_demand
+subnet_id: subnet-01bc006215b4bfa36
+num_instances: 1
+key_name: insight-aws2
+security_group_ids: sg-08d81fc64ffc3f309
+instance_type: m4.large
+tag_name: crystal-project-database-cluster
+vol_size: 100
+role: master
+use_eips: true

config/database-cluster/pg_hba.conf

@@ -0,0 +1,100 @@
+# PostgreSQL Client Authentication Configuration File
+# ===================================================
+#
+# Refer to the "Client Authentication" section in the PostgreSQL
+# documentation for a complete description of this file.  A short
+# synopsis follows.
+#
+# This file controls: which hosts are allowed to connect, how clients
+# are authenticated, which PostgreSQL user names they can use, which
+# databases they can access.  Records take one of these forms:
+#
+# local      DATABASE  USER  METHOD  [OPTIONS]
+# host       DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
+# hostssl    DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
+# hostnossl  DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
+#
+# (The uppercase items must be replaced by actual values.)
+#
+# The first field is the connection type: "local" is a Unix-domain
+# socket, "host" is either a plain or SSL-encrypted TCP/IP socket,
+# "hostssl" is an SSL-encrypted TCP/IP socket, and "hostnossl" is a
+# plain TCP/IP socket.
+#
+# DATABASE can be "all", "sameuser", "samerole", "replication", a
+# database name, or a comma-separated list thereof. The "all"
+# keyword does not match "replication". Access to replication
+# must be enabled in a separate record (see example below).
+#
+# USER can be "all", a user name, a group name prefixed with "+", or a
+# comma-separated list thereof.  In both the DATABASE and USER fields
+# you can also write a file name prefixed with "@" to include names
+# from a separate file.
+#
+# ADDRESS specifies the set of hosts the record matches.  It can be a
+# host name, or it is made up of an IP address and a CIDR mask that is
+# an integer (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that
+# specifies the number of significant bits in the mask.  A host name
+# that starts with a dot (.) matches a suffix of the actual host name.
+# Alternatively, you can write an IP address and netmask in separate
+# columns to specify the set of hosts.  Instead of a CIDR-address, you
+# can write "samehost" to match any of the server's own IP addresses,
+# or "samenet" to match any address in any subnet that the server is
+# directly connected to.
+#
+# METHOD can be "trust", "reject", "md5", "password", "gss", "sspi",
+# "ident", "peer", "pam", "ldap", "radius" or "cert".  Note that
+# "password" sends passwords in clear text; "md5" is preferred since
+# it sends encrypted passwords.
+#
+# OPTIONS are a set of options for the authentication in the format
+# NAME=VALUE.  The available options depend on the different
+# authentication methods -- refer to the "Client Authentication"
+# section in the documentation for a list of which options are
+# available for which authentication methods.
+#
+# Database and user names containing spaces, commas, quotes and other
+# special characters must be quoted.  Quoting one of the keywords
+# "all", "sameuser", "samerole" or "replication" makes the name lose
+# its special character, and just match a database or username with
+# that name.
+#
+# This file is read on server startup and when the postmaster receives
+# a SIGHUP signal.  If you edit the file on a running system, you have
+# to SIGHUP the postmaster for the changes to take effect.  You can
+# use "pg_ctl reload" to do that.
+
+# Put your actual configuration here
+# ----------------------------------
+#
+# If you want to allow non-local connections, you need to add more
+# "host" records.  In that case you will also need to make PostgreSQL
+# listen on a non-local interface via the listen_addresses
+# configuration parameter, or via the -i or -h command line switches.
+
+
+
+
+# DO NOT DISABLE!
+# If you change this first entry you will need to make sure that the
+# database superuser can access the database using some other method.
+# Noninteractive access to all databases is required during automatic
+# maintenance (custom daily cronjobs, replication, and similar tasks).
+#
+# Database administrative login by Unix domain socket
+local   all             postgres                                peer
+
+# TYPE  DATABASE        USER            ADDRESS                 METHOD
+
+# "local" is for Unix domain socket connections only
+local   all             all                                     peer
+# IPv4 local connections:
+host    all             all             127.0.0.1/32            md5
+# IPv6 local connections:
+host    all             all             ::1/128                 md5
+# Allow replication connections from localhost, by a user with the
+# replication privilege.
+#local   replication     postgres                                peer
+#host    replication     postgres        127.0.0.1/32            md5
+#host    replication     postgres        ::1/128                 md5
+host     all             all             0.0.0.0/0               md5