Skip to content

Standards Alignment Prototype

Jump to bottom Edit New page
science edited this page Feb 21, 2012 · 25 revisions

Priorities

  1. Get data services working (project elements described below)
    1. Just get alignment data going (Jim and Walt)
    2. All the features discussed (e.g., other languages, restricted access) (Jim, Walt, and Austin)
  2. Get a trimmed down node running w/out slice -- ready for development. Install data services on it - show it works. (Lou)
    1. Modify install should map services and associated view dependencies, so that if xyz services are installed, only xyz views are installed
  3. Prove out test cases on this work (Joe, Susan, Aaron, Agilix)
  4. Get docs in shape for examples and code lib (Damon, Simulator)
  5. Get one more example data service going: maybe ratings (Jim)
  6. Get replicate working better (Lou/John)
    1. Make replication meet the goals of the spec (security, data integrity) -- clarify anything in the code or spec that is not clear
      1. Consider Jim's idea of using incoming databases and conditional change monitor behavior (address security, data integrity, and duplicate document/disk space issues)
      2. Consider pull replication via Jim's idea to develop a service that allows a node to subscribe for push from a different node
  7. Get publish signing on a node working (Austin)

Development Approach

  • Rapid prototyping producing documentation on GitHub wiki
  • Separate branch for standards alignment prototyping using pull request model
  • Part of the asynchronous development process is writing tests and doing code reviews

Design Goal: Reduce Server Cost, Improve Performance, Reduce Resource Allocation

Reducing size of data and cost/size of server

Design Goal: Simplicity

  • Every step can be done with curl

Tests

  • Unit tests for components (may be language specific, using new or existing frameworks - should be callable from our overall test runner)
  • Integration/functional tests (use our existing framework)

Prototype Elements and Associated PT Stories

Map/Extraction Functions: Standards Alignment

Provide these capabilities (maybe one per view or maybe integrated):

  • Give me all standards aligned to URL/resource
  • Give me all URLs associated with a standard
  • Give me all alignment data
  • Give me all aligned resources for a URL pattern / site

1.1 Story (Jim): What defines "alignment data"?

  • Anything that starts with "http://purl.org/ASN/resources/*"
  • XML with dct:conforms_to
  • Criteria for relevant paradata (specific verbs for paradata)
  • validate with Joe and Brian to identify rules of thumb
  • define regex
  • wiki article

1.1 Story (Jim): How do we organize (pattern) the data so it can be accessed in a consistent manner

  • by date
  • by sender
  • by domain name (identity)
  • by url
  • by standard
  • related to API format definition

1a Story (Jim): How to use the Java View Server to write the map functions?

  • How to setup Maven
  • Java
  • not rich query interface
  • source code repository for views?
  • wiki article

1.2 Story: How to use the JavaScript to write the map functions?

  • Javascript
  • not rich query interface
  • wiki article

Calling / API Format

Use restful-like interface but modified to support parameters

1.1 Story (Walt/Jim): What is the API definition?

  • related to organize (pattern) data story
  • leverage CouchDB tools, but abstract the API to use them (e.g., how to use CouchDB List function/view/regex, so we may need abstract a way to pass arbitrary parameters (e.g., domain name regex) through API to a CouchDB List?)
  • unrestricted, but obivously no access to internal views

6 Story (Austin, LM security): Define how to restrict access to the API

  • Define security
    • Each map/extract design doc needs to hold an arbitrary parameter that describes if the map/extract is permitted to be accessed from the "extract" interface (extract must honor this)
    • Possibly also enable access from "extract" by convention of naming the map/extract view function with a particular prefix like "view-[name]" (e.g., "view-by-standards-url")
    • role-based access (single role initially as discriminator)?
  • document in wiki

6 Story (Austin): How to restrict access to the API?

  • Implement security
    • Each map/extract design doc needs to hold an arbitrary parameter that describes if the map/extract is permitted to be accessed from the "extract" interface (extract must honor this)
    • Possibly also enable access from "extract" by convention of naming the map/extract view function with a particular prefix like "view-[name]" (e.g., "view-by-standards-url")
    • role-based access

Result Formats

  • Use obtain result format
    • NOTE: Dan suggests that a new result format is not required/desirable.
    • Relax certain LR envelope elements for output by creating a new envelope doc_type: "result_data"
    • "result_data" doc_type would have compatibility with "resource_data" but used for returning data which are composites of inputs
  • Design goals include:
    • Ability to chain result sets together
    • Optional/bonus: Ability to submit result_data docs back into LR network

1.1 Story (Walt, Dan): What is the result format for the service?

  • model obtain
  • wiki article

1.1 Story (Walt, Dan): What is the result format for the envelope?

  • NOTE: Dan suggests new doc_type not a good idea..
  • model after resource_data, but with relaxed element definitions
  • doc_type="result_data"
  • wiki article

Filter Functions: Standards Alignment and/or General Purpose

  • Examine whether performance requires some filters to be map/extractions.
    • Filter by sender
    • Filter by date

Addressed in above 1a story (organize data, API definition)

View Server

  • Templates for view
    • Helper functions and other templatizing tools
    • How-to examples
  • Java view server
    • Performance: make sure >= javascript view server
  • Maybe provide javascript view servers templates as learning examples

4 Story (Jim): Provide example map functions in simulator

4 Story (TBD): Host simulator

Second Project: Submission Envelopes Signing by Nodes

Signing Submissions via Node

  • Wizard for creating public/private keys and uploading to key servers
  • Associate public/private keys to basic_auth login users
  • Signing code on submission inside node
    • Could sign all stuff with a single node public/private key and associate basic_auth credentials as "signed on behalf of"
Add a custom footer
Add a custom sidebar
Clone this wiki locally