Build and Deploy

Requirements

To Build :

• Apache Maven (Tested on v3.3.x)
• JDK (Tested on Oracle JDK 1.8)
• Working internet connection to retrieve maven dependencies

The following dependencies will be downloaded from Maven central. Feel free to look inside the pom.xml for the current versions being used.

• Apache Spark
• Apache Nutch
• Apache Kafka Client
• Apache Solr Client
• Scala

Note that the libraries like Solr-client, spark, kafka etc should match with your own deployment version. For instance, if you have Spark Cluster deployment of v1.6 with Scala 2.11, make sure to set them the same versions for the client libraries in pom.xml.

To deploy and run :

In Local Mode

• Apache Solr Server or Solr Cloud (Tested on v6.0.1): The necessary configuration files are placed under conf/solr directory

Build Instructions

• $ mvn clean org.apache.felix:maven-bundle-plugin:manifest install should give a sparkler-app/target/sparkler-app-xx.jar