Question: will this project be deprecated in favor of PlatformIO?

[ianfixes]

I'm trying to set up a template for Continuous Integration of Arduino library code (both compilation and unit tests), using Travis CI.

A Google search led me to travis-ci-arduino as one of the best examples of how to do the "does it compile" test on Travis. However, it looks like PlatformIO is what Travis recommends for this:
https://docs.travis-ci.com/user/integration/platformio/#Examples

Are there advantages to using this method instead of PlatformIO?

[probonopd]

Yes, no additional dependency (PlatformIO) and closer to "original" Arduino builds.

[ianfixes]

Are you saying that you evaluated PlatformIO for your purposes and found it to be lacking, or that you are satisfied with the solution you have?

Side question, how are you running these CI scripts from your local machine prior to pushing code to github? On OSX, the only practical thing I could do was read through install.sh myself and copy those arguments to the command line -- I have to invoke a binary that's in a different path (capital-A Arduino at /Applications/Arduino.app/Contents/MacOS/Arduino, which is annoying). Is that how you do it?

[probonopd]

Personally I use both PlatformIO and travis-ci-arduino. When I want to test that something compiles well on Arduino, I use travis-ci-arduino. I am not running these CI scripts from my local machine.

[ianfixes]

Hmm. Does this imply that you've noticed instances where PlatformIO passes but travis-ci-arduino fails?

[probonopd]

No, but I didn't really compare.

[ianfixes]

It turns out that PlatformIO is a paid service and not a freely available program. I've submitted a pull request to Travis CI to update their docs accordingly.

[ladyada]

FYI we just fixed some bugs, if you haven't had success - it should all be working now thx!

[ianfixes]

It's not that I haven't had success... it's that this script doesn't allow for unit testing. I am porting the CI idea from your install.sh script into a ruby gem. So far I have everything except the actual compilation command implemented in my sample script. Unit tests will be accomplished by writing some C++ libraries for Arduino mocks (allowing you to control the clock, fake serial ports, etc) that I'll bundle with the gem.

I'd love your comments in a week or so when I get this into a more stable state. Would a CI + unit testing script (inspired by the same "use one script everywhere" design of this repo) be of interest to you?

[ladyada]

ive never used ruby so cannot test, but if you have a PR that doesn't change around the original install.sh plz submit a PR!

[ianfixes]

Unfortunately, mine (https://github.com/ianfixes/arduino_ci) is different from install.sh. It still makes a minimal impact on the library being tested, but as shown here:

# .travis.yml
sudo: false
language: ruby
script:
   - bundle install
   - bundle exec arduino_ci_remote.rb

You need to add a Gemfile:

source 'https://rubygems.org'
gem 'arduino_ci'

And that's basically it -- a YAML file controls the rest. A working example of a CI-enabled Arduino project would look like this:
https://github.com/ianfixes/Arduino-Queue.h/tree/2018-01-25_ci

Most notably (and the point of this whole exercise), you can unit test your library with files like this:

#include <ArduinoUnitTests.h>
#include "../Queue.h"

unittest(empty_queue)
{
  Queue<int> queue(5);
  assertEqual(0, queue.count()); // initially empty
  assertEqual(0, queue.peek());  // 0 happens to be the default int val
  assertEqual(0, queue.pop());   // popping when empty also returns the default
  assertEqual(0, queue.count()); // still empty
}

unittest(fill_and_empty) 
{
  Queue<int> queue(5);
  queue.push(11);
  queue.push(19);
  queue.push(44);
  assertEqual(3,  queue.count());
  assertEqual(11,  queue.pop());
  assertEqual(19, queue.pop());
  assertEqual(44, queue.pop());
}

unittest_main()

Example output from a CI job (https://travis-ci.org/ianfixes/Arduino-Queue.h/builds/333540256#L522):

$ bundle exec arduino_ci_remote.rb
Downloading Arduino binary with wget
--2018-01-26 01:15:03--  https://downloads.arduino.cc/arduino-1.8.5-linux64.tar.xz
Resolving downloads.arduino.cc (downloads.arduino.cc)... 158.69.94.227
Connecting to downloads.arduino.cc (downloads.arduino.cc)|158.69.94.227|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 103807940 (99M) [application/octet-stream]
Saving to: ‘arduino-1.8.5-linux64.tar.xz’
100%[======================================>] 103,807,940 3.05MB/s   in 33s    
2018-01-26 01:15:36 (3.03 MB/s) - ‘arduino-1.8.5-linux64.tar.xz’ saved [103807940/103807940]
Extracting archive with tar
Installing library under test...                                               ✓
Library installed at /home/travis/Arduino/libraries/Arduino-Queue.h...         ✓
Switching to board for uno (arduino:avr:uno)...                                ✓
Unit testing queue-tests.cpp...
TAP version 13
1..5
# Subtest: empty
    ok 1 - assertEqual 0 == queue.count()
    ok 2 - assertEqual 0 == queue.peek()
    ok 3 - assertEqual 0 == queue.pop()
    ok 4 - assertEqual 0 == queue.count()
    1..4
ok 1 - empty
# Subtest: filling_and_emptying
    ok 1 - assertEqual 0 == queue.count()
    ok 2 - assertEqual testSize == queue.space()

(snip)

[ianfixes]

@ladyada I wanted to follow up on this.

What I wrote is a feature-complete replacement for your install.sh, using install.sh as my guide. My MVP was to support the same operations:

• Automatically download the Arduino executable
• Install dependent libraries
• Compile each example on each of several platforms
• Be able to customize which boards are used for compiling each example, since not all hardware supports each library

Then, I added the following features that you might find useful:

• Ability to write and run unit tests
• Install dependent libraries from GitHub or other sources
• Run CI tests on your local machine too (OSX, Windows, and Linux are supported)
• Ability to define additional platforms to run tests on
• Report out-of-bounds and use-after-free memory errors, if the compiler supports it

I've submitted adafruit/Adafruit_FONA#91 to show how my system works in practice, the results are here: https://travis-ci.org/adafruit/Adafruit_FONA/builds/350489990

Relevant portion:

...Unit testing comms.cpp                                                      ✓
Setting compiler warning level...                                              ✓
Verifying FONA3G_setbaud.ino...                                                ✓
Switching to board for leonardo (arduino:avr:leonardo)...                      ✓
Verifying FONA3G_setbaud.ino...                                                ✓
Switching to board for uno (arduino:avr:uno)...                                ✓
Verifying FONA_SMS_Response.ino...                                             ✓
Switching to board for leonardo (arduino:avr:leonardo)...                      ✓
Verifying FONA_SMS_Response.ino...                                             ✓
Switching to board for uno (arduino:avr:uno)...                                ✓
Verifying FONAtest.ino...                                                      ✓
Switching to board for leonardo (arduino:avr:leonardo)...                      ✓
Verifying FONAtest.ino...                                                      ✓
Switching to board for uno (arduino:avr:uno)...                                ✓
Verifying FONAtest_KEY_mod.ino...                                              ✓
Switching to board for leonardo (arduino:avr:leonardo)...                      ✓
Verifying FONAtest_KEY_mod.ino...                                              ✓
Switching to board for uno (arduino:avr:uno)...                                ✓
Verifying GPS.ino...                                                           ✓
Switching to board for leonardo (arduino:avr:leonardo)...                      ✓
Verifying GPS.ino...                                                           ✓
Switching to board for uno (arduino:avr:uno)...                                ✓
Verifying IncomingCall.ino...                                                  ✓
Switching to board for leonardo (arduino:avr:leonardo)...                      ✓
Verifying IncomingCall.ino...                                                  ✓

Are there any features I'm missing?

[ladyada]

sounds cool - we've got 1000 github repos so not sure when we'll get to implementing this but next time we do a FONA sweep we will check it out :)