Cyphers/master to r20 (#2978)

* nbench special handle op name for "NNP_XXX" (#2883)

* remove throw from dtor (#2854)

* add special handle for the 'NNP_xx' op name

* style

* add special handle for the 'NNP_xx' op name

* style

* use description as suggest by bob

* Remove parent from PlaidML tensor initializer (#2923)

* Remove parent from PlaidML tensor initializer

* Remove plaidml tensor parent plumbing

* style

* Add support for move semantics to AlignedBuffer (#2956)

* Add move operations to AlignedBuffer

* unit test

* Create mkldnn primitives at first iteration for codegen - part2 (#2859)

* Create mkldnn primitives at first iteration for CODEGEN.

 OPs: add, lstm, and rnn.

*  OPs: batchnorm.

*  OPs: concat and lrn.

Remove dead code.

* Skip in place concat, relu, reshape, and slice when building node_primitive_string_deps_index map.

* Change NGRAPH_ASSERT to NGRAPH_CHECK.

* Address PR Feedback.

* Create mkldnn primitives at first iteration for CODEGEN.
 OPs: convertlayout, relu, leakyrelu, boundedrelu, sigmoid, softmax, slice.

* Fix bugs.

*  OPs: quantizedconcat.

Check if there are descriptors before emitting code to read desc_file.

*  OPs: convolution backward.

Use macro to write mkldnn memory dims to generated file.

*  OPs: MaxPoolWithIndices and MaxPoolWithIndicesBackprop.

Add unit tests for MaxPoolWithIndices, MaxPoolWithIndicesBackprop, and MaxPoolBackprop.

* Fix style error.

*  OPs: AvgPoolBackprop and MaxPoolBackprop.

Add unit test for AvgPoolBackprop.

*  OPs: DeconvolutionBias.

*  OPs: Quantize and Dequantize.

*  OPs: QuantizedDot and QuantizedDotBias.

* Use reference kernel for QuantizedConvolution for CODEGEN when mkldnn does not support the parameter types.
Get scales for quantization ops in cpu_emitter.

* Fix Windows build error: add CPU_BACKEND_API.

* Use template for quantization ops.

*  OPs: QuantizedMatmul.

Emit referece kernel for QuantizedDot in CODEGEN.

* Remove QuantizedDot from get_scale_index.

* Address PR feedback.

* [FusedOps] Split (#2951)

* Split op skeleton

* Two ways to construct a fused Split to be able to use it in onnx importer

* refactor: move the util::split() helper functions to the core

* Split's decompose_op() implementation using a helper function

* Use fused Split in the onnx_importer

* Code formatting

* PR feedback

* Split helpers moved to ngraph/builder

* Basic UT - split a 1D tensor to 3 equal parts

* UT: Split 2D tensor into variable length parts

* Code formatting

* Catch the proper type of exception in the onnx_importer split()

* Initialize members in the correct order

* Type prop tests for Split

* Code formatting

* PR feedback

* Add more infrastructure for specialization of cloned graphs (#2949)

* Virtualize some things that crash when layout descriptor is missing

* More shape specialization

* (very bare) skeleton for dyn elimination

* Miscellaneous

* Lift i32->int64-only restriction on constant folding for Convert

* Add constant folding for ShapeOf, and some tests for new constant folders

* Tests for DynElimination

* Rename specialize_shapes to specialize_function, and add a unit test for value substitution

* Roll back overeager API change in dyn slice bprop (it has to handle right-indexed axes; bummer)

* Add a test for dynamic usage of transpose op

* Fix warning/error about variable shadowing

* Strengthen checks in apply_permutation

* Propagate Constant shapes through Transpose

* Add CHANGE_DYNAMIC_STATE where appropriate

* PR feedback, and fix unit test failure

* Fix PR reference in comment

* PR comments

* Comments for helper funcs

* Remove unique_ptr indirection for the AlignedBuffers

* Fix incorrect indexing of AlignedBuffer vector (whoops\!)

* Remove unnecessary CHANGE_DYAMIC_STATEs

* De-update pass property unit test for const folding

* Replace mystery runes with all_pass_property_off

* Change FusionType to enum class and use EnumMask (#2957)

* constexpr ctor for EnumMask

* added pass properties to core passes.

* change fusion type to have better type safety.

* refactor to use enum mask.

* remove extra code.

* added constants for FusionType backward compatibility.

* spelling.

* grammar fix.

* update visualize tree file extenstions and output formats (#2954)

* update visualize tree file extenstions and output formats

* fix runtime error

* Update version, clean up ToC, add more detail to section on inspectin… (#2947)

* Update version, clean up ToC, add more detail to section on inspecting graphs...

* Minor adjustments to version module
* Move distributed training page to extras since there's not much there
* Fix links that break when doing that
* Consistent casing on section titles
* Orphan governance page so we don't have blank/empty links
* Update release notes with new version module structure

* PR feedback

* Allow NGRAPH_VISUALIZE_TREE_OUTPUT_SHAPES to output partial shapes (#2959)

* Remove functions from cpu which were moved to core (#2962)

* Remove functions from cpu which were moved to core

* Fix a typo

* Remove unused function

* Move zero padded conv fusions from CPUFusion to CoreFusion. (#2969)

* Move zero padded conv fusions from CPUFusion to CoreFusion.

* Address PR feedback: move unit tests to core_fusion.

* Fix Convert for boolean output type in CODEGEN. (#2958)

* Create tensor for the primary backend (#2970)

* create tensor for the primary backend

* move private objects to protected

*  [Fused] LeakyRelu op (#2919)

* [Fused] LeakyRelu op

* Add LeakyRelu to serializer

* Add unit tests

* Fix merge branch 'master' into mkarzyns/fused_leaky_relu

* Change broadcasting rules to NumPy style

* Remove std:: and ngraph:: prefixes

* Rename CPU Runtime LeakyRelu to CPULeakyRelu

* Style apply

* Fix cpu_fusion.fuse_leaky_relu test

* Use eigen's tanh in the fused sigmoid multiply kernel (#2946)

* Merge branch 'master' into mkarzyns/fused_leaky_relu

* Add LeakyRelu to Intel GPU backend op list

* Add LeakyRelu to Intel GPU backend op list

* Make private members protected in hybrid classes (#2975)

* make private members protected in hybrid classes

* allow overriding the passes

* [ONNX] Unit tests for QLinearMatMul (#2706)

* [ONNX] Unit test models for QLinearMatMul

* [ONNX] Extended types support for NgraphTestCase

* [ONNX] Move the value comparators to the NgraphTestCase class

* Add test cases

* Add shape checking

* disable GPU tests

* IntelGPU backend: Switch to clDNN which is compatible with gcc4.8 (#2961)

* Added accessor methods for layer op attributes (#2964)

* Added accessor methods for layer op attributes

* style fixes and addressed PR feedback

* Add save/load API to runtime (#2955)

* API defined

* add unit test for save/load with INTERPRETER

* Update per review comments

* fix compiler error

* Backport fix from #2973 (#2976)

* CTCGreedyDecoder layer op (#2965)

* Added CTCGreedyDecoder layer op

* Added comment on seq_len validation checks

* Switch some get_inputs uses to use the newer inputs (#2968)

* Switch some get_inputs uses to use the newer inputs

* Review comments

* update a few files to build on windows (#2974)

* update a few files to build on windows

* more fixes

Author: diyessi

changes.md

@@ -2,6 +2,7 @@
 
 ## Passes
 * `LikeReplacement` pass must be run by all transformers.
+* `ngraph::pass::FusionType` is now an enum class. Constant values defined by `FusionType` are created for backward compatibility and will be removed in future releases.
 
 ## Nodes, Parameters
 

doc/sphinx/conf.py

@@ -73,11 +73,11 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.19'
+version = '0.20'
 
 # The Documentation full version, including alpha/beta/rc tags. Some features
 # available in the latest code will not necessarily be documented first
-release = 'rc4'
+release = '0.20.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -121,23 +121,20 @@
 
     # borrow this from the zephyr docs theme
     html_context = {
-        # 'show_license': html_show_license, we have a custom footer to attribute WTD and Sphinx contributors
-        # so we do not enable this
+        # 'show_license': html_show_license, we have a custom footer to attribute 
+        # RTD, WTD, and Sphinx contributors; so we do not enable this
         'docs_title': docs_title,
         'is_release': is_release,
         'theme_logo_only': False,
         'current_version': version,
-        'versions': ( ("latest", "/"),
-                     ("0.19.0", "/0.19.0/"),
+        'versions': ( ("latest", "../"),
+                     ("0.19.0", "/0.19.0/"), #not yet sure how we'll do this
                      ("0.18.0", "/0.18.0/"),
                      ("0.17.0", "/0.17.0/"),
                      ("0.16.0", "/0.16.0/"),
-                     ("0.15.0", "/0.15.0/"),
                     )
     }
 
-
-
 html_logo = '../ngraph_theme/static/favicon.ico'
 
 # The name of an image file (within the static path) to use as favicon of the
@@ -165,6 +162,25 @@
 }
 
 
+# Custom added feature to allow redirecting old URLs
+#
+# list of tuples (old_url, new_url) for pages to redirect
+# (URLs should be relative to document root, only)
+html_redirect_pages = [
+        ('backends', 'backend-support/index'),
+        ('core/core', 'core/overview.rst'),
+        ('core/fusion', 'core/fusion/index'),
+        ('frameworks/mxnet', 'frameworks/mxnet_intg.rst'),
+        ('frameworks/onnx', 'frameworks/onnx_intg.rst'),
+        ('frameworks/tensorflow', 'frameworks/tensorflow_connect.rst'),
+        ('frameworks/paddle', 'frameworks/paddle_integ.rst'),
+        ('inspection/inspection', 'inspection/index'),
+        ('releases/release-notes', 'releases/index'),
+        # ('getting_started/getting_starting', 'getting_started/index'),
+        ('project/project', 'project/index'),
+        ('python_api/', 'python_api/index'),
+]
+
 # -- Options for HTMLHelp output ------------------------------------------
 
 # Output file base name for HTML help builder.
@@ -227,7 +243,6 @@
 .. |codename| replace:: Intel nGraph
 .. |project| replace:: Intel nGraph Library
 .. |InG| replace:: Intel® nGraph
-.. |nGl| replace:: nGraph library
 .. |copy|   unicode:: U+000A9 .. COPYRIGHT SIGN
    :ltrim:
 .. |deg|    unicode:: U+000B0 .. DEGREE SIGN
@@ -239,7 +254,7 @@
 .. |trade|  unicode:: U+02122 .. TRADEMARK SIGN
    :ltrim:
 .. |reg|    unicode:: U+000AE .. REGISTERED TRADEMARK SIGN
-   :ltrim:
+   :ltrim:   
 
 """
 

doc/sphinx/ngraph_theme/ngversions.html

@@ -1,15 +1,16 @@
   <div class="rst-versions" data-toggle="rst-versions" role="note" aria-label="versions">
     <span class="rst-current-version" data-toggle="rst-current-version">
       <span class="docvs">nGraph Compiler stack</span>
-      v: {{ version }}           <dd><a href="../docs/latest">{{release}}</a></dd>
-      <span class="fa fa-caret-down"></span>
+      v: {{ version }}
+      <span class="▼"></span>
     </span>
     <div class="rst-other-versions">
       <dl>
         <dt>{{ _('Previous Verified Versions') }}</dt>
         <dd><!-- Until our https://docs.ngraph.ai/ publishing is set up, we link to GitHub -->  
           <ul>
-           <li><a href="https://github.com/NervanaSystems/ngraph/releases/tag/v0.18.1">0.18</a></li>
+           <li><a href="https://github.com/NervanaSystems/ngraph/releases/tag/v0.19.0-rc.2">0.19.0-rc.2</a></li>
+           <li><a href="https://github.com/NervanaSystems/ngraph/releases/tag/v0.18.1">0.18.1</a></li>
            <li><a href="https://github.com/NervanaSystems/ngraph/releases/tag/v0.17.0-rc.1">0.17.0-rc.1</a></li>
            <li><a href="https://github.com/NervanaSystems/ngraph/releases/tag/v0.16.0-rc.3">0.16.0-rc.3</a></li>
            <li><a href="https://github.com/NervanaSystems/ngraph/releases/tag/v0.16.0-rc.2">0.16.0-rc.2</a></li>

doc/sphinx/source/buildlb.rst

@@ -23,7 +23,7 @@ packages and prerequisites:
    :widths: 25, 15, 25, 20, 25
    :escape: ~
 
-   CentOS 7.4 64-bit, GCC 4.8, CMake 3.5.0, supported, ``wget zlib-devel ncurses-libs ncurses-devel patch diffutils gcc-c++ make git perl-Data-Dumper`` 
+   CentOS 7.4 64-bit, GCC 4.8, CMake 3.9.0, supported, ``wget zlib-devel ncurses-libs ncurses-devel patch diffutils gcc-c++ make git perl-Data-Dumper`` 
    Ubuntu 16.04 or 18.04 (LTS) 64-bit, Clang 3.9, CMake 3.5.1 + GNU Make, supported, ``build-essential cmake clang-3.9 clang-format-3.9 git curl zlib1g zlib1g-dev libtinfo-dev unzip autoconf automake libtool``
    Clear Linux\* OS for Intel® Architecture version 28880, Clang 8.0, CMake 3.14.2, experimental, bundles ``machine-learning-basic c-basic python-basic python-basic-dev dev-utils``
 
@@ -58,7 +58,7 @@ the ``CMakeLists.txt`` file for other experimental options' details:
    -- NGRAPH_NOP_ENABLE:               ON
    -- NGRAPH_GPUH_ENABLE:              OFF
    -- NGRAPH_GENERIC_CPU_ENABLE:       OFF
-   -- NGRAPH_DEBUG_ENABLE:             OFF
+   -- NGRAPH_DEBUG_ENABLE:             OFF  # Set to "ON" to enable logging
    -- NGRAPH_ONNX_IMPORT_ENABLE:       OFF
    -- NGRAPH_DEX_ONLY:                 OFF
    -- NGRAPH_CODE_COVERAGE_ENABLE:     OFF

doc/sphinx/source/conf.py

@@ -73,11 +73,11 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.19'
+version = '0.20'
 
 # The Documentation full version, including alpha/beta/rc tags. Some features
 # available in the latest code will not necessarily be documented first
-release = '0.19.0'
+release = '0.20.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/sphinx/source/core/constructing-graphs/import.rst

@@ -26,8 +26,7 @@ usually named ``<some_model>.onnx`` or ``<some_model>.onnx.pb``. These
    an ``.onnx.pb`` formatted file, you should be able to run the inference without
    needing to dig into anything from the "Frameworks" sections. You will, however, 
    need to have completed the steps outlined in our :doc:`../../buildlb` guide.  
-   If you intend to build nGraph for distributed-training, you will need 
-   to follow instructions on the documentation for :doc:`../../distr/index`.
+
 
 To demonstrate functionality, we'll use an already-serialized CIFAR10 model 
 trained via ResNet20. Remember that this model has already been trained and 

doc/sphinx/source/distr/index.rst

@@ -72,34 +72,20 @@ boost compared to native implementations. For a high-level overview, see the
    backend-support/cpp-api.rst
 
 
-.. toctree::
-   :maxdepth: 1
-   :caption: Distributed Training
-
-   distr/index.rst
-
-
 .. toctree::
    :maxdepth: 1
    :caption: Inspecting Graphs
 
    inspection/index.rst
 
-.. toctree::
-   :maxdepth: 1
-   :caption: Tutorials
-
-   tutorials/index.rst
-
 
 .. toctree::
    :maxdepth: 1
    :caption: Project Metadata
 
-   project/introduction.rst
    project/release-notes.rst
+   project/introduction.rst
    project/contribution-guide.rst
-   project/governance.rst
    project/doc-contributor-README.rst
    project/index.rst
    project/extras.rst 

doc/sphinx/source/index.rst

@@ -3,18 +3,30 @@
 Visualization Tools
 ###################
 
-nGraph provides serialization and deserialization facilities along with the 
-ability to create image formats. When visualization is enabled, a ``dot`` file 
-is generated, along with a ``png``. The default can be adjusted by setting the 
+nGraph provides serialization and deserialization facilities, along with the 
+ability to create image formats or a PDF. 
+
+When visualization is enabled, a ``dot`` file gets generated, along with a 
+``png``. The default can be adjusted by setting the 
 ``NGRAPH_VISUALIZE_TREE_OUTPUT_FORMAT`` flag to another format, like PDF. 
 
-Note: Large graphs are usually not legible with formats like PDF.  
+.. note:: Large graphs are usually not legible with formats like PDF.  
+
+Large graphs may require additional work to get into a human-readable format. 
+On the back end, very long edges will need to be cut to make (for example) a 
+hard-to-render training graph tractable. This can be a tedious process, so 
+incorporating the help of a rendering engine or third-party tool like those 
+listed below may be useful.  
+
 
-Large graphs may require additional work to get into a human-readable format. We 
-have provided a script to convert the `most common default output`_, nGraph JSON,
-to an output that is better able to handle detailed graphs; however, we do not 
-offer user support for this script. After running the script, you should have a 
-``.graphml`` that can be imported and inspected with third-party tools like: 
+.. Additional scripts
+.. ==================
+
+.. We have provided a script to convert the `most common default output`_, nGraph 
+.. ``JSON``, to an output that is better able to handle detailed graphs; however, 
+.. we do not offer user support for this script. The script will produce a 
+.. ``.graphml`` file that can be imported and inspected with third-party tools 
+.. like: 
 
 #. `Gephi`_
 
@@ -23,9 +35,9 @@ offer user support for this script. After running the script, you should have a
 .. #. `Netron`_ support tentatively planned to come soon
 
 
-.. _CMakeLists.txt: https://github.com/NervanaSystems/ngraph/blob/master/CMakeLists.txt
-.. _most common default output: https://github.com/NervanaSystems/ngraph/contrib/tools/graphml/ngraph_json_to_graphml.py
-.. _Netron: https://github.com/lutzroeder/netron/blob/master/README.md
-.. _Gephi: https://gephi.org
-.. _Cytoscape: https://cytoscape.org
-
+.. _CMakeLists.txt: https:github.com/NervanaSystems/ngraph/blob/master/CMakeLists.txt
+.. _most common default output: https:github.com/NervanaSystems/ngraph/contrib/tools/graphml/ngraph_json_to_graphml.py
+.. _visualize_tree.cpp: https://github.com/NervanaSystems/ngraph/blob/master/src/ngraph/pass/visualize_tree.cpp
+.. _Netron: https:github.com/lutzroeder/netron/blob/master/README.md
+.. _Gephi: https:gephi.org
+.. _Cytoscape: https:cytoscape.org

doc/sphinx/source/inspection/index.rst

@@ -1,7 +1,7 @@
 .. contribution-guide:
 
 ##################
-Contribution Guide
+Contribution guide
 ##################
 
 License