Missing tfx.orchestration in Colab tutorials

[michaelwsherman]

System information

• Have I specified the code to reproduce the issue: Yes
• Environment in which the code is executed: Colab
• TensorFlow version: 2.11.0
• TFX Version: 1.12.0
• Python version: 3.8.16
• Python dependencies (from pip freeze output): This is massive, I've put it at the bottom.

Describe the current behavior
In TFX tutorial colabs, when I run the code:
tfx.orchestration.LocalDagRunner().run(whatever)
I get:
AttributeError: module 'tfx.v1' has no attribute 'orchestration'

I've seen this in two separate TFX tutorial notebooks: penguin_simple.ipynb and penguin_tfdv.ipynb. I am following the instructions in the notebook to restart the runtime after pipping TFX.

I've experienced the issue in penguin_simple.ipynb using colab on two separate Google accounts. In my own local dev environment, I can run the code from penguin_simple.ipynb and it works. In both colab and my own environment, print('TFX version: {}'.format(tfx.__version__)) outputs TFX version: 1.12.0.

Investigating a bit, in colab dir(tfx) shows no orchestration, but in my own environment orchestration is present.

My environment is Python 3.7.10 but colab is Python version 3.8.16, maybe this is related?

Describe the expected behavior
The notebooks should work in colab.

Standalone code to reproduce the issue
Not quite standalone code, but if you go to the Simple TFX Pipeline Tutorial page and open the Run in Google Colab link, then run through the notebook (including restarting after pip install -U tfx), you'll get this error on the second to last code block.

pip freeze:

absl-py==1.3.0
aeppl==0.0.33
aesara==2.7.9
aiohttp==3.8.3
aiosignal==1.3.1
alabaster==0.7.12
albumentations==1.2.1
altair==4.2.0
apache-beam==2.43.0
appdirs==1.4.4
arviz==0.12.1
astor==0.8.1
astropy==4.3.1
astunparse==1.6.3
async-timeout==4.0.2
atari-py==0.2.9
atomicwrites==1.4.1
attrs==21.4.0
audioread==3.0.0
autograd==1.5
Babel==2.11.0
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==5.0.1
blis==0.7.9
bokeh==2.3.3
branca==0.6.0
bs4==0.0.1
CacheControl==0.12.11
cachetools==4.2.4
catalogue==2.0.8
certifi==2022.12.7
cffi==1.15.1
cftime==1.6.2
chardet==3.0.4
charset-normalizer==2.1.1
click==7.1.2
clikit==0.6.2
cloudpickle==2.2.0
cmake==3.22.6
cmdstanpy==1.0.8
colorcet==3.0.1
colorlover==0.3.0
community==1.0.0b1
confection==0.0.3
cons==0.4.5
contextlib2==0.5.5
convertdate==2.4.0
crashtest==0.3.1
crcmod==1.7
cufflinks==0.17.3
cvxopt==1.3.0
cvxpy==1.2.2
cycler==0.11.0
cymem==2.0.7
Cython==0.29.32
daft==0.0.4
dask==2022.2.1
datascience==0.17.5
db-dtypes==1.0.5
debugpy==1.0.0
decorator==4.4.2
defusedxml==0.7.1
descartes==1.1.0
dill==0.3.1.1
distributed==2022.2.1
dlib==19.24.0
dm-tree==0.1.7
dnspython==2.2.1
docker==4.4.4
docopt==0.6.2
docutils==0.17.1
dopamine-rl==1.0.5
earthengine-api==0.1.335
easydict==1.10
ecos==2.0.10
editdistance==0.5.3
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.1/en_core_web_sm-3.4.1-py3-none-any.whl
entrypoints==0.4
ephem==4.1.3
et-xmlfile==1.1.0
etils==0.9.0
etuples==0.3.8
fa2==0.3.5
fastai==2.7.10
fastavro==1.7.0
fastcore==1.5.27
fastdownload==0.0.7
fastdtw==0.3.4
fasteners==0.18
fastjsonschema==2.16.2
fastprogress==1.0.3
fastrlock==0.8.1
feather-format==0.4.1
filelock==3.8.2
firebase-admin==5.3.0
fix-yahoo-finance==0.0.22
Flask==1.1.4
flatbuffers==22.12.6
folium==0.12.1.post1
frozenlist==1.3.3
fsspec==2022.11.0
future==0.16.0
gast==0.4.0
GDAL==2.2.2
gdown==4.4.0
gensim==3.6.0
geographiclib==1.52
geopy==1.17.0
gin-config==0.5.0
glob2==0.7
google==2.0.3
google-api-core==1.32.0
google-api-python-client==1.12.11
google-apitools==0.5.31
google-auth==1.35.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-aiplatform==1.17.1
google-cloud-bigquery==2.34.4
google-cloud-bigquery-storage==2.13.2
google-cloud-bigtable==1.7.3
google-cloud-core==2.3.2
google-cloud-datastore==1.15.5
google-cloud-dlp==3.9.2
google-cloud-firestore==2.7.2
google-cloud-language==1.3.2
google-cloud-pubsub==2.13.11
google-cloud-pubsublite==1.6.0
google-cloud-recommendations-ai==0.7.1
google-cloud-resource-manager==1.6.3
google-cloud-spanner==3.26.0
google-cloud-storage==2.5.0
google-cloud-translate==3.8.4
google-cloud-videointelligence==1.16.3
google-cloud-vision==1.0.2
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.4.0
googleapis-common-protos==1.57.0
googledrivedownloader==0.4
graphviz==0.10.1
greenlet==2.0.1
grpc-google-iam-v1==0.12.4
grpcio==1.51.1
grpcio-status==1.48.2
gspread==3.4.2
gspread-dataframe==3.0.8
gym==0.25.2
gym-notices==0.0.8
h5py==3.1.0
hdfs==2.7.0
HeapDict==1.0.1
hijri-converter==2.2.4
holidays==0.17.2
holoviews==1.14.9
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.17.4
httpstan==4.6.1
humanize==0.5.1
hyperopt==0.1.2
idna==2.10
imageio==2.9.0
imagesize==1.4.1
imbalanced-learn==0.8.1
imblearn==0.0
imgaug==0.4.0
importlib-metadata==5.1.0
importlib-resources==5.10.1
imutils==0.5.4
inflect==2.1.0
intel-openmp==2022.2.1
intervaltree==2.1.0
ipykernel==5.3.4
ipython==7.9.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.7.1
itsdangerous==1.1.0
jax==0.3.25
jaxlib @ https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.3.25+cuda11.cudnn805-cp38-cp38-manylinux2014_x86_64.whl
jedi==0.18.2
jieba==0.42.1
Jinja2==2.11.3
joblib==1.2.0
jpeg4py==0.1.4
jsonschema==4.3.3
jupyter-client==6.1.12
jupyter-console==6.1.0
jupyter_core==5.1.0
jupyterlab-widgets==3.0.4
kaggle==1.5.12
kapre==0.3.7
keras==2.11.0
Keras-Preprocessing==1.1.2
keras-tuner==1.1.3
keras-vis==0.4.1
kiwisolver==1.4.4
korean-lunar-calendar==0.3.1
kt-legacy==1.0.4
kubernetes==12.0.1
langcodes==3.3.0
libclang==14.0.6
librosa==0.8.1
lightgbm==2.2.3
llvmlite==0.39.1
lmdb==0.99
locket==1.0.0
logical-unification==0.4.5
LunarCalendar==0.0.9
lxml==4.9.2
Markdown==3.4.1
MarkupSafe==2.0.1
marshmallow==3.19.0
matplotlib==3.2.2
matplotlib-venn==0.11.7
miniKanren==1.0.3
missingno==0.5.1
mistune==0.8.4
mizani==0.7.3
mkl==2019.0
ml-metadata==1.12.0
ml-pipelines-sdk==1.12.0
mlxtend==0.14.0
more-itertools==9.0.0
moviepy==0.2.3.5
mpmath==1.2.1
msgpack==1.0.4
multidict==6.0.3
multipledispatch==0.6.0
multitasking==0.0.11
murmurhash==1.0.9
music21==5.5.0
natsort==5.5.0
nbconvert==5.6.1
nbformat==5.7.0
netCDF4==1.6.2
networkx==2.8.8
nibabel==3.0.2
nltk==3.7
notebook==5.7.16
numba==0.56.4
numexpr==2.8.4
numpy==1.21.6
oauth2client==4.1.3
oauthlib==3.2.2
objsize==0.5.2
okgrade==0.4.3
opencv-contrib-python==4.6.0.66
opencv-python==4.6.0.66
opencv-python-headless==4.6.0.66
openpyxl==3.0.10
opt-einsum==3.3.0
orjson==3.8.3
osqp==0.6.2.post0
overrides==6.5.0
packaging==20.9
palettable==3.3.0
pandas==1.3.5
pandas-datareader==0.9.0
pandas-gbq==0.17.9
pandas-profiling==1.4.1
pandocfilters==1.5.0
panel==0.12.1
param==1.12.3
parso==0.8.3
partd==1.3.0
pastel==0.2.1
pathlib==1.0.1
pathy==0.10.1
patsy==0.5.3
pep517==0.13.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.1.2
pip-tools==6.2.0
platformdirs==2.6.0
plotly==5.5.0
plotnine==0.8.0
pluggy==0.7.1
pooch==1.6.0
portpicker==1.3.9
prefetch-generator==1.0.3
preshed==3.0.8
prettytable==3.5.0
progressbar2==3.38.0
prometheus-client==0.15.0
promise==2.3
prompt-toolkit==2.0.10
prophet==1.1.1
proto-plus==1.22.1
protobuf==3.19.6
psutil==5.4.8
psycopg2==2.9.5
ptyprocess==0.7.0
py==1.11.0
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.6
pycparser==2.21
pyct==0.4.8
pydantic==1.10.2
pydata-google-auth==1.4.0
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyerfa==2.0.0.1
pyfarmhash==0.3.2
Pygments==2.6.1
PyGObject==3.26.1
pylev==1.4.0
pymc==4.1.4
PyMeeus==0.5.12
pymongo==3.13.0
pymystem3==0.2.0
PyOpenGL==3.1.6
pyparsing==3.0.9
pyrsistent==0.19.2
pysimdjson==3.2.0
pysndfile==1.3.8
PySocks==1.7.1
pystan==3.3.0
pytest==3.6.4
python-apt==0.0.0
python-dateutil==2.8.2
python-louvain==0.16
python-slugify==7.0.0
python-utils==3.4.5
pytz==2022.6
pyviz-comms==2.2.1
PyWavelets==1.4.1
PyYAML==5.4.1
pyzmq==23.2.1
qdldl==0.1.5.post2
qudida==0.0.4
regex==2022.6.2
requests==2.28.1
requests-oauthlib==1.3.1
resampy==0.4.2
rpy2==3.5.5
rsa==4.9
scikit-image==0.18.3
scikit-learn==1.0.2
scipy==1.7.3
screen-resolution-extra==0.0.0
scs==3.2.2
seaborn==0.11.2
Send2Trash==1.8.0
setuptools-git==1.2
shapely==2.0.0
six==1.15.0
sklearn-pandas==1.8.0
smart-open==6.3.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soundfile==0.11.0
spacy==3.4.4
spacy-legacy==3.0.10
spacy-loggers==1.0.4
Sphinx==1.8.6
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib-websupport==1.2.4
SQLAlchemy==1.4.45
sqlparse==0.4.3
srsly==2.4.5
statsmodels==0.12.2
sympy==1.7.1
tables==3.7.0
tabulate==0.8.10
tblib==1.7.0
tenacity==8.1.0
tensorboard==2.11.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.11.0
tensorflow-data-validation==1.12.0
tensorflow-datasets==4.6.0
tensorflow-estimator==2.11.0
tensorflow-gcs-config==2.9.1
tensorflow-hub==0.12.0
tensorflow-io-gcs-filesystem==0.28.0
tensorflow-metadata==1.12.0
tensorflow-model-analysis==0.43.0
tensorflow-probability==0.17.0
tensorflow-serving-api==2.11.0
tensorflow-transform==1.12.0
termcolor==2.1.1
terminado==0.13.3
testpath==0.6.0
text-unidecode==1.3
textblob==0.15.3
tfx==1.12.0
tfx-bsl==1.12.0
thinc==8.1.5
threadpoolctl==3.1.0
tifffile==2022.10.10
toml==0.10.2
tomli==2.0.1
toolz==0.12.0
torch @ https://download.pytorch.org/whl/cu116/torch-1.13.0%2Bcu116-cp38-cp38-linux_x86_64.whl
torchaudio @ https://download.pytorch.org/whl/cu116/torchaudio-0.13.0%2Bcu116-cp38-cp38-linux_x86_64.whl
torchsummary==1.5.1
torchtext==0.14.0
torchvision @ https://download.pytorch.org/whl/cu116/torchvision-0.14.0%2Bcu116-cp38-cp38-linux_x86_64.whl
tornado==6.0.4
tqdm==4.64.1
traitlets==5.7.1
tweepy==3.10.0
typeguard==2.7.1
typer==0.7.0
typing_extensions==4.4.0
tzlocal==1.5.1
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.9.0
wasabi==0.10.1
wcwidth==0.2.5
webargs==8.2.0
webencodings==0.5.1
websocket-client==1.4.2
Werkzeug==1.0.1
widgetsnbextension==3.6.1
wordcloud==1.8.2.2
wrapt==1.14.1
xarray==2022.12.0
xarray-einstats==0.4.0
xgboost==0.90
xkit==0.0.0
xlrd==1.2.0
xlwt==1.3.0
yarl==1.8.2
yellowbrick==1.5
zict==2.2.0
zipp==3.11.0
zstandard==0.19.0

[gaikwadrahul8]

Hi, @michaelwsherman

Apologize for the inconvenience and Yes, currently we're experiencing that issue with orchestration and our team is making some code level changes so once it's done issue will be resolved and I'll update the same here once I got the confirmation from our concerned team about issue resolution

Thank you!

[gaikwadrahul8]

Hi, @michaelwsherman

I was able to replicate the issue without error and for that you'll have to uninstall shapely by adding this line of code !pip uninstall shapely -y and even it's updated in google colab nootbook tutorials also, for your reference I have added gist files penguin_simple and penguin_tfdv

Could you please try and confirm, Is it working both the tutorials as expected from your end ?

Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved ?

Thank you!

[michaelwsherman]

Can confirm both gists work, restarting where instructed to.

Note this applies to more of the TFX tutorials than these two, I just stopped checking after these two. I'd suggest fixing the others before resolving this issue.

[gaikwadrahul8]

Hi, @michaelwsherman

I checked all the TFX tutorials under the Tutorials section and they're working as expected after uninstalling shapely package and also it's updated in all those tutorials notebook except one tutorial and yes as mentioned in tutorials section Did you restart the runtime?, you've to restart runtime to work everything as expected.

@lego0901, I found one tutorial where new changes are not reflected so could you please look into this tutorial where uninstall shapely section is missing ?

Thank you!

[lego0901]

Thanks for catching the missing notebook!

I think PR you made will solve the issue correctly! And I guess we should wait for the TensorFlow folks to response.

I appreciate your consideration!

[gaikwadrahul8]

Hi, @lego0901

You're welcome and yes, I have submitted PR for that so we'll wait for response from TensorFlow folks.

Thank you!