Tensor Initialization Issue

[sethclim]

Looking for help understanding this internal tensorizer error.
The trace runs through the whole model fine, but fails during HLO compilation.

...root = neuronxcc/starfish/penguin/targets/transforms/TensorInitialization.py
root = neuronxcc/starfish/penguin/targets/transforms
root = neuronxcc/starfish/penguin/targets
root = neuronxcc/starfish/penguin
root = neuronxcc/starfish
[TEN404] Internal tensorizer error: TensorInitialization: - Please open a support ticket at https://github.com/aws-neuron/aws-neuron-sdk/issues/new. You may also be able to obtain more information using the 'XLA_IR_DEBUG' and 'XLA_HLO_DEBUG' environment variables.
Traceback (most recent call last):
File "/OpenVoice/trace_neuron.py", line 151, in
model_neuron = torch_neuronx.trace(model.voice_conversion, (spec, spec_lengths, xla_src_se, xla_tar_se, g_zeros_src,g_zeros_tgt))
File "/aws_neuron_venv_pytorch/lib64/python3.9/site-packages/torch_neuronx/xla_impl/trace.py", line 628, in trace
neff_filename, metaneff, flattener, packer, weights = _trace(
File "/aws_neuron_venv_pytorch/lib64/python3.9/site-packages/torch_neuronx/xla_impl/trace.py", line 703, in _trace
neff_artifacts = generate_neff(
File "/aws_neuron_venv_pytorch/lib64/python3.9/site-packages/torch_neuronx/xla_impl/trace.py", line 540, in generate_neff
neff_filename = hlo_compile(
File "/aws_neuron_venv_pytorch/lib64/python3.9/site-packages/torch_neuronx/xla_impl/trace.py", line 359, in hlo_compile
raise RuntimeError(f"neuronx-cc failed with {status}")
RuntimeError: neuronx-cc failed with 70

[delongmeng-aws]

Hi @sethclim , Thank you for filing the issue. We will take a look and get back to you.
In the meantime, could you provide steps to reproduce this issue? Ideally, we would like to get a HLO file to investigate compilation errors.

[sethclim]

Hi @delongmeng-aws,

Here's the repo where I was working on the trace OpenVoice

I made a readme with the setup here: TRACE_INSTALLATION.md
Script to trace here: trace_neuron.py

Here's the log and graph.hlo file:
log-neuron-cc.txt
graph.hlo

Let me know if you need anything else

Thanks so much!

[chongmni-aws]

Hi @sethclim, We are currently reviewing this with our compiler team and will provide an update soon.