fix(caching): avoid redundant computations during caching

[GoshaSerbin]

This PR fixes an inefficiency in the caching implementation caused by residuals not being utilized during cached inference.

Each model layer maintains its own cache and predicts features using the Taylor approximation, incurring both additional memory usage (storing previous timesteps) and extra computation. However, since residuals are not utilized during caching, the output of previous layers does not influence subsequent layers. As a result, maintaining caches for intermediate layers provides no benefit.

This change removes the unnecessary caching overhead for all layers except the final layer while preserving identical outputs.

Performance

Profiling with max_order=6:

Metric	Before	After
Reserved memory	45.23 GB	34.24 GB
Allocated memory	38.85 GB	31.16 GB
T2I total time	16.8 s	14.6 s

Results

• 28× reduction in caching memory overhead (equal to the number of model layers).
• ~15% faster inference.
• Memory savings scale proportionally with the Taylor approximation order.
• Outputs are bit-identical to the previous implementation.

Implementation

The optimization skips cache maintenance for all intermediate layers and keeps it only for the final layer. This is possible because intermediate layer caches are never utilized during cached inference.