Skip to content

Fix issue #50: SASLib.open hangs for large files - #84

Merged
tk3369 merged 1 commit into
masterfrom
fix/issue-50-open-hangs-large-files
May 30, 2026
Merged

Fix issue #50: SASLib.open hangs for large files#84
tk3369 merged 1 commit into
masterfrom
fix/issue-50-open-hangs-large-files

Conversation

@tk3369

@tk3369 tk3369 commented May 25, 2026

Copy link
Copy Markdown
Owner

Problem

SASLib.open and SASLib.read both fail on large SAS7BDAT files (e.g. a 2 GB file with 4,000+ columns). Fixes #50.

Confirmed behaviour on master against the Census AHS 2013 public-use file (2.0 GB, 70,044 rows x 4,041 columns):

  • open() completes but is ~2x slower than necessary (1.476s vs 0.785s)
  • read() crashes with BoundsError: attempt to access 1355-element Vector{UInt8} at index [1356] in populate_column_indices

Root cause

Two bugs in the metadata-reading path:

1. read_file_metadata ignored _process_page_meta's return value

_process_page_meta returns true when it has finished reading metadata (i.e. it has reached the first data page). The loop was discarding that signal and continuing to scan every page in the file — for a wide file this means reading through thousands of data pages unnecessarily.

2. _process_columnattributes_subheader used replace semantics instead of accumulate

The function called empty! before append!, so for files whose column attributes span multiple metadata pages (wide files with 4,000+ columns), only the last page's chunk of attributes survived. Meanwhile column_symbols accumulated all column names across pages, producing a length mismatch that caused the BoundsError in populate_column_indices.

A secondary issue: the guard condition used length(handler.column_types) where length is also a parameter name, causing a MethodError: objects of type Int64 are not callable. Fixed by qualifying as Base.length.

Fix

  • read_file_metadata: break out of the loop when _process_page_meta returns true
  • _process_columnattributes_subheader: switch to append!-only (accumulate across pages); add a Base.length guard to skip re-processing once all column attributes are loaded

Testing

  • All 237 existing tests pass
  • Manual regression against the Census AHS 2013 public-use file (2.0 GB, 70,044 rows x 4,041 columns):
master this PR
open() 1.476s 0.785s
read(1000) BoundsError 0.839s
read() full file BoundsError 12.04s

A new manual regression script test/test_large_sas.jl is included with download instructions for the test file. It is intentionally excluded from the standard Pkg.test() suite since the file must be downloaded separately.

Two bugs caused open() to scan the entire file when reading metadata
from large SAS7BDAT files (e.g. 2 GB, 4000+ columns):

1. read_file_metadata ignored the return value of _process_page_meta,
   so the metadata loop never stopped at the first data page. Fixed by
   breaking out of the loop when _process_page_meta returns true.

2. _process_columnattributes_subheader used empty!+append! (replace
   semantics) instead of accumulate semantics. For wide files whose
   column attributes span multiple metadata pages, only the last page's
   attributes survived while column_symbols accumulated everything,
   causing a BoundsError in populate_column_indices. Fixed by switching
   to append!-only with a guard (using Base.length to avoid shadowing
   by the length parameter) that skips re-processing once all column
   attributes are loaded.

Also adds test/test_large_sas.jl, a manual regression test against the
Census AHS 2013 public-use file (~2 GB). Includes download instructions
and is intentionally excluded from the standard Pkg.test() suite.
@tk3369
tk3369 merged commit a79c58c into master May 30, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SASLib.open hangs for large files

1 participant