--dynamic-space-size handling on SBCL is broken

[rpgoldman]

I was having trouble with an application I built with this command line:

buildapp --dynamic-space-size 100000 --output strike-force-one --load game-player.lisp --entry 'cl-user::main' --dumpfile-copy 'strike-force-one-dumpfile.lisp' --logfile 'build-game-player.log'

When I ran the resulting application, it ran out of memory quickly. But when I ran sbcl with --dynamic-space-size 100000 and passed it a script that did the equivalent, it did not run out of memory.

Tracking it down, I found that run-program for SBCL, invoked in main was not getting the --dynamic-space-size 100000 argument:

(RUN-PROGRAM "sbcl"
             '("--noinform" "--disable-debugger" "--no-userinit" "--no-sysinit"
               "--load"
               "/home/rpg/projects/scepter/shop-mcts/strike-force-one/dumper-2SKVI5f7.lisp"))

Further debugging shows that buildapp does not see the --dynamic-space-size argument:

Constructing a dumper from (--output strike-force-one --load game-player.lisp
                            --entry cl-user::main --dumpfile-copy
                            strike-force-one-dumpfile.lisp --logfile
                            build-game-player.log)
After processing the command line arguments, we have the following dumper object:
#<BUILDAPP::DUMPER strike-force-one cl-user::main>
  [standard-object]

Slots with :INSTANCE allocation:
  PACKAGE                        = #:DUMPER124
  ACTIONS                        = ((:LOAD "game-player.lisp"))
  ENTRY                          = cl-user::main
  DISPATCHED-ENTRIES             = NIL
  ASDF-DIRECTIVES                = NIL
  LOAD-PATHS                     = NIL
  SBCL                           = "sbcl"
  CCL                            = "ccl"
  OUTPUT                         = "strike-force-one"
  LOGFILE                        = "build-game-player.log"
  DUMPFILE-COPY                  = "strike-force-one-dumpfile.lisp"
  CORE-ONLY                      = NIL
  COMPRESS-CORE                  = NIL
  DYNAMIC-SPACE-SIZE             = NIL

The "constructing a dumper" message above formats the arguments that are passed to command-line-dumper.

I conjecture that SBCL is stripping out this argument before the buildapp executable can see it.

I will attempt to verify this.

[rpgoldman]

OK, I have verified the problem and here's what I did:

I replaced the command line parsing clause as follows:

          (:space-size ;; :dynamic-space-size
           (format t "Trying to parse dynamic-space-size from \"~a\" giving ~d~%"
                   value (parse-integer value))
           (setf (dynamic-space-size plan) (parse-integer value)))

and changed my command line to

buildapp --space-size 100000 --output strike-force-one --load game-player.lisp --entry 'cl-user::main' --dumpfile-copy 'strike-force-one-dumpfile.lisp' --logfile 'build-game-player.log'

and that gets the argument passed through:

After processing the command line arguments, we have the following dumper object:
#<BUILDAPP::DUMPER strike-force-one cl-user::main>
  [standard-object]

Slots with :INSTANCE allocation:
  PACKAGE                        = #:DUMPER124
  ACTIONS                        = ((:LOAD "game-player.lisp"))
  ENTRY                          = cl-user::main
  DISPATCHED-ENTRIES             = NIL
  ASDF-DIRECTIVES                = NIL
  LOAD-PATHS                     = NIL
  SBCL                           = "sbcl"
  CCL                            = "ccl"
  OUTPUT                         = "strike-force-one"
  LOGFILE                        = "build-game-player.log"
  DUMPFILE-COPY                  = "strike-force-one-dumpfile.lisp"
  CORE-ONLY                      = NIL
  COMPRESS-CORE                  = NIL
  DYNAMIC-SPACE-SIZE             = 100000
About to recursively start SBCL, dynamic-space-size is 100000
running this command to try to build the application:

(RUN-PROGRAM "sbcl"
             '("--dynamic-space-size" "100000" "--noinform"
               "--disable-debugger" "--no-userinit" "--no-sysinit" "--load"
               "/home/rpg/projects/scepter/shop-mcts/strike-force-one/dumper-2SKVI5f7.lisp"))

I'm not sure how to fix this problem. Renaming the argument as I did is inelegant but works. It would be nicer if we could keep SBCL from snarfing up that argument.

I suppose it's possible that ensuring that argument appears at the end would cause SBCL to not recognize it as a toplevel argument and pass it through. But that seems like a very brittle fix.

[PaulLockett]

Would love to see a resolution on this is causing an issue for dependents like pgloader. see here dimitri/pgloader#962

[rpgoldman]

@PaulLockett -- I have made a pull request for my proposed fix, which you could use for the present (you could just clone my repo instead of this one).

[PaulLockett]

@PaulLockett -- I have made a pull request for my proposed fix, which you could use for the present (you could just clone my repo instead of this one).

@rpgoldman hopefully your solution gets merged.

I ended up just building SBCL locally from source and setting the default to a higher value.