Complex case leads to free before allocation

[scolsen]

I've run into a case where a String is freed before it's actually used, here's the code:

(defn binary-partial [f x]
      (fn [y]
        (f x y)))
(doc lift-binary
  "Lifts a binary function into an applicative structure.")
    (defn lift-binary [f cx cy]
      (sequence (fmap &(binary-partial binary-partial f) cx) cy))

(deftype (Box a) [of a])

(defmodule Box
  (implements fmap fmap)
  (defn fmap [f b]
    (let [val (~f @(Box.of &b))]
    (Box.init val)))

  (implements fmap! fmap!)
  (defn fmap! [f b]
    (let [val (~f @(Box.of b))]
    (Box.set-of! b val)))

  (implements lift Box.init)

  (implements sequence seq)
  (defn seq [box-f box]
    (let [f @(Box.of &box-f)
          v @(Box.of &box)]
      (Box.init (f v))))

  (implements sequence! seq!)
  (defn seq! [box-f box]
    (let [f @(Box.of box-f)
          v @(Box.of box)]
      (Box.set-of! box (f v))))

)

Loading up this code and then running:

鲤 (defn foo [] (Applicative.lift-binary (fn [x y] (String.append &x &y)) (Box.init @"foo") (Box.init @"bar")))
鲤 (foo)
Compiled to 'out/Untitled' (executable)
Untitled(3776,0x10c0a7dc0) malloc: *** error for object 0x7fc502c05830: pointer being freed was not allocated
Untitled(3776,0x10c0a7dc0) malloc: *** set a breakpoint in malloc_error_break to debug
[RUNTIME ERROR] '"out/Untitled"' exited with return value -6.
鲤 

It emits a free before allocation here:

void FP__Lambda_binary_MINUS_partial__String_String_String_12_env_delete(FP__Lambda_binary_MINUS_partial__String_String_String_12_env* p) {
    Function_delete__String_String_String(p->f);
    String_delete(p->x);
}

commenting out String_delete(p->x); makes it run fine:

./foo
(Box @"foobar")

I thought this might have to do with capturing vars, but it perhaps doesn't since this works fine:

鲤 (defn foo [] (Applicative.lift-binary (fn [x y] (+ x y)) (Box.init 1) (Box.init 2)))
鲤 (foo)
Compiled to 'out/Untitled' (executable)
(Box 3)
=> 0

Seems specifically related to Strings or String.appendas well, not references in general:

鲤 (defn foo [] (Applicative.lift-binary (fn [x y] (add-ref &x &y)) (Box.init 1) (Box.init 2)))
鲤 (foo)
Compiled to 'out/Untitled' (executable)
(Box 3)
=> 0
鲤 

[scolsen]

Ah, the key is managed v. non-managed members:

void FP__Lambda_binary_MINUS_partial__int_int_int_12_env_delete(FP__Lambda_binary_MINUS_partial__int_int_int_12_env* p) {
    Function_delete__int_int_int(p->f);
    /* Ignore non-managed member 'x' : Int */
}

[scolsen]

Relevant code is in Concretize.hs lines 94-170, in particular line 147 which generates the deleter

[eriksvedang]

Interesting find! Do you have a fix in mind?

[scolsen]

no great ideas yet--I think we should fix this particular case, but it did give me a more general feature idea that might be interesting. We could add a persist, manual, or leak that turns off management for a particular binding so that one would need to allocate and delete manually and none of the automatic emissions for such things would happen.

Another really interesting way we could support it is by defining a special Mem type that takes a type as an argument and emits no allocations/deallocations unless explicitly called, for example one would have to do something like:

(sig allocate (Fn [a] (Mem a)))
(sig free (Fn [(Mem a)] ()))

(let-do [s (allocate "foo")]
        ...do some things
        (free s))
    

of course, one is just bypassing the borrow checker at that point--it'd be similar to unsafe calls/bypass mechanisms in other languages. But maybe something like this is already possible?

[scolsen]

actually, taking a quick look at which types are managed, something like what I'm describing is potentially already possible with the Ptr type

[scolsen]

I figured out a workaround but it's a bit complicated. Assuming address works in the REPL (I have a branch ready to enable this), one can do the following:

(def a @"")
(def b @"")
(Applicative.lift-binary (fn [x y] (String.append &(Pointer.to-value x) &(Pointer.to-value y))) (fmap &(adr a) (Box.init @"foo")) (fmap &(adr b) (Box.init @"bar")))
Compiled to 'out/Untitled' (executable)
(Box @"foobar")

here's how the adr works:

(defmacro adr [proxy] (list 'fn (array 'x) (list 'do (list 'set! proxy 'x) (list 'address proxy))))

And here's the form it produces:

鲤 (adr a)
=> (fn <> [x] (do (set! a x) (address a)))

The idea here is that we get around the deletion by fmapping the address of a proxy definition that holds the initial value in global scope, ensuring the lambda doesn't reclaim it.

EDIT: So this is a workaround of sorts, but it isn't entirely useful unless we have a form that allows producing global definitions in a local context.

[eriksvedang]

Huh, interesting workaround – but we really should make the original code work (or not typecheck at all).

Could this be a lifetime error? I don't really see (at a glance) where the problematic situation occurs in the original code.

[eriksvedang]

Does this lambda need to capture anything? (is that the error?)

(fn [x y] (String.append &x &y)) (Box.init @"foo") (Box.init @"bar"))

[scolsen]

Yeah, I don't fully understand it myself yet -- I think the error stems from delete being called on a reference to some value after the original value is already deleted

[scolsen]

Basically, I think what happens is that binary-partial binary-partial expands into a nested lambda, which closes over an x--but x is deleted once the outer lambda returns, even though it's needed in the inner lambda that's returned...I think that's roughly what's happening but I'm not certain--or rather, the first deletion isn't actually problematic, but since x is managed in both the outer and inner lambda contexts it results in two calls to delete on the same underlying memory.

[scolsen]

I'm actually able to produce a simpler case using address:

This case is actually probably OK since ensuring Pointer.to-value is safe is up to the programmer, but it's a simpler example of the same underlying problem, I think:

(def a "foo")
鲤 (Pointer.to-value (address @a))
Compiled to 'out/Untitled' (executable)
foo
Untitled(2155,0x10c2cbdc0) malloc: *** error for object 0x7fa7a7c05810: pointer being freed was not allocated
Untitled(2155,0x10c2cbdc0) malloc: *** set a breakpoint in malloc_error_break to debug
[RUNTIME ERROR] '"out/Untitled"' exited with return value -6.
鲤 

Here's the emitted C:

int main(int argc, char** argv) {
    carp_init_globals(argc, argv);
    String _12 = String_copy(a);
    String _14 = Pointer_to_MINUS_value__String(&_12); // _14 is just a dereferenced _12
    String* _15 = &_14; // ref
    String _16 = String_str(_15);
    String* _17 = &_16; // ref
    IO_println(_17);
    int _20 = 0;
    String_delete(_12);
    String_delete(_14); // !!!!! This is the problematic line! _14 is the same as _12, which was freed just above.
    // Commenting out the above line makes the program run fine, since we avoid the double free.
    String_delete(_16);
    return _20;
}

@eriksvedang fyi I think a similar thing to the forced example here happens when working with lambdas that return lambdas that close over some argument from an initial lambda (like binary-partial in this issue)

[scolsen]

So, I think this should be a lifetime error, perhaps? Otherwise I think it'd require us to implement lifetime/value tracking in closures that's quite a bit more complicated than the current impl?