[R] Recipe for random sampling

[gshotwell]

It would be great if there were a way to sample from an arrow dataset. I put together this somewhat hacky example, but I bet there's some thing a bit more elegant..

library(arrow)
library(dplyr)
library(nycflights13)

flights <- nycflights13::flights

flights$id <- seq_len(nrow(flights))

for(i in unique(flights$month)) {
  out <- filter(flights, month == i)
  arrow::write_parquet(out, paste0("flight_ds/", i, ".parquet"))
}

ds <- arrow::open_dataset("flight_ds")

sample <- sample(flights$id, 100)

ds %>% 
  filter(id %in% sample) %>% 
  collect()

[thisisnic]

Apologies, closed this by accident!

[thisisnic]

Thanks for opening this issue @gshotwell! I agree that this would be a really useful thing to a) be able to do in arrow and b) have a recipe for in this cookbook.

There's currently no optimised way of doing this in arrow, but I've opened up a request for this to be implemented at the C++ level; once this is done, we can look at implementing this in R.

https://issues.apache.org/jira/browse/ARROW-14254

In the short-term, the snippet below shows how you can achieve this if you have existing files without having to write any new data to them, though I don't think we want to add a recipe for this as apparently it's a bit slow - I'd like to wait until we have a nice way of doing it properly utilising the underlying C++ functionality before adding it to the cookbook.

tf <- tempfile()
dir.create(tf)

# I've added the grouping just so it's written as a partitioned dataset
iris %>% group_by(Species) %>%
  write_dataset(tf)

iris_dataset <- open_dataset(tf)

# sample as many values as needed
rows <- sample(seq_len(nrow(iris_dataset)), 10)

# return the sample
collect(iris_dataset[rows,])

[nealrichardson]

We could wire up the row selection used in [ (which I added last year just to be able to run the vroom benchmarks, not because it was a good idea or done particularly well) to slice_sample() but I'm not sure it's a good idea given the inefficiency of the implementation.

[wjones127]

As a part of apache/arrow#11894, I added an example to the datasets vignette to show how to create a sample from a dataset that doesn't fit into memory. This will be available in Arrow 7.0.0. Added #115 to add that to the cookbook.

[eitsupi]

I think this is a great use case for combining arrow and duckdb.
There is no support for slice_* functions in arrow and duckdb does not yet have the ability to open a dataset.

library(arrow)
library(dplyr)

tf <- tempfile()
dir.create(tf)

nycflights13::flights |>
  group_by(month) |>
  write_dataset(tf)

df <- open_dataset(tf, hive_style = TRUE) |>
  group_by(month) |>
  to_duckdb() |>
  slice_sample(n = 2) |>
  collect()

[thisisnic]

@eitsupi Fancy submitting a PR with that in a recipe?

[eitsupi]

I would like to contribute to improving the documentation, but I don't know where to add that example.
Wouldn't a function help page be better?

[thisisnic]

Oh, sorry, I misread your comment as saying that that code example did work, and not that it didn't work as of yet! I get a segfault when calling to_duckdb() on a dataset, as in your example above - that'd be something that it'd be worth opening an issue on the Arrow JIRA about.

[eitsupi]

Oh, sorry, I misread your comment as saying that that code example did work, and not that it didn't work as of yet! I get a segfault when calling to_duckdb() on a dataset, as in your example above - that'd be something that it'd be worth opening an issue on the Arrow JIRA about.

On my local this process runs fine. And it appears to be included in the test case.
https://github.com/apache/arrow/blob/0330353a93eff799616bf66e0e994236393458f7/r/tests/testthat/test-duckdb.R#L138-L171

[thisisnic]

Interesting, I should probably open a bug report somewhere for something then. If you're proposing a function help page, which page? I don't see anywhere it would make sense to include it in the Arrow R docs, though happy to discuss.

If it was a recipe, the cookbook is currently lacking a datasets chapter; a couple have been drafted, but finding the time to work on it is tricky. This would work as a subsection in there though.

[eitsupi]

If you're proposing a function help page, which page? I don't see anywhere it would make sense to include it in the Arrow R docs

The following example is worth replacing because it is not necessary to use to_duckdb now.
https://github.com/apache/arrow/blob/f9f0e65e5a370bd26f49b12c5e8bb51218cd82a6/r/R/duckdb.R#L48-L52

[thisisnic]

Nicely spotted! It'd be worth submitting a PR for that or opening an issue on the Arrow JIRA then.