Awesome library!

This is probably a generic duckdb question but figured I'd ask here first. If I store a reference to a dataset in a class attribute, how do I get query to find my dataset?

Repro:

class DatasetQuery:
    
    def __init__(self, dataset_name, split="train"):
        ds = datasets.load_dataset(dataset_name, split="train")
        self.dataset = ds
    
    def query(self, query_str):
        return query(query_str)

dq = DatasetQuery("huggingnft/boredapeyachtclub")
dq.query("select * from ?? limit 10;")

What do I put in the from clause? I tried ds and self.dataset but neither work. I get ValueError: The datasetdsnot found in the namespace.

I tried running an example from the repo but the code is broken:

imdb_dset = load_dataset("imdb", split="train")
dataset = query(
    "SELECT text FROM imdb_dset"
)

results in AttributeError: 'duckdb.DuckDBPyConnection' object has no attribute 'fetch_arrow_chunk'

I am using datasets_sql version 0.1.1 and datasets version 2.5.2

I'd like to query a large remote dataset (on the hub or elsewhere) and then stream the results of the query so that I don't have to download the entire dataset to my machine.

For example, you could query diffusiondb for images generated with prompts containing the word "ceo" to visualize biases:

SELECT * from poloclub/diffusiondb
WHERE contains('prompt', 'ceo')

This combined with https://github.com/huggingface/datasets-server/issues/398 would open the door for a lot of cool applications of gradio + datasets where users could interactively explore datasets that don't fit on their machines and create spaces without having to download/store large datasets.

I see that data can be streamed from duckdb with pyarrow: https://duckdb.org/2021/12/03/duck-arrow.html . I wonder if this can be leveraged for this use case.