You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
For a case of loading the same data files from remote data source by multiple times, it'll be significantly improved if there's a local file caching mechanism.
Describe the solution you'd like Alluxio is an open source project, which can do the exact caching thing for this case. We'd like to use Alluxio as the file caching service working with our plugin to provide a solution for the case of frequent remote reading.
We'd like to optimize the file reading and partitioning in our plugin according to the core of Alluxio.
Tasks:
Productionize plugin to work with different filesystems(AWS, Azure, GCP, DBFS)
Create tests for Alluxio with different filesystems
Create a user guide for Alluxio settings
Verify Alluxio can run in an on-prem cluster
The text was updated successfully, but these errors were encountered:
For now, we have merged PR #1562 for the V1 data source. And we will discuss if we need to add alluxio support for the V2 data source. So for now, just close the issue. I will re-open an another issue if we have plans to support V2 data source.
Is your feature request related to a problem? Please describe.
For a case of loading the same data files from remote data source by multiple times, it'll be significantly improved if there's a local file caching mechanism.
Describe the solution you'd like
Alluxio is an open source project, which can do the exact caching thing for this case. We'd like to use Alluxio as the file caching service working with our plugin to provide a solution for the case of frequent remote reading.
We'd like to optimize the file reading and partitioning in our plugin according to the core of Alluxio.
Tasks:
The text was updated successfully, but these errors were encountered: