...
On this page:
What is Caching?
In Conduit, a cache is a data storage layer which stores a subset of data, transient in nature, so that future requests for that data are served up faster than is possible by accessing the data’s primary storage location. In addition to data cache Conduit also maintains query results to efficiently reuse previously retrieved results.
On this page:
| Table of Contents | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
Why Caching?
Often, Database administrators are concerned about provided unfettered access to their data sources. There are good reasons for this:
...
Freshness of data is of primary concern and latency is not an issue (data not terribly large, simple query, etc.)
Few users need access to the data and access is a low priority compared to other, more critical data sources
...
Caching options within Conduit
Conduit provides the choice of leveraging caching for the data sources of any of its connectors.
...
When enabled, Conduit will create temporary secure parquet store of all connector's datasets for a quick future access
Recommended to enable for large datasets and/or when expensive queries are expected
Selected datasets for the connector will be cached in the parquet store. All queries for this connector will be ran against the parquet store
Caching expiration is 24 hours by default, and can be customized for each connector's dataset as needed
When connector data is cached, query results will be cached in memory for small/medium results set to further enhance performance. Query Cache will expire with data cache
Conduit SQL Engine will be used to run all queries when Connector Caching is selected List of existing parquet files and their expected expiration times can be accessed on Performance>Parquet Store page
...
How to Leverage Caching in Conduit
For each connector, Conduit allows for caching the datasets included in that connector. On the `Virtualization` step of the Connector Wizard, you will see options 'Enable Query Caching’ and 'Enable Connector Caching’.
...
Finished! - Conduit makes the option to leverage caching as simple as that. A few clicks provide detailed control over how your data is being access and updated through your Conduit connector.
Monitoring the Conduit Cache
Click the Performance tab on the navigation bar the top of the page. Then click the Parquet Store link in the drop-down menu. This takes you to the monitoring screen of the data cache.
...
Cache in progress can be cancelled only by clearing it on the Parquet Store page. If you wish to cancel cache creation and initiate recaching, clear cache in progress on the Parquet Store page, and then re-save connector with 'Cache now' checked.
Related pages
...