Quick Start Guide: Installing Conduit on a VM from any cloud provider or on-premise Linux server

For support, contact us at support@bpcs.com to schedule time for one of our architects to assist.

TABLE OF CONTENTS

Prerequisites

You have a VM/Linux server with the following minimum requirements:

1. 8 cores

2. 64 GB of disk size

3. 32 GB of RAM 

4. Ubuntu Linux 18.04 LTS (only)

5. Authorize inbound traffic for ports 80, 443 and 10002. You can limit access to the IP's that will access Conduit.

6. FQDN for the VM instance (URL of the VM instance)

7. Download the Conduit installation scripts (see Latest Release Notes)

8. (Optional) Prepare credentials for cloud storage to configure Conduit cache storage location. 
   See more details on Configure parquet store file system and Configure your backup policy.

Conduit Engine configuration

Conduit uses Spark as a query engine for some of the processing that cannot be handled natively. By default, the installation script (see next section) uses the following default settings:

• spark_executor_cores = 8

• spark_executor_memory = 22G

• spark_worker_cores = 8

• spark_worker_memory = 23G

• conduit_server_memory_opts = '-Xms4G -Xmx8G'

The above settings can be overriden upon installation by specifying them when running the Conduit installation script.

Installation

1. Upload the conduit-<version>.tar.gz file and shell script conduit-transformless-install-onprem.sh to a folder on the VM

2. With the script files loaded onto the VM, ensure that the deployment script (conduit-transformless-install-onprem.sh file) is an executable (if not: $ chmod +x <filename>)

3. Run the deployment script as root ./ conduit-transformless-install-onprem.sh 

   sudo ./conduit-transformless-install-onprem.sh

4. When prompted, enter the fully qualified domain name (FQDN) for the VM instance. Example: ec2-74-728-49-824.us-west-2.compute.amazonaws.com or mydomain.conduit.com

5. When prompted to enable SSL support and generate LetsEncrypt certificates, type "true" to have secure https communication enabled.

6. When prompted, enter a name, email and password for the initial Conduit administrator account.

7. Enter the type of file system used by Conduit for its data source cache (Parquet Store) (file, hdfs, abfs, s3, gcs). Read more at Configure parquet store file system.

8. Once the script completes, open a web browser and navigate to the FQDN entered in Step 4. The FQDN entered in the browser should begin with https://
   (note that in some cases automatic SSL certificate creation can fail, especially for EC2 instances with the generated cloud FQDN, in this case the deployment script falls back to http://. A new SSL certificate can later be configured, contact support in this case)

9. Login with the administrator credentials you entered in Step 6 for the administrator account.

Post Installation configuration

After installation, there are a few steps that are needed to ensure a complete experience for all Conduit users.

Please navigate to Post Installation Configuration and follow the steps.

If you are installing behind a firewall, please ensure that Conduit ports are whitelisted. 

License Configuration

A valid license key, issued by Blueprint, is required in order to utilize Conduit.

1. In Conduit, go to the Settings (click the Admin Username at top right of screen and you will find Settings)

2. Copy the generated License Token from the License Settings tab

3. Email the license token to support@bpcs.com. You will receive your license key/public key pair that you will have to paste in.

Ready to Build!

Now you are ready to start creating connectors! Check out the Connectors page for guidance on how to begin.

You can also begin to configure your authentication methods and user groups. Managing Users and Security will help get you there.

Related articles

• Post Installation Configuration

• Installing from Azure Marketplace

• REST Endpoints

• Security Management

• Partitioning Your Data

• Configure parquet store file system