Skip to content

Commit 2f1596b

Browse files
dcassanegodcassanego
committed
merging master down
2 parents f26569b + 5b26a10 commit 2f1596b

12 files changed

+336
-173
lines changed

‎README.md

+166-56
Original file line numberDiff line numberDiff line change
@@ -1,74 +1,184 @@
1-
# Database Lab Engine Terraform Module
1+
[[_TOC_]]
22

3-
This [Terraform Module](https://www.terraform.io/docs/language/modules/index.html) is responsible for
4-
deploying the [Database Lab Engine](https://gitlab.com/postgres-ai/database-lab) to cloud hosting
5-
providers.
3+
# How to setup Database Lab using Terraform in AWS
64

7-
Supported Cloud Platforms:
8-
* AWS
5+
This [Terraform Module](https://www.terraform.io/docs/language/modules/index.html) is responsible for deploying the [Database Lab Engine](https://gitlab.com/postgres-ai/database-lab) to cloud hosting providers.
96

10-
## Prerequisites
11-
* [AWS Account](https://aws.amazon.com)
12-
* [Terraform Installed](https://learn.hashicorp.com/tutorials/terraform/install-cli) (min version 1.0.0)
13-
* AWS [Route 53](https://aws.amazon.com/route53/) Hosted Zone (For setting up TLS) for a domain or sub-domain you control.
14-
15-
## Setup
16-
* You must have AWS Access Keys and a default region in your Terraform environment (See section on required IAM Permissions).
17-
* The DLE runs on an EC2 instance which can be accessed using a selected set of SSH keys uploaded to EC2.
18-
Use the Terraform parameter `keypair` to specify which EC2 Keypair to use.
19-
20-
## Usage
21-
22-
### Terraform Module
23-
This Terraform module can be run independently or combined with any other standard Terraform module.
24-
You can learn more about using Terraform and the Terraform CLI [here](https://www.terraform.io/docs/cli/commands/index.html).
25-
26-
Please note that:
27-
28-
* You can override default parameters (defined in `variables.tf`) either with the CLI or by creating a file called `terraform.tfvars` in the module root directory
29-
* This module currently configures the Database Lab Engine (DLE) to use ["logical" provisioning mode](https://postgres.ai/docs/guides/data/rds) (dump/restore). "Physical" provisioning (copying data directory from the source or fetching it from archives) and other DLE configuration options are planned, but not currently supported by this module.
30-
* All variables starting with `postgres_` represent the database connection information for the data (from that database) to be fetched by the Database Lab Engine. That database must be accessible from the instance hosting the DLE.
31-
* You can view the DLE log by running `sudo docker logs --since 1m -f dblab_server` when ssh'ed to the instance. See [here](https://postgres.ai/docs/guides/administration/engine-manage#observe-database-lab-engine-logs) for more information.
7+
Your source PostgreSQL database can be located anywhere, but DLE with other components will be created on an EC2 instance under your AWS account. Currently, only "logical" mode of data retrieval (dump/restore) is supported – the only available method for managed PostgreSQL cloud services such as RDS Postgres, RDS Aurora Postgres, Azure Postgres, or Heroku. "Physical" mode is not yet supported, but it will be in the future. More about various data retrieval options for DLE: https://postgres.ai/docs/guides/data.
328

33-
### Database Lab Engine
34-
Once your EC2 instance is running and the Database Lab Engine (DLE) is deployed, you can learn more about how to use the DLE
35-
on its [documentation site](https://postgres.ai/docs/guides).
9+
## Supported Cloud Platforms:
10+
- AWS
3611

37-
#### Important Note
38-
When the DLE creates new database clones, it makes them available on incremental ports in the 6000 range (e.g. 6000, 6001, ...).
39-
The DLE CLI will also report that the clone is available on a port in the 6000 range. However, please note that these are the
40-
ports when accessing the DLE from `localhost`. This Terraform module deploys [Envoy](https://www.envoyproxy.io/) to handle
41-
SSL termination and port forwarding to connect to DLE generated clones.
42-
43-
Bottom Line: When connecting to clones, add `3000` to the port number reported by the DLE CLI to connect to the clone.
44-
for example, if the CLI reports that a new clone is available at port `6001` connect that clone at port `9001`.
45-
46-
## Required IAM Permissions
47-
To successfully run this Terraform module, the IAM User/Role must have the following permissions:
48-
49-
* Read/Write permissions on EC2
50-
* Read/Write permissions on Route53
51-
* Read/Write permissions on Cloudwatch
12+
## Prerequisites
13+
- [AWS Account](https://aws.amazon.com)
14+
- [Terraform Installed](https://learn.hashicorp.com/tutorials/terraform/install-cli) (minimal version: 1.0.0)
15+
- AWS [Route 53](https://aws.amazon.com/route53/) Hosted Zone (For setting up TLS) for a domain or sub-domain you control
16+
- You must have AWS Access Keys and a default region in your Terraform environment (See section on required IAM Permissions)
17+
- The DLE runs on an EC2 instance which can be accessed using a selected set of SSH keys uploaded to EC2. Use the Terraform parameter `aws_keypair` to specify which EC2 Keypair to use
18+
- Required IAM Permissions: to successfully run this Terraform module, the IAM User/Role must have the following permissions:
19+
* Read/Write permissions on EC2
20+
* Read/Write permissions on Route53
21+
* Read/Write permissions on Cloudwatch
22+
23+
## How to use
24+
- It is recommended to clone this Git repository and adjust for your needs. Below we provide the detailed step-by-step instructions for quick start (see "Quick start") for a PoC setup
25+
- To configure parameters used by Terraform (and the Database Lab Engine itself), you will need to modify `terraform.tfvars` and create a file with secrets (`secret.tfvars`)
26+
- This Terraform module can be run independently or combined with any other standard Terraform module. You can learn more about using Terraform and the Terraform CLI [here](https://www.terraform.io/docs/cli/commands/index.html)
27+
- The variables can be set in multiple ways with the following precedence order (lowest to highest):
28+
- default values in `variables.tf`
29+
- values defined in `terraform.tfvars`
30+
- values passed on the command line
31+
- All variables starting with `postgres_` represent the source database connection information for the data (from that database) to be fetched by the DLE. That database must be accessible from the instance hosting the DLE (that one created by Terraform)
32+
33+
## Quick start
34+
The following steps were tested on Ubuntu 20.04 but supposed to be valid for other Linux distributions without significant modification.
35+
36+
1. SSH to any machine with internet access, it will be used as deployment machine
37+
1. Install Terraform https://learn.hashicorp.com/tutorials/terraform/install-cli. Example for Ubuntu:
38+
```shell
39+
sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl
40+
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
41+
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" # Adjust if you have ARM platform.
42+
sudo apt-get update && sudo apt-get install terraform
43+
# Verify installation.
44+
terraform -help
45+
```
46+
1. Get TF code for Database Lab:
47+
```shell
48+
git clone https://gitlab.com/postgres-ai/database-lab-infrastructure.git
49+
cd database-lab-infrastructure/
50+
```
51+
1. Edit `terraform.tfvars` file. In our example, we will use Heroku demo database as a source:
52+
```config
53+
dle_version_full = "2.4.0"
54+
55+
aws_ami_name = "DBLABserver*"
56+
aws_keypair = "YOUR_AWS_KEYPAIR"
57+
58+
aws_deploy_region = "us-east-1"
59+
aws_deploy_ebs_availability_zone="us-east-1a"
60+
aws_deploy_ec2_instance_type = "t2.large"
61+
aws_deploy_ec2_instance_tag_name = "DBLABserver-ec2instance"
62+
aws_deploy_ebs_size="40"
63+
aws_deploy_ebs_type="gp2"
64+
aws_deploy_allow_ssh_from_cidrs = ["0.0.0.0/0"]
65+
aws_deploy_dns_api_subdomain="tf-test" # subdomain in aws.postgres.ai, fqdn will be ${dns_api_subdomain}-engine.aws.postgres
66+
67+
source_postgres_version="13"
68+
source_postgres_host="ec2-3-215-57-87.compute-1.amazonaws.com"
69+
source_postgres_port="5432"
70+
source_postgres_dbname="d3dljqkrnopdvg"
71+
source_postgres_username="postgres"
72+
73+
dle_debug_mode="true"
74+
dle_retrieval_refresh_timetable="0 0 * * 0"
75+
postgres_config_shared_preload_libraries="pg_stat_statements"
76+
77+
platform_project_name="aws_test_tf"
78+
```
79+
1. Create `secret.tfvars` containing `source_postgres_password`, `platform_access_token`, and `vcs_github_secret_token`. An example:
80+
```config
81+
source_postgres_password="YOUR_DB_PASSWORD" # todo: put pwd for heroku example DB here
82+
platform_access_token="YOUR_ACCESS_TOKEN" # to generate, open https://console.postgres.ai/, choose your organization,
83+
# then "Access tokens" in the left menu
84+
vcs_github_secret_token="vcs_secret_token" # to generate, open https://github.com/settings/tokens/new
85+
```
86+
1. Initialize
87+
```shell
88+
terraform init
89+
```
90+
1. Set environment variables with AWS credentials:
91+
```shell
92+
export AWS_ACCESS_KEY_ID="keyid" # todo: how to get it?
93+
export AWS_SECRET_ACCESS_KEY="accesskey"
94+
```
95+
1. Deploy:
96+
```
97+
terraform apply -var-file="secret.tfvars" -auto-approve
98+
```
99+
1. If everything goes well, you should get an output like this:
100+
```config
101+
vcs_db_migration_checker_verification_token = "gsio7KmgaxECfJ80kUx2tUeIf4kEXZex"
102+
dle_verification_token = "zXPodd13LyQaKgVXGmSCeB8TUtnGNnIa"
103+
ec2_public_dns = "ec2-18-118-126-25.us-east-2.compute.amazonaws.com"
104+
ec2instance = "i-0b07738148950af25"
105+
ip = "18.118.126.25"
106+
platform_joe_signing_secret = "lG23qZbUh2kq0ULIBfW6TRwKzqGZu1aP"
107+
public_dns_name = "demo-api-engine.aws.postgres.ai" # todo: this should be URL, not hostname – further we'll need URL, with protocol – `https://`
108+
```
109+
110+
1. To verify result and check the progress, you might want to connect to the just-created EC2 machine using IP address or hostname from the terraform output. In our example, it can be done using this one-liner (you can find more about DLE logs and configuration on this page: https://postgres.ai/docs/guides/administration/engine-manage):
111+
```shell
112+
echo "sudo docker logs dblab_server -f" | ssh ubuntu@18.118.126.25 -i postgres_ext_test.pem
113+
```
114+
115+
Once you see the message like:
116+
```
117+
2021/07/02 10:28:51 [INFO] Server started listening on :2345.
118+
```
119+
– it means that the DLE server started successfully and is waiting for you commands
120+
121+
1. Sign in to the [Postgres.ai Platform](https://console.postgres.ai/) and register your new DLE server:
122+
1. Go to `Database Lab > Instances` in the left menu
123+
1. Press the "Add instance" button
124+
1. `Project` – specify any name (this is how your DLE server will be named in the platform)
125+
1. `Verification token` – use the token generated above (`verification_token` value); do NOT press the "Generate" button here
126+
1. `URL` – use the value generated above // todo: not convenient, we need URL but reported was only hostname
127+
1. Press the "Verify URL" button to check the connectivity. Then press "Add". If everything is right, you should see the DLE page with green "OK" status:
128+
<img src="/uploads/8371e7f79de199aa017ff2df82b8f704/image.png" width="400" />
129+
1. Add Joe chatbot for efficient SQL optimization workflow:
130+
1. Go to the "SQL Optimization > Ask Joe" page using the left menu, click the "Add instance" button, specify the same project as you defined in the previous step
131+
1. `Signing secret` – use `platform_joe_signing_secret` from the Terraform output
132+
1. `URL` – use `public_dns_name` values from the Terraform output with port `444`; in our example, it's `https://demo-api-engine.aws.postgres.ai:444`
133+
1. Press "Verify URL" to check connectivity and then press "Add". You should see:
134+
<img src="/uploads/252e5f74cd324fc4df301bbf7c2bdd25/image.png" width="400" />
135+
136+
Now you can start using Joe chatbot for SQL execution plans troubleshooting and verification of optimization ideas. As a quick test, go to `SQL Optimization > Ask Joe` in the left menu, and enter `\dt+` command (a psql command to show the list of tables with sizes). You should see how Joe created a thin clone behind the scenes and immediately ran this psql command, presenting the result to you:
137+
<img src="/uploads/d9e9e1fdafb0ded3504691cec9018868/image.png" width="400" />
138+
139+
1. TODO: CI checker !!!!!!
140+
141+
1. Install and try the client CLI (`dblab`)
142+
1. TODO: install CLI !!!!!
143+
1. Initialize CLI:
144+
```shell
145+
dblab init --environment-id=<ANY NAME FOR ENVIRONMENT> --url=https://<public_dns_name> --token=<your_personal_token_from_postgres_ai_platform>
146+
```
147+
1. Try it:
148+
```shell
149+
dblab instance status
150+
```
151+
It should return the OK status:
152+
```json
153+
{
154+
"status": {
155+
"code": "OK",
156+
"message": "Instance is ready"
157+
},
158+
...
159+
}
160+
```
161+
162+
## Important Note
163+
When the DLE creates new database clones, it makes them available on incremental ports in the 6000 range (e.g. 6000, 6001, ...). The DLE CLI will also report that the clone is available on a port in the 6000 range. However, please note that these are the ports when accessing the DLE from `localhost`. This Terraform module deploys [Envoy](https://www.envoyproxy.io/) to handle SSL termination and port forwarding to connect to DLE generated clones.
164+
165+
Bottom Line: When connecting to clones, add `3000` to the port number reported by the DLE CLI to connect to the clone. for example, if the CLI reports that a new clone is available at port `6001` connect that clone at port `9001`.
52166
53167
## Known Issues
54-
#### Certificate Authority Authorization (CAA) for your Hosted Zone
55-
Depending on your DNS provider and configuration, you may need to create a CAA record in your hosted zone.
56-
On instance creation, this Terraform module will use [Let's Encrypt](https://letsencrypt.org/) to generate
57-
a valid SSL Certificate. For that to succeed, Let's Encrypt must be recognized as a valid issuing CA by
58-
your domain. To do this, add a DNS record that looks like this:
168+
### Certificate Authority Authorization (CAA) for your Hosted Zone
169+
Depending on your DNS provider and configuration, you may need to create a CAA record in your hosted zone.vOn instance creation, this Terraform module will use [Let's Encrypt](https://letsencrypt.org/) to generate a valid SSL Certificate. For that to succeed, Let's Encrypt must be recognized as a valid issuing CA by your domain. To do this, add a DNS record that looks like this:
59170
60171
```
61172
Domain Record type Value
62173
example.com. CAA 0 issue "letsencrypt.org"
63174
```
64175
65176
## Troubleshooting
66-
You can get help deploying the DLE. Here are two great ways to do this:
67-
68-
* Join the [Database Lab Community Slack](https://database-lab-team.slack.com)
69-
* Reach out to the Postgres.ai team on [Intercom chat widget](https://postgres.ai/) (located at the bottom right corner)
177+
You can get help deploying the DLE. Here are two great ways to do this:
178+
- Join the [Database Lab Community Slack](https://database-lab-team.slack.com)
179+
- Reach out to the Postgres.ai team on [Intercom chat widget](https://postgres.ai/) (located at the bottom right corner)
70180
71181
## Reporting Issues & Contributing
72-
We want to make deploying and managing the Database Lab Engine as easy as possible! Please report bugs
182+
We want to make deploying and managing the Database Lab Engine as easy as possible! Please report bugs
73183
and submit feature ideas using Gitlab's [Issue feature](https://gitlab.com/postgres-ai/database-lab-infrastructure/-/issues/new).
74184

‎ami.tf

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
data "aws_ami" "ami" {
22
most_recent = true
3-
owners = ["${var.ami_owner}"]
3+
owners = ["${var.aws_ami_owner}"]
44

55
filter {
66
name = "name"
7-
values = ["${format("%s-%s*", var.dle_ami_name, var.dle_version_full)}"]
7+
values = ["${var.aws_ami_name}"]
88
}
99
filter {
1010
name = "root-device-type"

0 commit comments

Comments
 (0)