Introduction
In today’s data-driven world, seamless data integration is crucial for businesses striving to harness the power of their data. Whether it’s consolidating data from multiple sources, ensuring data consistency across platforms, or preparing data for advanced analytics, an efficient data integration tool is a game-changer. This is where Airbyte comes in. As a powerful, open-source data integration platform, Airbyte makes it easy to sync data from any source to any destination. In this blog, we’ll explore the capabilities of Airbyte, its potential use-cases, and how you can leverage our AWS Marketplace products to streamline your data integration processes.
For more detailed information and knowledge regarding Airbyte and Data Integration, refer to out blog Effortless Data Integration using Airbyte.
Get Started:
Get started with our Enterprise Marketplace offering in under 10 mins.
How to Access our product?
To access our Data Integration solution, please visit the AWS Marketplace offering of our product.
The deployment of the product involves these steps:
- Subscribe to Marketplace product
- Select the Fulfillment Method according to the needs (We offer Dedicated-Private-VPC and Private-Existing-VPC methods)
- Enter all the required details asked in the CloudFormation UI and create the stack.
Subscribe to our AWS Marketplace product:
Our product comes in the AMI + CloudFormation delivery method, in which the whole infrastructure is made using CloudFormation and the Software comes in an EC2 instance.
Follow these steps to subscribe to our product:
- Visit this AWS Marketplace offering page and Click on Continue to subscribe.
- Some Terms and Conditions are shown. Review and accept them.
- Pending Subscription. Wait till it gets activated.
- After activation, configure the software. We provide this solution in 2 Fulfillment methods:
- Private Dedicated VPC (Recommended).
- Private Existing VPC.
- Choose the Fulfillment method and region (us-east-1 only currently).
- Click on Continue to Launch.
- Choose the action as Launch CloudFormation and click Launch.
- You will be redirected to the CloudFormation console.
Understanding the Fulfillment Methods
We are providing 2 Fulfillment methods for our solution:
- Private Dedicated VPC (Recommended): This Delivery method is designed to meet your internal security requirements. Below are the created resources:
- Automated VPC Setup: Our CloudFormation template automates the creation of a dedicated VPC, subnets, route tables, and necessary networking components, ensuring a secure and isolated environment for your application. (2 Public and 2 Private subnets).
- Application Load Balancer (ALB): An ALB is set up to distribute incoming traffic across multiple EC2 instances, improving reliability and fault tolerance of your application.
- SSL Certificate Support: Easily configure SSL certificates for your domain, enabling secure HTTPS connections for your application. Simply provide the necessary SSL certificate details, including sub-domain and Hosted Zone ID, for seamless integration.
- Optional SSL Configuration: While SSL certificate setup is optional, it’s recommended for ensuring secure communication between clients and your application.
- Custom Domain Support: You can specify a custom domain for your application, allowing you to showcase your brand and provide a more professional experience for your users.
- Default Auth Mechanism: While launching the CloudFormation template, the user must enter the Username and Password for the Application and the Config Database. Which are then stored in the Secrets Manager for future accessing.
- Pricing: This architecture bills around $90 per Month, see this estimate. (Additional charges may occur according to the usage of Route 53 and ACM, which are optional in this architecture). Refer the below architecture diagram for better understanding:
- Private Existing VPC: A cost-saving architecture of the above fulfillment method, which uses your existing infrastructure to create the whole application. Below are the created resources.
- Integration with Existing Infrastructure: Leverage your existing VPC and subnets by providing their details during setup, ensuring seamless integration with your current AWS environment.
- Flexible Networking Configuration: With the ability to input your existing VPC and subnet details, you have full control over the networking configuration, allowing you to align the setup with your specific requirements.
- ALB Integration: Integrate an Application Load Balancer into your existing VPC setup to efficiently distribute incoming traffic across your EC2 instances, enhancing reliability and availability.
- Optional SSL Support: If desired, configure SSL certificates for your custom domain to enable secure HTTPS connections for your application. Simply provide the necessary SSL certificate details for seamless integration.
- Enhanced Security: By utilizing your existing VPC infrastructure, you maintain the security and isolation benefits already established within your AWS environment.
- Default Auth Mechanism: While launching the CloudFormation template, the user must enter the Username and Password for the Application and the Config Database. Which are then stored in the Secrets Manager for future accessing.
- Pricing: This architecture bills around $60 per Month (Assuming that a NAT Gateway is provided by the user), see this estimate (Additional charges may occur according to the usage of Route 53, ACM, which are optional in this architecture). Refer architecture below:
Filling the CloudFormation:
Private Dedicated VPC (Recommended):
- Application Configuration: This section contains the credential details
- UIUserName: This expects a username for the application.
- UIPassword: This expects a password for the application. We suggest you keep a strong and unique password for the security of the application.
- Instance Configuration: This contains the details of the EC2 instance. We recommend using the default value for optimum performance.
- Instance Type: This expects the Instance type of the EC2. We suggest m5.xlarge type to ensure performance and t3.xlarge for saving costs.
- Network Configuration: This contains the details of the Network parameters that form the infrastructure. All the parameters expect CIDR notation; an error will be caused if it is incorrect.
- VpcCidrBlock: This is the IPv4 network range for the VPC
- PrivateSubnetCIDR: IPv4 address of Private Subnet
- SubnetAPublicCIDR: IPv4 address of the Public Subnet 1
- SubnetBPublicCIDR: IPv4 address of the Public Subnet 2
- RequireALB: If the User needs an application load Balancer to access the application. (YES / NO).
- CloudFront Distribution: This is an optional value, which creates a CloudFront Distribution to get a temporary HTTPS URL for the application. (Connection from Internet to CloudFront is HTTPS, from Cloudfront to EC2 (Internally) is HTTP.)
- RequireCloudFront: Whether the user wants a CloudFront Distribution created by the stack.
- SecurityGroupCIDR: Security Group CIDR for ALB users to access the application. (Works only if RequireALB is true.)
Private Existing VPC:
In the Private Existing VPC setup, all parameters except network configurations remain identical to the Private Dedicated VPC setup. The Network Configuration details are shown below:
- Network Configuration: This contains the details of the Network parameters that form the infrastructure. The user should have a VPC, 2 public and 2 private subnets created in advance, to select them here in the parameters.
- VpcId: Select the VPC (by ID), in which the application launches.
- PrivateSubnet: Strictly select the Private Subnet (by ID) here. (Error may occur if Private Subnet is not selected)
- PublicSubnetA: Strictly select the First Public Subnet (by ID) here. (Error may occur if Public Subnet is not selected)
- PublicSubnetB: Strictly select the Second Public Subnet (by ID) here. (Error may occur if Public Subnet is not selected)
- ExistingNatGatewayId: Enter the ID of the NAT Gateway, if there is one already created in the above selected VPC. Leave it empty if you do not have it, the CloudFormation template will create it.
- InternetGateway: InternetGateway ID, for the Public Subnet to use in order to access the Internet.
Important Notes:
- Secrets Management: User-provided usernames and passwords for both the application and configuration database are securely stored as secrets within AWS Secrets Manager, seamlessly integrated into the CloudFormation template. This ensures that sensitive credentials are safeguarded and easily manageable, enhancing the overall security posture of the deployment.
- CloudFront Distribution: Enabling CloudFront Distribution in the stack, will get the user a HTTPS connection to their application. Primarily, from Internet to CloudFront will be a HTTPS connection, from CloudFront to internal EC2 instance (Inside AWS), the connection will be HTTP. This is only eligible if the RequireALB is selected as YES.
- Clean Up made simple: When finished with their tasks and confident in data backups, users can delete the entire CloudFormation Stack from the console UI. Select the stack by name and click delete. Note: Deletion may fail if there are explicit dependencies. In such cases, delete dependencies first, then stack. This streamlined process ensures efficient resource management and system cleanliness.