Data Engineering Using AWS Analytics Services

Description :

Warning: Illegal string offset 'ID' in /home/onlinecou/public_html/wp-content/plugins/advanced-custom-fields-pro/includes/api/api-field.php on line 890

Warning: Illegal string offset 'key' in /home/onlinecou/public_html/wp-content/plugins/advanced-custom-fields-pro/includes/api/api-field.php on line 891

Warning: Illegal string offset 'label' in /home/onlinecou/public_html/wp-content/plugins/advanced-custom-fields-pro/includes/api/api-field.php on line 892

Warning: Illegal string offset 'name' in /home/onlinecou/public_html/wp-content/plugins/advanced-custom-fields-pro/includes/api/api-field.php on line 893

Warning: Illegal string offset 'menu_order' in /home/onlinecou/public_html/wp-content/plugins/advanced-custom-fields-pro/includes/api/api-field.php on line 894

Warning: Illegal string offset 'parent' in /home/onlinecou/public_html/wp-content/plugins/advanced-custom-fields-pro/includes/api/api-field.php on line 895

Warning: Illegal string offset 'key' in /home/onlinecou/public_html/wp-content/plugins/advanced-custom-fields-pro/includes/api/api-field.php on line 899
Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lake or Data Warehouse and then from Data Lake or Data Warehouse to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using AWS Analytics Stack. It includes services such as Glue, Elastic Map Reduce (EMR), Lambda Functions, Athena, QuickSight, and many more. Here are the high-level steps which you will follow as part of the course. Setup Development Environment Getting Started with AWS Development Life Cycle of Pyspark Overview of Glue Components Setup Spark History Server for Glue Jobs Deep Dive into Glue Catalog Exploring Glue Job APIs Glue Job Bookmarks Data Ingestion using Lambda Functions Streaming Pipeline using Kinesis Consuming Data from s3 using boto3 Populating GitHub Data to Dynamodb Getting Started with AWS Introduction – AWS Getting Started Create s3 Bucket Create IAM Group and User Overview of Roles Create and Attach Custom Policy Configure and Validate AWS CLI Development Lifecycle for Pyspark Setup Virtual Environment and Install Pyspark Getting Started with Pycharm Passing Run Time Arguments Accessing OS Environment Variables Getting Started with Spark Create Function for Spark Session Setup Sample Data Read data from files Process data using Spark APIs Write data to files Validating Writing Data to Files Productionizing the Code Overview of Glue Components Introduction – Overview of Glue Components Create Crawler and Catalog Table Analyze Data using Athena Creating S3 Bucket and Role Create and Run the Glue Job Validate using Glue CatalogTable and Athena Create and Run Glue Trigger Create Glue Workflow Run Glue Workflow and Validate