Skip to content
Snippets Groups Projects
Commit 88e0ce80 authored by abir.chebbi's avatar abir.chebbi
Browse files

update the code + Readme

parent 0d56a4cb
No related branches found
No related tags found
No related merge requests found
...@@ -4,7 +4,7 @@ import rioxarray as rxa ...@@ -4,7 +4,7 @@ import rioxarray as rxa
import tempfile import tempfile
s3 = boto3.client('s3') s3 = boto3.client('s3')
bucket_name= 'ndvi-satellite-images' bucket_name= 'satellite-images-ndvi'
def lambda_handler(event, context): def lambda_handler(event, context):
......
...@@ -5,7 +5,7 @@ import os ...@@ -5,7 +5,7 @@ import os
from datetime import datetime from datetime import datetime
s3 = boto3.client('s3') s3 = boto3.client('s3')
bucket_name = 'ndvi-satellite-images' bucket_name = 'satellite-images-ndvi'
def lambda_handler(event, context): def lambda_handler(event, context):
......
...@@ -8,48 +8,46 @@ The Normalized Difference Vegetation Index (NDVI) is a critical measure for eval ...@@ -8,48 +8,46 @@ The Normalized Difference Vegetation Index (NDVI) is a critical measure for eval
This project aims to calculate the average NDVI for Argentina using AWS Step Functions to orchestrate parallel processing. The workflow involves reading NDVI images from an S3 bucket, calculating the mean NDVI, and then providing options to save the results in various formats. This project aims to calculate the average NDVI for Argentina using AWS Step Functions to orchestrate parallel processing. The workflow involves reading NDVI images from an S3 bucket, calculating the mean NDVI, and then providing options to save the results in various formats.
The NDVI images, sourced from the 'ndvi-satellite-images' S3 bucket, are provided by NASA satellites, representing bi-weekly NDVI values. Image filenames follow the "argentina_date.tif" pattern, with dates formatted as a four-digit year followed by a three-digit day of the year. The NDVI images, sourced from the **'satellite-images-ndvi'** S3 bucket, are provided by NASA satellites, representing bi-weekly NDVI values. Image filenames follow the **"argentina_date.tif"** pattern, where **date** is formatted as a four-digit year followed by a three-digit day of the year.
## Lambda functions: ## Lambda functions:
We create three lambda functions:
### Compute NDVI Mean ### Lambda Function 1: Compute NDVI Mean:
This lambda function will read an NDVI image from S3 bucket, compute the mean NDVI value using the ‘rioxarray’ library, and return the result. This lambda function will read an NDVI image from S3 bucket, compute the mean NDVI value using the ‘rioxarray’ library, and return the result.
Follow these steps to configure your lambda function: Follow these steps to configure your lambda function:
1. Lambda function creation: 1. Lambda function creation:
* Navigate to AWS Lambda and initiate the creation of a new function. * Navigate to AWS Lambda and initiate the creation of a new function from scratch.
* For the runtime environment, select Python 3.10. * For the runtime environment, select Python 3.10.
* Under permissions, assign the following IAM policies: * Under permissions, choose the exisiting role **faas_ndvi_role**.
* AmazonS3FullAccess: This allows the function to read files from S3 buckets.
* AWSLambdaBasicExecutionRole: This grants the function permission to write logs for monitoring and debugging purposes.
2. Dependency Management: 2. Dependency Management:
Add a custom layer named **ndvi-rioxarray**. This layer should include the rioxarray library along with any other dependencies necessary for processing NDVI images. Add a custom layer named **ndvi-rioxarray**. This layer should include the rioxarray library along with any other dependencies necessary for processing NDVI images.
3. Function Code: 3. Function Code:
In the Lambda function's code editor, paste the contents of the **Compute NDVI Mean.py** script. This script contains the logic to read an NDVI image, compute its mean NDVI value using rioxarray, and return the computed mean. In the Lambda function's code editor, paste the contents of the **Compute NDVI Mean.py** script. This script contains the logic to read an NDVI image, compute its mean NDVI value using rioxarray, and return the computed mean.
4. Go to 'Configuration'. Edit 'General configuration':
* timeout = 15 min
* Memory = 10240 MB
* Ephemeral storage = 10240 MB
### Save Results to CSV ### Lambda Function 2: Save Results to CSV
This function will take the NDVI data and save it to a CSV file in S3. This function will take the NDVI data and save it to a CSV file in S3.
1. Lambda function creation: 1. Lambda function creation:
* Navigate to AWS Lambda and initiate the creation of a new function. * Navigate to AWS Lambda and initiate the creation of a new function from scratch.
* For the runtime environment, select Python 3.10. * For the runtime environment, select Python 3.10.
* Under permissions, assign the following IAM policies: * Under permissions, choose the exisiting role **faas_ndvi_role**.
* AmazonS3FullAccess: This allows the function to post results in S3 bucket.
* AWSLambdaBasicExecutionRole: This grants the function permission to write logs for monitoring and debugging purposes.
2. Function Code: 2. Function Code:
In the Lambda function's code editor, paste the contents of the **Save Results to CSV.py** script. This script contains the logic to save the results into CSV file and put it in S3 bucket. In the Lambda function's code editor, paste the contents of the **Save Results to CSV.py** script. This script contains the logic to save the results into CSV file and put it in S3 bucket.
### Plot NDVI Time Series ### Lambda Function 3: Plot NDVI Time Series
This function will receive NDVI mean values over time, plot them, and generate a PNG image. Save the PNG image in S3 bucket This function will receive NDVI mean values over time, plot them, and generate a PNG image. Save the PNG image in S3 bucket
1. Lambda function creation: 1. Lambda function creation:
* Navigate to AWS Lambda and initiate the creation of a new function. * Navigate to AWS Lambda and initiate the creation of a new function from scratch.
* For the runtime environment, select Python 3.10. * For the runtime environment, select Python 3.10.
* Under permissions, assign the following IAM policies: * Under permissions, choose the exisiting role **faas_ndvi_role**.
* AmazonS3FullAccess: This allows the function to rpost results in S3 bucket.
* AWSLambdaBasicExecutionRole: This grants the function permission to write logs for monitoring and debugging purposes.
2. Dependency Management: 2. Dependency Management:
Add a custom layer named **matplotlib**. This layer should include the rioxarray library along with any other dependencies necessary for processing NDVI images. Add a custom layer named **matplotlib**. This layer should include the rioxarray library necessary for plotting time series data.
3. Function Code: 3. Function Code:
In the Lambda function's code editor, paste the contents of the **Plot NDVI Time Series.py** script. This script contains the logic to plot the results, save it into png format and put it in S3 bucket. In the Lambda function's code editor, paste the contents of the **Plot NDVI Time Series.py** script. This script contains the logic to plot the results, save it into png format and put it in S3 bucket.
...@@ -66,20 +64,21 @@ The following image represent the application workflow. Each step in the flowcha ...@@ -66,20 +64,21 @@ The following image represent the application workflow. Each step in the flowcha
* Choose **Blank** to create a workflow from scratch. * Choose **Blank** to create a workflow from scratch.
Inside the workflow studio, navigate to the left-hand panel and select **Flow** control. Inside the workflow studio, navigate to the left-hand panel and select **Flow** control.
#### Step 1: Map block #### Step 1: Map block
Choose **Map**, which allows for parallel processing. Note that based on your workflow, you might not need to use Map if there's no parallel processing required. Choose **Map**, which allows for parallel processing:
* Assign a suitable name * Assign a suitable name
* Select the processing mode as 'Distributed,' facilitating the creation of separate child workflows for each image. * Select the processing mode as 'Distributed,' facilitating the creation of separate child workflows for each image.
* Choose the item source as 'Amazon S3,' then select 'S3 object list' as the S3 item source. Navigate to the S3 bucket 'ndvi-satellite-images,' where the NDVI images are stored. * Choose the item source as 'Amazon S3,' then select 'S3 object list' as the S3 item source. Navigate to the S3 bucket **'satellite-images-ndvi'**, where the NDVI images are stored.
* Navigate to the output panel and select 'Add original input to output using ResultPath'. * Navigate to the output panel and select 'Add original input to output using ResultPath' and add a ResultPath.
#### Step 2: Configure a state to compute NDVI mean #### Step 2: Configure a state to compute NDVI mean
* Drag and drop AWS Lambda from the Actions panel to add a task to compute the mean of NDVI. Configure: * Drag and drop AWS Lambda(Invoke) from the Actions panel to add a task to compute the mean of NDVI. Configure:
* Assign a name to the state. * Assign a name to the state.
* Within the 'API Parameters' section, select 'Enter function name' to trigger the previously created lambda function responsible for computing the mean. Input the ARN (Amazon Resource Name) of the function accordingly. * Within the 'API Parameters' section, select the lambda function previously created responsible for computing the mean.
#### Step 3: Add Choice Flow #### Step 3: Add Choice Flow
* Drag and drop the 'Choice' state from the Flow panel. * Drag and drop the 'Choice' state from the Flow panel.
* Configure the Choice state by adding rules: * Configure the Choice state by adding rules:
* Set Rule #1. * Set Rule #1.
* Define a condition based on the variable $.type. * Add a condition
* Define a condition based on the variable '$.type'.
* Select 'is equal to' as the operator and set the value to a String constant of "CSV". * Select 'is equal to' as the operator and set the value to a String constant of "CSV".
#### Step 4: Configure Lambda Tasks for the Condition #### Step 4: Configure Lambda Tasks for the Condition
* Within the 'CSV' condition set in the Choice state, include the Lambda function which generates output data in CSV format. * Within the 'CSV' condition set in the Choice state, include the Lambda function which generates output data in CSV format.
......
...@@ -4,7 +4,7 @@ import json ...@@ -4,7 +4,7 @@ import json
from datetime import datetime from datetime import datetime
s3 = boto3.client('s3') s3 = boto3.client('s3')
bucket_name = 'ndvi-satellite-images' bucket_name = 'satellite-images-ndvi'
def lambda_handler(event, context): def lambda_handler(event, context):
tmp = [ arr for arr in event["result"] if arr is not None] tmp = [ arr for arr in event["result"] if arr is not None]
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment