lambda function to call other functions to write data
I need help with setting up aws lambda functions to do the following:
- once a day, download data from external API and transform into 3 different dataframes in python
-- already set up and working as expected
- write each of the dataframes to different dynamodb table
-- I can create a new table and write to it, but as each write takes a long time, I want to, if possible, pass a dataframe as a parameter to another lambda function that will then perform the create table and write the records (between 20k and 300k each)
I am a little bit lost in the documentation about how to call lambdas from another lambda and how to pass the dataframe (transformed into a list of dictionaries). Would the original lambda time out while waiting for the others to finish writing? Could anyone point me to relevant documentation or examples? Thank you!
- once a day, download data from external API and transform into 3 different dataframes in python
-- already set up and working as expected
- write each of the dataframes to different dynamodb table
-- I can create a new table and write to it, but as each write takes a long time, I want to, if possible, pass a dataframe as a parameter to another lambda function that will then perform the create table and write the records (between 20k and 300k each)
I am a little bit lost in the documentation about how to call lambdas from another lambda and how to pass the dataframe (transformed into a list of dictionaries). Would the original lambda time out while waiting for the others to finish writing? Could anyone point me to relevant documentation or examples? Thank you!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thank you for this, since I would like to keep the event in lambda if possible (I was able to get the data written through the step function process, but it's clunky).
One question about the event variable. I am assuming that is where I pass the records from my pandas dataframe (as a dictonary), right? This is what I wrote:
event = {
"trade_date": trade_date,
"items": items_nh
}
trade_date is just a date string "2020-03-31" for example, and items_nh is the output of df.to_dict()
However, if I use json.dumps(event) in Payload to pass to the child function, I get the error that the Decimal is not JSON serializable, but if I put just event, then I get "Parameter validation failed: Invalid type for parameter Payload". How should I deal with different parameters I want to pass? Thank you!
Also, my apologies for the delay in responding.
One question about the event variable. I am assuming that is where I pass the records from my pandas dataframe (as a dictonary), right? This is what I wrote:
event = {
"trade_date": trade_date,
"items": items_nh
}
trade_date is just a date string "2020-03-31" for example, and items_nh is the output of df.to_dict()
However, if I use json.dumps(event) in Payload to pass to the child function, I get the error that the Decimal is not JSON serializable, but if I put just event, then I get "Parameter validation failed: Invalid type for parameter Payload". How should I deal with different parameters I want to pass? Thank you!
Also, my apologies for the delay in responding.
ASKER