Can we trigger AWS Lambda function from aws Glue PySpark job?
Currently i'm able to run Glue PySpark job, but is this possible to call a lambda function from Glue this job ? Using below code from my PySpark Glue job i'm calling lambda function.
lambda_client = boto3.client('lambda', region_name='us-west-2')
response = lambda_client.invoke(FunctionName='test-lambda')
Error:
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Invoke operation: User: arn:aws:sts::208244724522:assumed-role/AWSGlueServiceRoleDefault/GlueJobRunnerSession is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:208244724522:function:hw-test
But I added proper lambda roles to my Glue iam role, still getting above error. Any specific role need to add ?
Thanks.
pyspark aws-lambda aws-glue
edited Nov 14 '18 at 13:58
Sangam Belose
1,95941824
asked Nov 14 '18 at 8:09
RK.RK.
246
add a comment |
Currently i'm able to run Glue PySpark job, but is this possible to call a lambda function from Glue this job ? Using below code from my PySpark Glue job i'm calling lambda function.
lambda_client = boto3.client('lambda', region_name='us-west-2')
response = lambda_client.invoke(FunctionName='test-lambda')
Error:
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Invoke operation: User: arn:aws:sts::208244724522:assumed-role/AWSGlueServiceRoleDefault/GlueJobRunnerSession is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:208244724522:function:hw-test
But I added proper lambda roles to my Glue iam role, still getting above error. Any specific role need to add ?
Thanks.
pyspark aws-lambda aws-glue
edited Nov 14 '18 at 13:58
Sangam Belose
1,95941824
asked Nov 14 '18 at 8:09
RK.RK.
246
I saw few links where this is not possible. Thats why raised a question here.
– RK.
Nov 14 '18 at 8:14
can you share IAM role and policy?
– statut
Nov 14 '18 at 11:14
Below two roles attached to my Gluedefault role id : AWSLambdaBasicExecutionRole AWSLambdaVPCAccessExecutionRole
– RK.
Nov 14 '18 at 11:36
add a comment |
Currently i'm able to run Glue PySpark job, but is this possible to call a lambda function from Glue this job ? Using below code from my PySpark Glue job i'm calling lambda function.
lambda_client = boto3.client('lambda', region_name='us-west-2')
response = lambda_client.invoke(FunctionName='test-lambda')
Error:
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Invoke operation: User: arn:aws:sts::208244724522:assumed-role/AWSGlueServiceRoleDefault/GlueJobRunnerSession is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:208244724522:function:hw-test
But I added proper lambda roles to my Glue iam role, still getting above error. Any specific role need to add ?
Thanks.
pyspark aws-lambda aws-glue
edited Nov 14 '18 at 13:58
Sangam Belose
1,95941824
asked Nov 14 '18 at 8:09
RK.RK.
246
Currently i'm able to run Glue PySpark job, but is this possible to call a lambda function from Glue this job ? Using below code from my PySpark Glue job i'm calling lambda function.
lambda_client = boto3.client('lambda', region_name='us-west-2')
response = lambda_client.invoke(FunctionName='test-lambda')
Error:
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Invoke operation: User: arn:aws:sts::208244724522:assumed-role/AWSGlueServiceRoleDefault/GlueJobRunnerSession is not authorized to perform: lambda:InvokeFunction on resource: arn:aws:lambda:us-west-2:208244724522:function:hw-test
But I added proper lambda roles to my Glue iam role, still getting above error. Any specific role need to add ?
Thanks.
pyspark aws-lambda aws-glue
pyspark aws-lambda aws-glue
edited Nov 14 '18 at 13:58
Sangam Belose
1,95941824
asked Nov 14 '18 at 8:09
RK.RK.
246
edited Nov 14 '18 at 13:58
Sangam Belose
1,95941824
asked Nov 14 '18 at 8:09
RK.RK.
246
edited Nov 14 '18 at 13:58
Sangam Belose
1,95941824
edited Nov 14 '18 at 13:58
Sangam Belose
1,95941824
edited Nov 14 '18 at 13:58
Sangam Belose
1,95941824
1,95941824
asked Nov 14 '18 at 8:09
RK.RK.
246
asked Nov 14 '18 at 8:09
RK.RK.
246
asked Nov 14 '18 at 8:09
RK.RK.
246
246
I saw few links where this is not possible. Thats why raised a question here.
– RK.
Nov 14 '18 at 8:14
can you share IAM role and policy?
– statut
Nov 14 '18 at 11:14
Below two roles attached to my Gluedefault role id : AWSLambdaBasicExecutionRole AWSLambdaVPCAccessExecutionRole
– RK.
Nov 14 '18 at 11:36
add a comment |
I saw few links where this is not possible. Thats why raised a question here.
– RK.
Nov 14 '18 at 8:14
can you share IAM role and policy?
– statut
Nov 14 '18 at 11:14
Below two roles attached to my Gluedefault role id : AWSLambdaBasicExecutionRole AWSLambdaVPCAccessExecutionRole
– RK.
Nov 14 '18 at 11:36
I saw few links where this is not possible. Thats why raised a question here.
– RK.
Nov 14 '18 at 8:14
I saw few links where this is not possible. Thats why raised a question here.
– RK.
Nov 14 '18 at 8:14
can you share IAM role and policy?
– statut
Nov 14 '18 at 11:14
can you share IAM role and policy?
– statut
Nov 14 '18 at 11:14
Below two roles attached to my Gluedefault role id : AWSLambdaBasicExecutionRole AWSLambdaVPCAccessExecutionRole
– RK.
Nov 14 '18 at 11:36
Below two roles attached to my Gluedefault role id : AWSLambdaBasicExecutionRole AWSLambdaVPCAccessExecutionRole
– RK.
Nov 14 '18 at 11:36
add a comment |
1 Answer
1
active
oldest
votes
To invoke AWS Lambda you can use the following policy:
"Version": "2012-10-17",
"Statement": [
"Sid": "AllowToExampleFunction",
"Effect": "Allow",
"Action": "lambda:InvokeFunction",
"Resource": "arn:aws:lambda:<region>:<123456789012>:function:<example_function>"
]
Your roles are not suitable for Lambda invocations as
AWSLambdaBasicExecutionRole – Grants permissions only for the Amazon CloudWatch Logs actions to write logs. You can use this policy
if your Lambda function does not access any other AWS resources except
writing logs.
AWSLambdaVPCAccessExecutionRole – Grants permissions for Amazon Elastic Compute Cloud (Amazon EC2) actions to manage elastic network
interfaces (ENIs).
Please see documentation here about these roles.
answered Nov 14 '18 at 11:42
statutstatut
651412
Yes Thanks. AWSLambdaRole also can be used instead.
– RK.
Nov 15 '18 at 9:50
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53295575%2fcan-we-trigger-aws-lambda-function-from-aws-glue-pyspark-job%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
To invoke AWS Lambda you can use the following policy:
"Version": "2012-10-17",
"Statement": [
"Sid": "AllowToExampleFunction",
"Effect": "Allow",
"Action": "lambda:InvokeFunction",
"Resource": "arn:aws:lambda:<region>:<123456789012>:function:<example_function>"
]
Your roles are not suitable for Lambda invocations as
AWSLambdaBasicExecutionRole – Grants permissions only for the Amazon CloudWatch Logs actions to write logs. You can use this policy
if your Lambda function does not access any other AWS resources except
writing logs.
AWSLambdaVPCAccessExecutionRole – Grants permissions for Amazon Elastic Compute Cloud (Amazon EC2) actions to manage elastic network
interfaces (ENIs).
Please see documentation here about these roles.
answered Nov 14 '18 at 11:42
statutstatut
651412
Yes Thanks. AWSLambdaRole also can be used instead.
– RK.
Nov 15 '18 at 9:50
add a comment |
To invoke AWS Lambda you can use the following policy:
"Version": "2012-10-17",
"Statement": [
"Sid": "AllowToExampleFunction",
"Effect": "Allow",
"Action": "lambda:InvokeFunction",
"Resource": "arn:aws:lambda:<region>:<123456789012>:function:<example_function>"
]
Your roles are not suitable for Lambda invocations as
AWSLambdaBasicExecutionRole – Grants permissions only for the Amazon CloudWatch Logs actions to write logs. You can use this policy
if your Lambda function does not access any other AWS resources except
writing logs.
AWSLambdaVPCAccessExecutionRole – Grants permissions for Amazon Elastic Compute Cloud (Amazon EC2) actions to manage elastic network
interfaces (ENIs).
Please see documentation here about these roles.
answered Nov 14 '18 at 11:42
statutstatut
651412
Yes Thanks. AWSLambdaRole also can be used instead.
– RK.
Nov 15 '18 at 9:50
add a comment |
To invoke AWS Lambda you can use the following policy:
"Version": "2012-10-17",
"Statement": [
"Sid": "AllowToExampleFunction",
"Effect": "Allow",
"Action": "lambda:InvokeFunction",
"Resource": "arn:aws:lambda:<region>:<123456789012>:function:<example_function>"
]
Your roles are not suitable for Lambda invocations as
AWSLambdaBasicExecutionRole – Grants permissions only for the Amazon CloudWatch Logs actions to write logs. You can use this policy
if your Lambda function does not access any other AWS resources except
writing logs.
AWSLambdaVPCAccessExecutionRole – Grants permissions for Amazon Elastic Compute Cloud (Amazon EC2) actions to manage elastic network
interfaces (ENIs).
Please see documentation here about these roles.
answered Nov 14 '18 at 11:42
statutstatut
651412
To invoke AWS Lambda you can use the following policy:
"Version": "2012-10-17",
"Statement": [
"Sid": "AllowToExampleFunction",
"Effect": "Allow",
"Action": "lambda:InvokeFunction",
"Resource": "arn:aws:lambda:<region>:<123456789012>:function:<example_function>"
]
Your roles are not suitable for Lambda invocations as
AWSLambdaBasicExecutionRole – Grants permissions only for the Amazon CloudWatch Logs actions to write logs. You can use this policy
if your Lambda function does not access any other AWS resources except
writing logs.
AWSLambdaVPCAccessExecutionRole – Grants permissions for Amazon Elastic Compute Cloud (Amazon EC2) actions to manage elastic network
interfaces (ENIs).
Please see documentation here about these roles.
answered Nov 14 '18 at 11:42
statutstatut
651412
answered Nov 14 '18 at 11:42
statutstatut
651412
answered Nov 14 '18 at 11:42
statutstatut
651412
answered Nov 14 '18 at 11:42
statutstatut
651412
651412
Yes Thanks. AWSLambdaRole also can be used instead.
– RK.
Nov 15 '18 at 9:50
add a comment |
Yes Thanks. AWSLambdaRole also can be used instead.
– RK.
Nov 15 '18 at 9:50
Yes Thanks. AWSLambdaRole also can be used instead.
– RK.
Nov 15 '18 at 9:50
Yes Thanks. AWSLambdaRole also can be used instead.
– RK.
Nov 15 '18 at 9:50
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53295575%2fcan-we-trigger-aws-lambda-function-from-aws-glue-pyspark-job%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I saw few links where this is not possible. Thats why raised a question here.
– RK.
Nov 14 '18 at 8:14
can you share IAM role and policy?
– statut
Nov 14 '18 at 11:14
Below two roles attached to my Gluedefault role id : AWSLambdaBasicExecutionRole AWSLambdaVPCAccessExecutionRole
– RK.
Nov 14 '18 at 11:36