java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties

GitHub | borisclemencon | 7 months ago
tip
Click on the to mark the solution that helps you, Samebug will learn from it.
As a community member, you’ll be rewarded for you help.
  1. 0

    IAM Role not taken into account

    GitHub | 7 months ago | borisclemencon
    java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
  2. 0

    `aws_iam_role` not being used

    GitHub | 9 months ago | Gauravshah
    java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).

    Root Cause Analysis

    1. java.lang.IllegalArgumentException

      AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties

      at com.databricks.spark.redshift.S3Credentials.initialize()
    2. com.databricks.spark
      AWSCredentialsUtils$$anonfun$load$1.apply
      1. com.databricks.spark.redshift.S3Credentials.initialize(S3Credentials.java:67)
      2. com.databricks.spark.redshift.AWSCredentialsUtils$.com$databricks$spark$redshift$AWSCredentialsUtils$$loadFromURI(AWSCredentialsUtils.scala:60)
      3. com.databricks.spark.redshift.AWSCredentialsUtils$$anonfun$load$1.apply(AWSCredentialsUtils.scala:48)
      4. com.databricks.spark.redshift.AWSCredentialsUtils$$anonfun$load$1.apply(AWSCredentialsUtils.scala:48)
      4 frames
    3. Scala
      Option.getOrElse
      1. scala.Option.getOrElse(Option.scala:121)
      1 frame
    4. com.databricks.spark
      RedshiftRelation.buildScan
      1. com.databricks.spark.redshift.AWSCredentialsUtils$.load(AWSCredentialsUtils.scala:48)
      2. com.databricks.spark.redshift.RedshiftRelation.buildScan(RedshiftRelation.scala:89)
      2 frames
    5. org.apache.spark
      DataSourceStrategy$$anonfun$pruneFilterProject$1.apply
      1. org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$8.apply(DataSourceStrategy.scala:260)
      2. org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$8.apply(DataSourceStrategy.scala:260)
      3. org.apache.spark.sql.execution.datasources.DataSourceStrategy$$anonfun$pruneFilterProject$1.apply(DataSourceStrategy.scala:303)
      3 frames