[SOLVED] AWS error downloading object from S3, "profile file cannot be null"

Issue

I've already seen this, but there was no answer to explain my problem. I first used the sample provided here (GetObject class), and it worked immediately on my desktop. However, my friend could not get it to work on his machine, nor will it work on our EC2 instance.

It was mentioned that there are to be credentials files specified, which makes sense, but I never had to do that and am pretty sure the default permissions were set to enable accessing this bucket.

Here's the stacktrace:

Exception in thread "main" java.lang.IllegalArgumentException: profile file cannot be null
    at com.amazonaws.util.ValidationUtils.assertNotNull(ValidationUtils.java:37)
    at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:142)
    at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:133)
    at com.amazonaws.auth.profile.ProfilesConfigFile.<init>(ProfilesConfigFile.java:100)
    at com.amazonaws.auth.profile.ProfileCredentialsProvider.getCredentials(ProfileCredentialsProvider.java:135)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1029)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1049)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:949)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:662)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:636)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:619)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:587)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:574)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:446)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4035)
    at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4474)
    at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4448)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4020)
    at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1307)
    at GetObject.main(GetObject.java:26)

I can guarantee that neither the bucketName nor the key params in the GetObjectRequest are null. What's the discrepancy here? Why might it succeed only on my PC? Is this at all related to the fact that I had to supplement numerous jars that the aws-sdk jar was supposed to have already (jackson-databind, jackson-core, jackson-annotations, httpclient, httpcore, commons-logging, and joda-time)? It seems similar, what with the otherwise inexplicable errors (giving non-null params, something in aws-sdk says it's null).

Solution

It looks like you solved this in the comments, but I got burned on this and want to leave a clearer answer for future readers. To be super clear, the problem here has nothing to do with files in S3. This error message has nothing to do with the file on your hard drive nor the file that you're trying to push/pull from S3. The problem is that you're initializing S3 with something like:

AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());

When you do that, it looks in ~/.aws/credentials for a list of profiles. This might work great on your computer but won't work anywhere that you're getting AWS access through an IAM role (ex. Lambda, Docker, EC2 instance, etc). The fix, is to initialize the AmazonS3Client like:

AmazonS3 s3Client = new AmazonS3Client();

If you're using code that requires some kind of credentials provider, you can also do:

AmazonS3 s3Client = new AmazonS3Client(DefaultAWSCredentialsProviderChain.getInstance());

Hopefully that helps the next person. In my case, I was using DynamoDB and SQS, but I had the same error. I originally ignored this question because I thought your problem was S3 related and was super confused. Wrist slapped.

On newer versions of the SDK, you'll need to use following code:

AmazonS3 s3Client = AmazonS3ClientBuilder.standard()                  
                      .withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
                      .build();

Answered By - Ryan Shillington