Java Read a List of Files in S3 Bucket

• 
• Updated appointment May 27, 2020

• 138.6k
• 4

This tutorial explains some basic file/binder operations in an AWS S3 bucket using AWS SDK for .Internet (C#). First, we create a directory in S3, then upload a file to it, then we volition listing the content of the directory and finally delete the file and folder. We show these operations in both low-level and high-level APIs.

Introduction

In this tutorial, I am going to bear witness you how to use AWS SDK for .NET to do some basic file operations on an S3 bucket. AWS provides both depression-level and high-level APIs. Beginning, we are going to come across how to use low-level APIs and then we will perform the same operations using loftier-levelAPIs.

In order to run the post-obit code, you need to install the AWSSDK.S3 Nuget package.

I take created an S3 saucepan named: my-bucket-proper noun-123 and I have created a folder named my-binder inside the saucepan,

Depression-Level APIs

The low-level APIs are mapped closely to the underlying Residuum API, hither nosotros use a Asking object to provide asking information and AWS responds with theResponse object.

Understanding S3 Path (when using low-level API)

First, we need to sympathise that there is no concept of a folder in S3, everything is an object. If I want to create a folder called sub-folder, I need to append the folder name with a / to permit AWS know that what I want is a folder, not a file... so I need to create,

my-s3-saucepan-name-123/my-folder/sub-folder/

If I don't include the trailing slash AWS will create an object called sub-folder instead of a folder.

Initializing AmazonS3Client

This is how nosotros initialize S3 clients, which we are going to use for the remaining examples,

1. cord  bucketName = "my-bucket-name-123" ;
2. string  awsAccessKey = "AKI............" ;
3. cord  awsSecretKey = "+8Bo.................................." ;
5. IAmazonS3 client =new  AmazonS3Client(_awsAccessKey, _awsSecretKey, RegionEndpoint.APSoutheast2);

Note

AmazonS3Client is thread safe, you can go far static or apply a singletone case.

Creating a folder

Here nosotros are going to create a binder called sub-folder inside my-folder

1. cord  folderPath = "my-folder/sub-folder/" ;
3. PutObjectRequest request =new  PutObjectRequest()
4. {
5.     BucketName = _bucketName,
6.     Fundamental = folderPath
7. };
9. PutObjectResponse response = client.PutObject(request);

Note1

If you forget the trailing slash in the path (i.due east. "my-binder/sub-folder") information technology would create an object chosen sub-folder.

Note2

If yous include a slash at the beginning of the path (i.e. "/my-folder/sub-folder/") information technology will create a folder with name every bit an empty string and put the remaining folders inside it.

Copying a file into a folder

The post-obit lawmaking would copy test.txt within sub-folder,

1. FileInfo file = new  FileInfo(@ "c:\exam.txt" );
2. string  path = "my-folder/sub-folder/examination.txt" ;
4. PutObjectRequest request =new  PutObjectRequest()
5. {
6.     InputStream = file.OpenRead(),
7.     BucketName = _bucketName,
8.     Key = path
9. };
11. PutObjectResponse response = customer.PutObject(request);

Listing content of a binder

The following code would list the contents of sub-folder,

1. ListObjectsRequest request = new  ListObjectsRequest
2. {
3.     BucketName = _bucketName,
4.     Prefix ="my-binder/sub-binder/"
5. };
7. ListObjectsResponse response = client.ListObjects(asking);
8. foreach  (S3Object obj in  response.S3Objects)
9. {
10.     Console.WriteLine(obj.Primal);
11. }

Deleting file/folder

In the following code first nosotros delete test.txt and then sub-folder,

2. string  filePath = "my-folder/sub-binder/test.txt" ;
3. var deleteFileRequest =new  DeleteObjectRequest
4. {
5.     BucketName = _bucketName,
6.     Key = filePath
7. };
8. DeleteObjectResponse fileDeleteResponse = client.DeleteObject(deleteFileRequest);
11. string  folderPath = "my-folder/sub-binder/" ;
12. var deleteFolderRequest =new  DeleteObjectRequest
13. {
14.     BucketName = _bucketName,
15.     Central = folderPath
16. };
17. DeleteObjectResponse folderDeleteResponse = customer.DeleteObject(deleteFolderRequest);

High-level APIs

High-level APIs are designed to mimic the semantic of File I/O operations. They are very similar to working with FileInfo and Directory.

Understanding the S3 path (when using high-level APIs)

When using high-level APIs, we need to use windows' styles paths, and so use a backslash (Not slash) in your path,

   "my-folder\sub-folder\test.txt"

Also note that, similar to low-level APIs we demand a trailing backslash to indicate a binder, for example, "my-folder\sub-folder\" indicates that sub-folder is a folder whereas "my-folder\sub-folder" indicates that sub-folder is an object inside my-folder.

Initializing AmazonS3Client

Apply the same code as low-level APIs (above) to initialize AmazonS3Client.

Creating a folder

Here we are going to create a binder chosen high-level-folder and create another binder chosen my-binder inside it.

1. string  path = @ "loftier-level-folder" ;
3. S3DirectoryInfo di =new  S3DirectoryInfo(client, _bucketName, path);
4. if  (!di.Exists)
5. {
6.     di.Create();
7.     di.CreateSubdirectory("sub-folder" );
8. }

Copying file into binder

The following code would copy exam.txt inside sub-folder,

1. FileInfo localFile = new  FileInfo(@ "c:\test.txt" );
2. string  path = @ "high-level-binder\sub-folder\test.txt" ;
4. S3FileInfo s3File =new  S3FileInfo(client, _bucketName, path);
5. if  (!s3File.Exists)
6. {
7. using  (var s3Stream = s3File.Create())
8.     {
9.         localFile.OpenRead().CopyTo(s3Stream);
10.     }
11. }

Listing content of a folder

The following code would list the content of a sub-folder,

1. cord  path = @ "high-level-binder\sub-folder\" ;
3. S3DirectoryInfo di =new  S3DirectoryInfo(customer, _bucketName, path);
4. IS3FileSystemInfo[] files = di.GetFileSystemInfos();
5. foreach  (S3FileInfo file in  files)
6. {
7.     Console.WriteLine($"{file.Name}" );
8. }

Notation

Unlike depression-level API, here the folder proper noun (sub-folder) is not listed.

Deleting file/folder

In the following code first nosotros delete exam.txt and then sub-folder,

2. cord  filePath = @ "high-level-binder\sub-folder\test.txt" ;
3. S3FileInfo s3File =new  S3FileInfo(client, _bucketName, filePath);
4. if (s3File.Exists)
5. {
6.     s3File.Delete();
7. }
10. string  folderPath = @ "high-level-folder\sub-folder\" ;
11. S3DirectoryInfo directory =new  S3DirectoryInfo(client, _bucketName, folderPath);
12. if  (directory.Exists)
13. {
14.     directory.Delete();
15. }

Wrapping up

When designing information systems, we follow a practice called single source of truth (SSOT), it ensures that every data chemical element is edited in only ane place. SSOT simplifies the information system and makes working with it a lot easier. Personally I like to extend this practice to every attribute of blueprint... when designing a UI, there is no point in giving users ii different ways to buy a product, it complicates the UI and confuses the user... this principle played a major role when desiging Shopless.

In my opinion, the same is true for designing a software library. I think past providing two different APIs (low-level and loftier-level), AWS has unnecessarily complicated the procedure of communicating with the S3 bucket, especially because these APIs use different paths systems... personally I spent hours on an error because I was using a loftier-level style path, with low-level APIs. Now, to make the thing worse, AWS is providing all the same another fashion of interacting with S3 buckets, and that is using AWS TransferUtility which runs on top of depression-level API and is the recommended style for reading/writing big objects (larger than a few GB). Have a wait at this AWS documentation and see which one is the best selection for your needs.

cramerthasins.blogspot.com

Source: https://www.c-sharpcorner.com/blogs/working-with-files-and-folders-in-s3-using-aws-sdk-for-net

Share this post