Accessing the Oracle Object Store
OCI Object Store Examples¶
The following are a series of examples showing the loading of data into the Oracle Object Store. For these to work with your own data you'll need to have your own Oracle Cloud account and uploaded a key. You can find details on how to achieve this here
I'll be using the Oracle OCI Python SDK which wrappers the REST API. You can find details on the API here
Before we do anything we'll need to load the required needed Python modules.
import oci
import keyring
import ast
import os
Configuration needed to connect¶
I'm using the "keyring" Python module to hold the config for my connection to OCI (to avoid needlessly exposing sensitive information). It's of the form
{
"user": "your user ocid",
"key_file": "the path to your private key file",
"fingerprint": "the fingerprint of your public key",
"tenancy": "your tenancy ocid",
"region": "the region you are working with"
}
After retrieving it from my keyring store I then need to convert it into a dictionary before using it. You can also validate the config you are using as well. Handy if this is the first time you've configured it.
my_config = ast.literal_eval(keyring.get_password('oci_opj','doms'))
oci.config.validate_config(my_config)
Create object storage client¶
Then I just need to retireve a Object Storage client to start working with data
object_storage_client = oci.object_storage.ObjectStorageClient(my_config)
namespace = object_storage_client.get_namespace().data
bucket_name = "doms_object_store"
Upload the contents of user directory to a bucket¶
I'll create a bucket and then select all of the files from a user defined directory and upload them to the newly created bucket
import os, io
directory = '/Users/dgiles/datagenerator/bin/generateddata'
files_to_process = [file for file in os.listdir(directory) if file.endswith('csv')]
Create a bucket named "Sales_Data" and give it the tenancy ocid from your config.
try:
create_bucket_response = object_storage_client.create_bucket(
namespace,
oci.object_storage.models.CreateBucketDetails(
name='Sales_Data',
compartment_id=my_config['tenancy']
)
)
except Exception as e:
print(e.message)
Then we just need to loop through the list of files in the directory specified and upload them to the newly created bucket
bucket_name = 'Sales_Data'
for upload_file in files_to_process:
print('Uploading file {}'.format(upload_file))
object_storage_client.put_object(namespace, bucket_name, upload_file, io.open(os.path.join(directory,upload_file),'r'))
Retrieve a list of objects in a bucket¶
The folowing retrieves a bucket and gets a list of objects in the bucket
bucket = object_storage_client.get_bucket(namespace, bucket_name)
object_list = object_storage_client.list_objects(namespace, bucket_name)
for o in object_list.data.objects:
print(o.name)
Download the contents of an object¶
The following downloads a file from a named bucket in chunks and writes it to user defined directory on the client
# Attempt to download a file
object_name = "CUSTOMERS.csv"
destination_dir = '/Users/dgiles/Downloads'.format(object_name)
get_obj = object_storage_client.get_object(namespace, bucket_name, object_name)
with open(os.path.join(destination_dir,object_name), 'wb') as f:
for chunk in get_obj.data.raw.stream(1024 * 1024, decode_content=False):
f.write(chunk)
Delete a bucket¶
We can just as simply delete the bucket we've just created but first we'll need to delete all of the objects inside of it.
object_list = object_storage_client.list_objects(namespace, bucket_name)
for o in object_list.data.objects:
print('Deleting object {}'.format(o.name))
object_storage_client.delete_object(namespace, bucket_name, o.name)
print('Deleting bucket')
response = object_storage_client.delete_bucket(namespace, bucket_name)