Difference between revisions of "Openstack Object Storage"

From CAC Documentation wiki
Jump to navigation Jump to search
 
(16 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
__TOC__
 
__TOC__
  
Red Cloud object storage can be accessed using the following interfaces:
+
[[Red Cloud]] object storage can be accessed using the following interfaces:
 +
 
 +
=='''Web Interface'''==
 +
Red Cloud object storage is accessible via [[ Openstack | Openstack web interface ("Horizon")]. After logging into the web interface,
 +
 
 +
* Under the "Project" tab, click on "Object Store,", or
 +
* Go to [https://redcloud.cac.cornell.edu/dashboard/project/containers https://redcloud.cac.cornell.edu/dashboard/project/containers] for direct access.
  
 
=='''Swift''' ==
 
=='''Swift''' ==
Line 8: Line 14:
  
 
=='''S3''' ==
 
=='''S3''' ==
'''Known Issue''': At this time, Red Cloud's s3 interface supports only AWS signature v2. `InvalidAccessKeyId` error will be returned if AWS signature v4 is used.
+
# To use the S3 interface, first create ec2 credentials for your Openstack account using the <code>openstack ec2 credentials create</code> command in the [[Using_Openstack_CLI_Client | Openstack CLI client]].
 
+
#* List existing ec2 credentials in your Openstack account using the <code>openstack ec2 credentials list</code> command or revoke one using <code>openstack ec2 credentials delete</code>.
# To use the S3 interface, first create ec2 credentials for your Openstack account using the `openstack ec2 credentials create` command in the [[Using_Openstack_CLI_Client | Openstack CLI client]].
+
# You can use an s3 client or SDK to access the object storage. For reference, here's a configuration file for [https://s3tools.org/s3cmd s3cmd]
#* List existing ec2 credentials in your Openstack account using the `openstack ec2 credentials list` command or revoke one using `openstack ec2 credentials delete`.
 
# You can now use an s3 client or SDK to access the object storage. Remember to specify to use AWS signature v2 if your tool defaults to signature v4. For reference, here's a configuration file for [https://s3tools.org/s3cmd s3cmd]
 
 
  [default]
 
  [default]
  access_key = a06875e7572f49e9b36c084a003f485f
+
  access_key = <<ACCESS KEY>>
  secret_key = ba7547430d0341db908d3c4e6c1e2014
+
  secret_key = <<SECRET KEY>>
 
  host_base = redcloud.cac.cornell.edu:8443
 
  host_base = redcloud.cac.cornell.edu:8443
 
  host_bucket = redcloud.cac.cornell.edu:8443
 
  host_bucket = redcloud.cac.cornell.edu:8443
Line 48: Line 52:
  
 
=='''Globus'''==
 
=='''Globus'''==
Red Cloud object storage is accessible via the '''cac#redcloud''' Globus endpoint. See [https://www.cac.cornell.edu/wiki/index.php?title=Archival_Storage#Using_Globus documentation here on Globus access]
+
Red Cloud object storage is accessible via the '''[https://app.globus.org/file-manager?origin_id=2aaf7737-4b26-4845-b53d-5790d7f804cc cac#redcloud]''' [https://globus.org Globus] collection. See [[File Transfer using Globus | documentation here on Globus access]].
  
==='''Enabling Your Red Cloud Account on the Endpoint'''===
+
==='''Access Red Cloud Object Storage via Globus'''===
  
# First create ec2 credentials for your Openstack account using the `openstack ec2 credentials create` command in the [[Using_Openstack_CLI_Client | Openstack CLI client]].
+
# First create ec2 credentials for your Openstack account using the <code>openstack ec2 credentials create</code> command in the [[Using_Openstack_CLI_Client | Openstack CLI client]].
#* List existing ec2 credentials in your Openstack account using the `openstack ec2 credentials list` command or revoke one using `openstack ec2 credentials delete`.
+
#* List existing ec2 credentials in your Openstack account using the <code>openstack ec2 credentials list</code> command or revoke one using <code>openstack ec2 credentials delete</code>.
 
#* If you have multiple Red Cloud accounts, you can only configure one account for Globus access at any given time.  
 
#* If you have multiple Red Cloud accounts, you can only configure one account for Globus access at any given time.  
# ssh to <code>globus-openstack.cac.cornell.edu</code> and log in using your CAC user name and password.
+
# Follow Globus instructions on [https://docs.globus.org/how-to/access-aws-s3/ how to access files on AWS S3 with Globus] using
# Create the <code>~/.globus</code> directory: <code>mkdir ~/.globus</code>
+
#* The '''[https://app.globus.org/file-manager?origin_id=2aaf7737-4b26-4845-b53d-5790d7f804cc cac#redcloud]''' Globus collection, and
# Create the following <code>~/.globus/s3</code> file:
+
#* The ec2 credentials you obtained using the <code>openstack ec2 credentials list</code> command in the previous step.
  <CAC user name>;<access key>;<secret key>
 

Latest revision as of 11:24, 26 August 2022

Red Cloud object storage can be accessed using the following interfaces:

Web Interface

Red Cloud object storage is accessible via [[Openstack | Openstack web interface ("Horizon")]. After logging into the web interface,

Swift

S3

  1. To use the S3 interface, first create ec2 credentials for your Openstack account using the openstack ec2 credentials create command in the Openstack CLI client.
    • List existing ec2 credentials in your Openstack account using the openstack ec2 credentials list command or revoke one using openstack ec2 credentials delete.
  2. You can use an s3 client or SDK to access the object storage. For reference, here's a configuration file for s3cmd
[default]
access_key = <<ACCESS KEY>>
secret_key = <<SECRET KEY>>
host_base = redcloud.cac.cornell.edu:8443
host_bucket = redcloud.cac.cornell.edu:8443
signature_v2 = True
bucket_location = US
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
encoding = UTF-8
encrypt = False
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/local/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
human_readable_sizes = False
list_md5 = False
preserve_attrs = True
progress_meter = True
recursive = False
recv_chunk = 4096
reduced_redundancy = False
send_chunk = 4096
skip_existing = False
socket_timeout = 300 
urlencoding_mode = normal
use_https = True
verbosity = WARNING

Globus

Red Cloud object storage is accessible via the cac#redcloud Globus collection. See documentation here on Globus access.

Access Red Cloud Object Storage via Globus

  1. First create ec2 credentials for your Openstack account using the openstack ec2 credentials create command in the Openstack CLI client.
    • List existing ec2 credentials in your Openstack account using the openstack ec2 credentials list command or revoke one using openstack ec2 credentials delete.
    • If you have multiple Red Cloud accounts, you can only configure one account for Globus access at any given time.
  2. Follow Globus instructions on how to access files on AWS S3 with Globus using
    • The cac#redcloud Globus collection, and
    • The ec2 credentials you obtained using the openstack ec2 credentials list command in the previous step.