amazon s3 - Getting started with secure AWS CloudFront streaming with Python -

i have created s3 bucket, uploaded video, created streaming distribution in cloudfront. tested static html player , works. have created keypair through account settings. have private key file sitting on desktop @ moment. that's am.

my aim point django/python site creates secure urls , people can't access videos unless they've come 1 of pages. problem i'm allergic way amazon have laid things out , i'm getting more , more confused.

i realise isn't going best question on stackoverflow i'm can't fool out here can't make heads or tails out of how set secure cloudfront/s3 situation. appreciate , willing (once 2 days has passed) give 500pt bounty best answer.

i have several questions that, once answered, should fit 1 explanation of how accomplish i'm after:

• in documentation (there's example in next point) there's lots of xml lying around telling me need post things various places. there online console doing this? or literally have force via curl (et al)?

• how create origin access identity cloudfront , bind distribution? i've read this document but, per first point, don't know it. how keypair fit this?

• once that's done, how limit s3 bucket allow people download things through identity? if xml jobby rather clicking around web ui, please tell me , how i'm supposed account.

• in python, what's easiest way of generating expiring url file. have boto installed don't see how file streaming distribution.

• are there applications or scripts can take difficulty of setting garb up? use ubuntu (linux) have xp in vm if it's windows-only. i've looked @ cloudberry s3 explorer pro - makes sense online ui.

you're right, takes lot of api work set up. hope in aws console soon!

update: have submitted code boto - of boto v2.1 (released 2011-10-27) gets easier. boto < 2.1, use instructions here. boto 2.1 or greater, updated instructions on blog: http://www.secretmike.com/2011/10/aws-cloudfront-secure-streaming.html once boto v2.1 gets packaged more distros i'll update answer here.

to accomplish want need perform following steps detail below:

1. create s3 bucket , upload objects (you've done this)
2. create cloudfront "origin access identity" (basically aws account allow cloudfront access s3 bucket)
3. modify acls on objects cloudfront origin access identity allowed read them (this prevents people bypassing cloudfront , going direct s3)
4. create cloudfront distribution basic urls , 1 requires signed urls
5. test can download objects basic cloudfront distribution not s3 or signed cloudfront distribution
6. create key pair signing urls
7. generate urls using python
8. test signed urls work

---

1 - create bucket , upload object

the easiest way through aws console completeness i'll show how using boto. boto code shown here:

import boto  #credentials stored in environment aws_access_key_id , aws_secret_access_key s3 = boto.connect_s3()  #bucket name must follow dns guidelines new_bucket_name = "stream.example.com" bucket = s3.create_bucket(new_bucket_name)  object_name = "video.mp4" key = bucket.new_key(object_name) key.set_contents_from_filename(object_name) 

2 - create cloudfront "origin access identity"

for now, step can performed using api. boto code here:

import boto  #credentials stored in environment aws_access_key_id , aws_secret_access_key cf = boto.connect_cloudfront()  oai = cf.create_origin_access_identity(comment='new identity secure videos')  #we need following 2 values later steps: print("origin access identity id: %s" % oai.id) print("origin access identity s3canonicaluserid: %s" % oai.s3_user_id) 

3 - modify acls on objects

now we've got our special s3 user account (the s3canonicaluserid created above) need give access our s3 objects. can using aws console opening object's (not bucket's!) permissions tab, click "add more permissions" button, , pasting long s3canonicaluserid got above "grantee" field of new. make sure give new permission "open/download" rights.

you can in code using following boto script:

import boto  #credentials stored in environment aws_access_key_id , aws_secret_access_key s3 = boto.connect_s3()  bucket_name = "stream.example.com" bucket = s3.get_bucket(bucket_name)  object_name = "video.mp4" key = bucket.get_key(object_name)  #now add read permission our new s3 account s3_canonical_user_id = "<your s3canonicaluserid above>" key.add_user_grant("read", s3_canonical_user_id) 

4 - create cloudfront distribution

note custom origins , private distributions not supported in boto until version 2.0 has not been formally released @ time of writing. code below pulls out code boto 2.0 branch , hacks going it's not pretty. 2.0 branch handles more elegantly - use if possible!

import boto boto.cloudfront.distribution import distributionconfig boto.cloudfront.exception import cloudfrontservererror  import re  def get_domain_from_xml(xml):     results = re.findall("<domainname>([^<]+)</domainname>", xml)     return results[0]  #custom class hack until boto v2.0 released class hackedstreamingdistributionconfig(distributionconfig):      def __init__(self, connection=none, origin='', enabled=false,                  caller_reference='', cnames=none, comment='',                  trusted_signers=none):         distributionconfig.__init__(self, connection=connection,                                     origin=origin, enabled=enabled,                                     caller_reference=caller_reference,                                     cnames=cnames, comment=comment,                                     trusted_signers=trusted_signers)      #override to_xml() function     def to_xml(self):         s = '<?xml version="1.0" encoding="utf-8"?>\n'         s += '<streamingdistributionconfig xmlns="http://cloudfront.amazonaws.com/doc/2010-07-15/">\n'          s += '  <s3origin>\n'         s += '    <dnsname>%s</dnsname>\n' % self.origin         if self.origin_access_identity:             val = self.origin_access_identity             s += '    <originaccessidentity>origin-access-identity/cloudfront/%s</originaccessidentity>\n' % val         s += '  </s3origin>\n'           s += '  <callerreference>%s</callerreference>\n' % self.caller_reference         cname in self.cnames:             s += '  <cname>%s</cname>\n' % cname         if self.comment:             s += '  <comment>%s</comment>\n' % self.comment         s += '  <enabled>'         if self.enabled:             s += 'true'         else:             s += 'false'         s += '</enabled>\n'         if self.trusted_signers:             s += '<trustedsigners>\n'             signer in self.trusted_signers:                 if signer == 'self':                     s += '  <self/>\n'                 else:                     s += '  <awsaccountnumber>%s</awsaccountnumber>\n' % signer             s += '</trustedsigners>\n'         if self.logging:             s += '<logging>\n'             s += '  <bucket>%s</bucket>\n' % self.logging.bucket             s += '  <prefix>%s</prefix>\n' % self.logging.prefix             s += '</logging>\n'         s += '</streamingdistributionconfig>\n'          return s      def create(self):         response = self.connection.make_request('post',             '/%s/%s' % ("2010-11-01", "streaming-distribution"),             {'content-type' : 'text/xml'},             data=self.to_xml())          body = response.read()         if response.status == 201:             return body         else:             raise cloudfrontservererror(response.status, response.reason, body)   cf = boto.connect_cloudfront()  s3_dns_name = "stream.example.com.s3.amazonaws.com" comment = "example streaming distribution" oai = "<oai id step 2 above e23krhs6gduf5l>"  #create distribution not need signed urls hsd = hackedstreamingdistributionconfig(connection=cf, origin=s3_dns_name, comment=comment, enabled=true) hsd.origin_access_identity = oai basic_dist = hsd.create() print("distribution basic urls: %s" % get_domain_from_xml(basic_dist))  #create distribution need signed urls hsd = hackedstreamingdistributionconfig(connection=cf, origin=s3_dns_name, comment=comment, enabled=true) hsd.origin_access_identity = oai #add required signers (self means own account) hsd.trusted_signers = ['self'] signed_dist = hsd.create() print("distribution signed urls: %s" % get_domain_from_xml(signed_dist)) 

5 - test can download objects cloudfront not s3

you should able verify:

• stream.example.com.s3.amazonaws.com/video.mp4 - should give accessdenied
• signed_distribution.cloudfront.net/video.mp4 - should give missingkey (because url not signed)
• basic_distribution.cloudfront.net/video.mp4 - should work fine

the tests have adjusted work stream player, basic idea basic cloudfront url should work.

6 - create keypair cloudfront

i think way through amazon's web site. go aws "account" page , click on "security credentials" link. click on "key pairs" tab click "create new key pair". generate new key pair , automatically download private key file (pk-xxxxxxxxx.pem). keep key file safe , private. note down "key pair id" amazon need in next step.

7 - generate urls in python

as of boto version 2.0 there not seem support generating signed cloudfront urls. python not include rsa encryption routines in standard library have use additional library. i've used m2crypto in example.

for non-streaming distribution, must use full cloudfront url resource, streaming use object name of video file. see code below full example of generating url lasts 5 minutes.

this code based loosely on php example code provided amazon in cloudfront documentation.

from m2crypto import evp import base64 import time  def aws_url_base64_encode(msg):     msg_base64 = base64.b64encode(msg)     msg_base64 = msg_base64.replace('+', '-')     msg_base64 = msg_base64.replace('=', '_')     msg_base64 = msg_base64.replace('/', '~')     return msg_base64  def sign_string(message, priv_key_string):     key = evp.load_key_string(priv_key_string)     key.reset_context(md='sha1')     key.sign_init()     key.sign_update(str(message))     signature = key.sign_final()     return signature  def create_url(url, encoded_signature, key_pair_id, expires):     signed_url = "%(url)s?expires=%(expires)s&signature=%(encoded_signature)s&key-pair-id=%(key_pair_id)s" % {             'url':url,             'expires':expires,             'encoded_signature':encoded_signature,             'key_pair_id':key_pair_id,             }     return signed_url  def get_canned_policy_url(url, priv_key_string, key_pair_id, expires):     #we manually construct policy string ensure formatting matches signature     canned_policy = '{"statement":[{"resource":"%(url)s","condition":{"datelessthan":{"aws:epochtime":%(expires)s}}}]}' % {'url':url, 'expires':expires}      #now base64 encode (must url safe)     encoded_policy = aws_url_base64_encode(canned_policy)     #sign non-encoded policy     signature = sign_string(canned_policy, priv_key_string)     #now base64 encode signature (url safe well)     encoded_signature = aws_url_base64_encode(signature)      #combine these full url     signed_url = create_url(url, encoded_signature, key_pair_id, expires);      return signed_url  def encode_query_param(resource):     enc = resource     enc = enc.replace('?', '%3f')     enc = enc.replace('=', '%3d')     enc = enc.replace('&', '%26')     return enc   #set parameters url key_pair_id = "apkaiazczrkvio4bq" #from aws accounts page priv_key_file = "cloudfront-pk.pem" #your private keypair file resource = 'video.mp4' #your resource (just object name streaming videos) expires = int(time.time()) + 300 #5 min  #create signed url priv_key_string = open(priv_key_file).read() signed_url = get_canned_policy_url(resource, priv_key_string, key_pair_id, expires)  #flash player doesn't query params encode them enc_url = encode_query_param(signed_url) print(enc_url) 

8 - try out urls

hopefully should have working url looks this:

video.mp4%3fexpires%3d1309979985%26signature%3dmunf7pw1689fhmesn6jzqmwnvxcaie9mk1x~koudjky7antux0oagl~1gw-on6zh5nflboocx3fuhmc9fusahtjuzwyjvzlzyt9ilyoyfwmsm2ylcdbqpy5iynfbi8cuajd~cjydxzbwpxtspo3yifnji~r2afpwx8qp3fs38yw_%26key-pair-id%3dapkaiazrkvio4bq 

put js , should have looks (from php example in amazon's cloudfront documentation):

var so_canned = new swfobject('http://location.domname.com/~jvngkhow/player.swf','mpl','640','360','9');     so_canned.addparam('allowfullscreen','true');     so_canned.addparam('allowscriptaccess','always');     so_canned.addparam('wmode','opaque');     so_canned.addvariable('file','video.mp4%3fexpires%3d1309979985%26signature%3dmunf7pw1689fhmesn6jzqmwnvxcaie9mk1x~koudjky7antux0oagl~1gw-on6zh5nflboocx3fuhmc9fusahtjuzwyjvzlzyt9ilyoyfwmsm2ylcdbqpy5iynfbi8cuajd~cjydxzbwpxtspo3yifnji~r2afpwx8qp3fs38yw_%26key-pair-id%3dapkaiazrkvio4bq');     so_canned.addvariable('streamer','rtmp://s3nzpoyjpct.cloudfront.net/cfx/st');     so_canned.write('canned'); 

---

summary

as can see, not easy! boto v2 lot setting distribution. find out if it's possible url generation code in there improve great library!