如何打造基于s3cmd的短地址服务
这篇文章主要讲解了“如何打造基于s3cmd的短地址服务”，文中的讲解内容简单清晰，易于学习与理解，下面请大家跟着小编的思路慢慢深入，一起来研究和学习“如何打造基于s3cmd的短地址服务”吧！
为攀枝花等地区用户提供了全套网页设计制作服务，及攀枝花网站建设行业解决方案。主营业务为做网站、成都网站设计、攀枝花网站设计，以传统方式定制建设网站，并提供域名空间备案等一条龙服务，秉承以专业、用心的态度为用户提供真诚的服务。我们深信只要达到每一位用户的要求，就会得到认可，从而选择与我们长期合作。这样，我们也可以走得更远！
需求描述
有线上项目使用S3去发布移动端APP，因为需要做权限认证所以对应的Object无法开“public-read”权限，尝试使用Presign方式生成的签名URL存在以下问题
生成的签名URL地址包含敏感信息，会暴露bucket名称和accesskey，带来安全隐患。
生成的签名URL地址太长，影响用户体验。
生成的地址有效时间内可以任意访问，无法做到比较严格的反盗链。
解决方案
使用openresty实现对S3数据访问流程的封装，客户端的数据下载全部走openresty上面搭建的服务，该服务提供一次性的短地址生成服务(访问几次以后对应短地址失效)，具体流程如下:
1. 用户数据上传到S3，并设置相应Object权限为“public-read”，取得对应的对外访问URL。
2. 以之前生成的对外访问URL为基础，生成对应的短地址服务。
3. 客户端使用短地址进行访问，超出访问次数则无法访问。
如果想进一步提升安全性，可以将RGW和Openresty通过内网互联，数据上传全部走内部网络，外部访问走外网。
具体实现
1. Openresty安装
2.安装依赖lua库文件
默认openresty的lib路径为/usr/local/openresty/lualib
需要依赖两个lua文件，其中url.lua主要用于url地址的解析，redis_iresty.lua用于redis的连接
将下面的源码文件保存为 /usr/local/openresty/lualib/resty/url.lua https://github.com/golgote/neturl/blob/master/lib/net/url.lua 将下面的源码文件保存为 /usr/local/openresty/lualib/resty/redis_iresty.lua https://gist.github.com/moonbingbing/9915c66346e8fddcefb5
3. RGW服务配置
root@demohost:/etc/openresty# cat /etc/ceph/ceph.conf ... [client.radosgw.demo] rgw DNS name = s3.cephbook.com #S3 endpoint对应的域名 rgw frontends = "civetweb port=7480" host = demohost keyring = /etc/ceph/ceph.client.radosgw.keyring log file = /home/ceph/log/radosgw.log
4. Openresty配置
root@demohost:/etc/openresty# cat /etc/openresty/nginx.conf
...
server {
listen 80;
server_name ceph.vip; #短地址主机头设置
location = /url { #生成短地址入口
content_by_lua_file /etc/openresty/url.lua;
}
location / { #提供短地址对应的数据访问
proxy_http_version 1.1;
proxy_set_header Host $host;
access_by_lua_file /etc/openresty/access.lua;
proxy_pass http://127.0.0.1:7480; #对应后端的radosgw服务入口
}
}生成短地址服务的源码如下
root@demohost:/etc/openresty# cat url.lua
local request_method = ngx.var.request_method
local url_ = require "resty.url" #对应之前的url库
local redis = require "resty.redis_iresty" #对应之前的redis_iresty库
local s3_endpoint = "s3.cephbook.com" #这里设置只允许生成与S3 endpoint相关的短地址
local endpoint = s3_endpoint .. "$"
local charset = {}
for i = 48, 57 do table.insert(charset, string.char(i)) end
for i = 65, 90 do table.insert(charset, string.char(i)) end
for i = 97, 122 do table.insert(charset, string.char(i)) end
function string.random(length)
local urandom = assert(io.open('/dev/urandom','rb'))
local a, b, c, d = urandom:read(4):byte(1,4)
urandom:close()
local seed = a*0x1000000 + b*0x10000 + c *0x100 + d
math.randomseed(seed)
if length > 0 then
return string.random(length - 1) .. charset[math.random(1, #charset)]
else
return ""
end
end
function genera_url(red,url_id,host,path)
-- counts表示短地址最大访问次数，current表示当前次数
ok, err = red:hmset(url_id,'host',host,'uri',path,'counts',3,'current',1)
if not ok then
ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED
ngx.say("failed to hmset: ", err)
return ngx.exit(ngx.HTTP_METHOD_NOT_IMPLEMENTED)
end
-- 设置短地址记录最多能够在redis里面存储的时长，避免资源耗尽
ok, err = red:expire(url_id,3600)
if not ok then
ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED
ngx.say("failed to set expire: ", err)
return ngx.exit(ngx.HTTP_METHOD_NOT_IMPLEMENTED)
end
end
function get_info_by_id(red,url_id)
ok, err = red:hgetall(url_id)
if not ok then
ngx.status = ngx.HTTP_SERVICE_UNAVAILABLE
ngx.say("failed to getall : ", err)
return ngx.exit(ngx.HTTP_SERVICE_UNAVAILABLE)
end
local h = red:array_to_hash(ok)
return h
end
if request_method == "POST" then
ngx.req.read_body()
local data = ngx.req.get_body_data()
local u = url_.parse(data)
local hostname_check, hostname_err = ngx.re.match(u.host,endpoint)
if not hostname_check then
ngx.status = ngx.HTTP_BAD_REQUEST
ngx.say("not a available s3_endpoint")
ngx.exit(ngx.HTTP_BAD_REQUEST)
end
if u.host then
if string.len(u.path) > 1 then
local url_id = string.random(7)
local red = redis:new()
genera_url(red,url_id,u.host,u.path)
local h = get_info_by_id(red,url_id)
ngx.status = ngx.HTTP_CREATED
ngx.say("ShortURL: ",ngx.var.scheme,"://",ngx.var.host,"/",url_id)
ngx.say("host: ", h.host)
ngx.say("uri: ", h.uri)
ngx.say("counts: ", h.counts)
ngx.say("current: ", h.current)
ngx.exit(ngx.HTTP_CREATED)
else
ngx.status = ngx.HTTP_BAD_REQUEST
ngx.say("path err")
ngx.exit(ngx.HTTP_BAD_REQUEST)
end
else
ngx.status = ngx.HTTP_BAD_REQUEST
ngx.say("not a available url")
ngx.exit(ngx.HTTP_BAD_REQUEST)
end
end访问短地址数据服务的源码如下
root@demohost:/etc/openresty# cat /etc/openresty/access.lua
local redis = require "resty.redis_iresty"
local red = redis:new()
local uri = ngx.var.uri
local url_id = string.sub(uri,2,string.len(uri))
res, err = red:hgetall(url_id)
if not res then
ngx.exit(ngx.HTTP_NOT_FOUND)
else
local h = red:array_to_hash(res)
-- 超出访问次数则拒绝访问，如果考虑资源占用可以自己加上删除对应redis记录操作
if h.counts < h.current then
ngx.status = ngx.HTTP_GONE
ngx.say("current=",h.current," > counts=",h.counts)
ngx.exit(ngx.HTTP_GONE)
end
ngx.req.set_header("host", h.host)
ngx.req.set_uri(h.uri, false)
ok, err = red:hincrby(url_id,'current',1)
if not ok then
ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED
ngx.say("incrby key failed: ", err)
ngx.exit(ngx.HTTP_METHOD_NOT_IMPLEMENTED)
return
end
endhttps://github.com/openresty/lua-nginx-module#http-status-constants
5. 使用流程
上传数据
使用s3cmd或者是其他方式上传文件，并设置对应的Object访问权限为"public-read"。
root@demohost:/tmp# s3cmd put myfile s3://demo --acl-public 'myfile' -> 's3://demo/myfile' [1 of 1] 21577 of 21577 100% in 0s 227.71 kB/s done 'myfile' -> 's3://demo/myfile' [1 of 1] 21577 of 21577 100% in 0s 203.11 kB/s done Public URL of the object is: http://demo.s3.cephbook.com/myfile
生成短地址
root@demohost:/tmp# curl ceph.vip/url -d "http://demo.s3.cephbook.com/myfile" -v * Hostname was NOT found in DNS cache * Connected to ceph.vip port 80 (#0) > POST /url HTTP/1.1 > User-Agent: curl/7.38.0 > Host: ceph.vip > Accept: */* > Content-Length: 29 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 29 out of 29 bytes < HTTP/1.1 201 Created * Server openresty/1.11.2.5 is not blacklisted < Server: openresty/1.11.2.5 < Date: Wed, 15 Nov 2017 09:56:14 GMT < Content-Type: application/octet-stream < Transfer-Encoding: chunked < Connection: keep-alive < ShortURL: http://ceph.vip/Vo3t5Ex #生成的短地址 host: demo.s3.cephbook.com uri: /myfile counts: 3 current: 1 * Connection #0 to host ceph.vip left intact
测试访问
#访问3次 root@demohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v ... root@demohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v ... root@demohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v ... #第四次以后就不能访问了 root@demohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v * Hostname was NOT found in DNS cache * Trying ceph.vip... * Connected to ceph.vip (ceph.vip) port 80 (#0) > GET /Vo3t5Ex HTTP/1.1 > User-Agent: curl/7.38.0 > Host: ceph.vip > Accept: */* > < HTTP/1.1 410 Gone * Server openresty/1.11.2.5 is not blacklisted < Server: openresty/1.11.2.5 < Date: Wed, 15 Nov 2017 10:00:46 GMT < Content-Type: application/octet-stream < Transfer-Encoding: chunked < Connection: keep-alive < current=4 > counts=3 * Connection #0 to host ceph.vip left intact
感谢各位的阅读，以上就是“如何打造基于s3cmd的短地址服务”的内容了，经过本文的学习后，相信大家对如何打造基于s3cmd的短地址服务这一问题有了更深刻的体会，具体使用情况还需要大家实践验证。这里是创新互联，小编将为大家推送更多相关知识点的文章，欢迎关注！
当前标题：如何打造基于s3cmd的短地址服务
文章分享：http://scyanting.com/article/pccpjo.html
