Sharing

2012年10月5日 星期五

Openstack Folsom - Swift Installation

Swift Installation


http://docs.openstack.org/developer/swift/howto_installmultinode.html#config-proxy
http://docs.openstack.org/trunk/openstack-compute/install/apt/content/ch_installing-openstack-object-storage.html

Add Repository


先加入 Openstack Folsom 的 PPA 路徑

root@swift-proxy:~$ apt-get install -y python-software-properties
root@swift-proxy:~$ add-apt-repository ppa:openstack-ubuntu-testing/folsom-trunk-testing
root@swift-proxy:~$ add-apt-repository ppa:openstack-ubuntu-testing/folsom-deps-staging
root@swift-proxy:~$ apt-get update && apt-get -y dist-upgrade

Basic Package Installation


在 proxy-server 及所有的 storage server 都要裝這些 package, 並且 swift.conf 內的 hash 值要相同
root@swift-proxy:~$ apt-get install swift openssh-server rsync memcached python-netifaces python-xattr python-memcache python-swiftclient
root@swift-proxy:~$ mkdir -p /etc/swift
root@swift-proxy:~$ chown -R swift:swift /etc/swift/
root@swift-proxy:~$ cat swift.conf
[swift-hash]
# random unique string that can never change (DO NOT LOSE). I’m using 03c9f48da2229770.
# od -t x8 -N 8 -A n < /dev/random
# The above command can be used to generate random a string.
swift_hash_path_suffix = 34c486c41efd7f62
root@swift-storage:/etc/swift$ dpkg -l | grep swift
ii  python-swift                                    1.7.1+git201209042100~precise-0ubuntu1             distributed virtual object store - Python libraries
ii  python-swiftclient                              1:1.2.0.6.a99a37f+git201210020230~precise-0ubuntu1 Client libary for Openstack Swift API.
ii  swift                                           1.7.1+git201209042100~precise-0ubuntu1             distributed virtual object store - common files

Storage Server Installation

root@swift-storage:~$ apt-get install swift-account swift-container swift-object xfsprogs
root@swift-storage:~$ dpkg -l | grep swift
ii  swift-account                                   1.7.1+git201209042100~precise-0ubuntu1             distributed virtual object store - account server
ii  swift-container                                 1.7.1+git201209042100~precise-0ubuntu1             distributed virtual object store - container server
ii  swift-object                                    1.7.1+git201209042100~precise-0ubuntu1             distributed virtual object store - object server
設定 partition
root@swift-storage:~$ mkfs.xfs -i size=1024 /dev/vda3
meta-data=/dev/vda3              isize=1024   agcount=4, agsize=2949120 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=11796480, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=5760, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@swift-storage:~$ echo "/dev/vda3 /srv/node/vda3 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0" >> /etc/fstab
root@swift-storage:~$ mkdir -p /srv/node/vda3
root@swift-storage:~$ mount /srv/node/vda3
root@swift-storage:~$ chown -R swift:swift /srv/node
root@swift-storage:~$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/vda2        3094288 1592280   1344824  55% /
udev             4080360       4   4080356   1% /dev
tmpfs            1635780     252   1635528   1% /run
none                5120       0      5120   0% /run/lock
none             4089444       0   4089444   0% /run/shm
/dev/vda3       47162880   32976  47129904   1% /srv/node/vda3
設定rsync
# 設定 /etc/rsyncd.conf , 記得 address 要改一下
root@swift:~$ cat /etc/rsyncd.conf
uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 172.17.123.15

[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock

[container]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/container.lock

[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock

# 把 RSYNC_ENABLE 改成 true
root@swift:~$ head /etc/default/rsync
# defaults file for rsync daemon mode
# start rsync in daemon mode from init.d script?
#  only allowed values are "true", "false", and "inetd"
#  Use "inetd" if you want to start the rsyncd from inetd,
#  all this does is prevent the init.d script from printing a message
#  about not starting rsyncd (you still need to modify inetd's config yourself).
RSYNC_ENABLE=true

root@swift:~$ service rsync restart
 * Restarting rsync daemon rsync
 * rsync daemon not running, attempting to start.
   ...done.
裝完後會在 /etc/swift 下看到三個 .conf 檔, account-server.conf、container-server.conf、object-server.conf, 但在裝的過程中, 發現他還需要一個 object-expirer.conf, 目前不確定這個檔案一定要裝, 但至少我加上去也沒有造成什麼問題, 然後 container-server.conf 預設的內容也有少, 後來我是直接去看 git 內的 container-server.conf-sample 才知道的.

account-server.conf 的內容

[DEFAULT]
bind_ip = 0.0.0.0
workers = 2

[pipeline:main]
pipeline = account-server

[app:account-server]
use = egg:swift#account

[account-replicator]

[account-auditor]

[account-reaper]

container-server.conf 的內容

記得要多加 container-sync
[DEFAULT]
bind_ip = 0.0.0.0
workers = 2

[pipeline:main]
pipeline = container-server

[app:container-server]
use = egg:swift#container

[container-replicator]

[container-updater]

[container-auditor]

[container-sync]

object-server.conf 的內容

[DEFAULT]
bind_ip = 0.0.0.0
workers = 2

[pipeline:main]
pipeline = object-server

[app:object-server]
use = egg:swift#object

[object-replicator]

[object-updater]

[object-auditor]

container-expirer.conf 的內容

[DEFAULT]

[object-expirer]
interval = 300

[pipeline:main]
pipeline = catch_errors cache proxy-server

[app:proxy-server]
use = egg:swift#proxy

[filter:cache]
use = egg:swift#memcache

[filter:catch_errors]
use = egg:swift#catch_errors

Proxy Server Installation

root@swift-proxy:~$ apt-get install swift-proxy memcached python-keystoneclient keystone
root@swift-proxy:~$ dpkg -l | grep swift
ii  python-swift                                    1.7.1+git201209042100~precise-0ubuntu1 distributed virtual object store - Python libraries
ii  swift                                           1.7.1+git201209042100~precise-0ubuntu1 distributed virtual object store - common files
ii  swift-proxy                                     1.7.1+git201209042100~precise-0ubuntu1 distributed virtual object store - proxy server
root@swift-proxy:~$ dpkg -l | grep keystone
ii  python-keystone                                 2012.2+git201209252030~precise-0ubuntu1            OpenStack identity service - Python library
ii  python-keystoneclient                           1:0.1.3.19+git201210011900~precise-0ubuntu1        Client libary for Openstack Keystone API
在 /etc/swift 下生成 certificate
root@swift-proxy:~$ cd /etc/swift/
root@swift-proxy:/etc/swift$ openssl req -new -x509 -nodes -out cert.crt -keyout cert.key
Generating a 1024 bit RSA private key
.....++++++
..........................++++++
writing new private key to 'cert.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:TW
State or Province Name (full name) [Some-State]:Taiwan
Locality Name (eg, city) []:Taipei
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:
設定 memcache
root@swift:~$ cat /etc/memcached.conf | grep -B 3 0.0.0.0
# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 172.17.123.14
root@swift:~$ service memcached restart
然後設定 proxy-server 的內容, 因為前一篇我在 keystone 內設定 swift 的 port 是 8080, 所以我在這裡也要設定成 8080. 另外 [filter:keystoneauth]、[filter:authtoken] 的內容, 我都直接參考 git 內的 sample 來設定, 按照官網上的設定方式似乎會有問題. 另外我會在 /etc/hosts 先加上 keystone 的 ip, 所以在設定檔內的 service_host 我填的是 hostname, 而不是 ip
[DEFAULT]
bind_port = 8080
user = swift

[pipeline:main]
pipeline = catch_errors healthcheck cache authtoken keystoneauth proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
#paste.filter_factory = keystone.middleware.swift_auth:filter_factory
operator_roles = Member,admin,swiftoperator

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_port = 5000
service_host = keystone
auth_port = 35357
auth_host = keystone
auth_protocol = http
admin_tenant_name = service
admin_user = swift
admin_password = password
signing_dir = /etc/swift

[filter:cache]
use = egg:swift#memcache
set log_name = cache
#memcache_servers = 172.17.123.14:11211

[filter:catch_errors]
use = egg:swift#catch_errors

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:proxy-logging]
use = egg:swift#proxy_logging

[filter:ratelimit]
use = egg:swift#ratelimit
設定 ring, 我的 storage server ip 為 172.17.123.15, 剛剛加入的 partition 是 vda3, 我把它設定到 z1 去, 如果要加入更多其它 partition, 就必須要多執行幾次 add 的動作, 而且要規劃好放在那一個 zone, 可以參考以下兩份文件.
http://cssoss.files.wordpress.com/2012/05/openstackbookv3-0_csscorp2.pdf
http://www.hastexo.com/resources/docs/installing-openstack-essex-20121-ubuntu-1204-precise-pangolin/appendix-c-setting-op-1
root@swift:~$ cd /etc/swift/
root@swift:/etc/swift$ swift-ring-builder account.builder create 18 3 1
root@swift:/etc/swift$ swift-ring-builder container.builder create 18 3 1
root@swift:/etc/swift$ swift-ring-builder object.builder create 18 3 1
root@swift:/etc/swift$ swift-ring-builder account.builder add z1-172.17.123.15:6002/vda3 100
Device z1-172.17.123.15:6002/vda3_"" with 100.0 weight got id 0
root@swift:/etc/swift$ swift-ring-builder container.builder add z1-172.17.123.15:6001/vda3 100
Device z1-172.17.123.15:6001/vda3_"" with 100.0 weight got id 0
root@swift:/etc/swift$ swift-ring-builder object.builder add z1-172.17.123.15:6000/vda3 100
Device z1-172.17.123.15:6000/vda3_"" with 100.0 weight got id 0
root@swift:/etc/swift$ swift-ring-builder account.builder
account.builder, build version 1
262144 partitions, 3 replicas, 1 zones, 1 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  zone      ip address  port      name weight partitions balance meta
             0     1   172.17.123.15  6002      vda3 100.00          0 -100.00
root@swift:/etc/swift$ swift-ring-builder container.builder
container.builder, build version 1
262144 partitions, 3 replicas, 1 zones, 1 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  zone      ip address  port      name weight partitions balance meta
             0     1   172.17.123.15  6001      vda3 100.00          0 -100.00
root@swift:/etc/swift$ swift-ring-builder object.builder
object.builder, build version 1
262144 partitions, 3 replicas, 1 zones, 1 devices, 100.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  zone      ip address  port      name weight partitions balance meta
             0     1   172.17.123.15  6000      vda3 100.00          0 -100.00
root@swift-proxy:/etc/swift$ swift-ring-builder account.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 0.00.
root@swift-proxy:/etc/swift$ swift-ring-builder container.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 0.00.
root@swift-proxy:/etc/swift$ swift-ring-builder object.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 0.00.
root@swift-proxy:/etc/swift$ chown -R swift:swift /etc/swift
做完 rebalance 之後會產生三個 ring.gz 檔案, 記得把這些檔案放到所有的 storage server 的 /etc/swift 下, 而且傳過去之後, 記得要把 owner 改成 swift:swift

Startup Service

在 proxy server 下
root@swift-proxy:~$ swift-init proxy start
Starting proxy-server...(/etc/swift/proxy-server.conf)
在 storage server 下
root@swift-storage:~$ swift-init all start
Starting container-updater...(/etc/swift/container-server.conf)
Starting account-auditor...(/etc/swift/account-server.conf)
Starting object-replicator...(/etc/swift/object-server.conf)
Unable to locate config for proxy-server
Starting container-replicator...(/etc/swift/container-server.conf)
Starting object-auditor...(/etc/swift/object-server.conf)
Starting object-expirer...(/etc/swift/object-expirer.conf)
Starting container-auditor...(/etc/swift/container-server.conf)
Starting container-server...(/etc/swift/container-server.conf)
Starting account-server...(/etc/swift/account-server.conf)
Starting account-reaper...(/etc/swift/account-server.conf)
Starting container-sync...(/etc/swift/container-server.conf)
Starting account-replicator...(/etc/swift/account-server.conf)
Starting object-updater...(/etc/swift/object-server.conf)
Starting object-server...(/etc/swift/object-server.conf)

Test and Verify

編輯一個檔, 設定一些等下會用到的區域變數, 並且匯到 .bashrc 之中, 下次再進來就不需要重新設定
root@swift-proxy:~$ cat novarc
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password
export OS_AUTH_URL="http://keystone:5000/v2.0/"
export SERVICE_ENDPOINT="http://keystone:35357/v2.0"
export SERVICE_TOKEN=password
root@swift-proxy:~$ source novarc
root@swift-proxy:~$ echo "source novarc">>.bashrc
另外在 keystone 的 endpoint-list 也請記得修改一下 ip, 改成 proxy server 的 ip, 不要再設成 127.0.0.1, 除非你是裝在同一台
root@swift-proxy:~$ keystone endpoint-list
+----------------------------------+-----------+-----------------------------------------------+-----------------------------------------------+----------------------------------------+
|                id                |   region  |                   publicurl                   |                  internalurl                  |                adminurl                |
+----------------------------------+-----------+-----------------------------------------------+-----------------------------------------------+----------------------------------------+
| 580b71d126804c5197b91c79fd74a330 | RegionOne |           http://keystone:5000/v2.0           |           http://keystone:5000/v2.0           |       http://keystone:35357/v2.0       |
| 6c788747593d475f831b6ff128bde995 | RegionOne |      http://cinder:8776/v1/$(tenant_id)s      |      http://cinder:8776/v1/$(tenant_id)s      |  http://cinder:8776/v1/$(tenant_id)s   |
| 95e16e71a8f04ac68ae401df5284ce3e | RegionOne | http://swift-proxy:8080/v1/AUTH_$(tenant_id)s | http://swift-proxy:8080/v1/AUTH_$(tenant_id)s |       http://swift-proxy:8080/v1       |
+----------------------------------+-----------+-----------------------------------------------+-----------------------------------------------+----------------------------------------+
測試的時間到了
http://docs.openstack.org/trunk/openstack-compute/install/apt/content/verify-swift-installation.html
root@swift-proxy:~$ swift list
root@swift-proxy:~$ swift post test
root@swift-proxy:~$ swift list
test
root@swift-proxy:~$ swift upload test /etc/motd
etc/motd
root@swift-proxy:~$ swift list test
etc/motd
root@swift-proxy:~$ swift stat
   Account: AUTH_eefa301a6a424e7da3d582649ad0e59e
Containers: 1
   Objects: 1
     Bytes: 451
Accept-Ranges: bytes
X-Timestamp: 1349422137.92607
X-Trans-Id: tx251024dd19464f55b2945092b6f3678a

What is Swift?

Mirantis 有一篇介紹 Swift 及 Ceph, 但我覺得他對於 Swift 的簡介十分的易懂明瞭, 對於第一次接觸的人來說, 是個很棒的入門篇
http://www.mirantis.com/blog/object-storage-openstack-cloud-swift-ceph/

再深入一點的文章, 這篇講解 Ring 的概念, 並且也點出了 Swift 的不足之處
http://julien.danjou.info/blog/2012/openstack-swift-consistency-analysis
http://www.mirantis.com/blog/under-the-hood-of-swift-the-ring/

這篇就比較純文字一點, 先看過前面幾篇再來看這篇會比較適合
http://programmerthoughts.com/openstack/swift-tech-overview/
他也從 CAP 理論的角度來探討 Swift 的特性是滿足 AP (Available + Partition Tolerance), 所以在 Consistency 方面會放鬆一些, 也就是偶而會拿到不一致的資料
Swift achieves high scalability by relaxing constraints on consistency. While swift provides read-your-writes consistency for new objects, listings and aggregate metadata (like usage information) may not be immediately accurate. Similarly, reading an object that has been overwritten with new data may return an older version of the object data. However, swift provides the ability for the client to request the most up-to-date version at the cost of request latency. 







沒有留言: