tag:blogger.com,1999:blog-76306268130035699742024-03-15T00:19:04.821+08:00Tech Note技术文档备忘&分享iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.comBlogger55125tag:blogger.com,1999:blog-7630626813003569974.post-84684728619710199972021-08-23T22:13:00.001+08:002021-08-23T22:13:56.074+08:00Temporary Post Used For Theme Detection (f90738ca-3bec-47c3-9f6b-8aaebb41fd01 - 3bfe001a-32de-4114-a6b4-4005b770f6d7)<p>This is a temporary post that was not deleted. Please delete this manually. (d73c9843-87b3-4331-9e1e-5ed193cc8e3f - 3bfe001a-32de-4114-a6b4-4005b770f6d7)</p>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-82458541652345402002020-03-29T22:37:00.001+08:002020-05-23T22:53:43.386+08:00Bitwarden 配置 Let’s Encrypt 证书<p style="text-indent: 2em;">由于在内网环境部署的Bitwarden,不能通过安装脚本自动申请<a href="https://letsencrypt.org">Let’s Encrypt</a> 的证书。</p> <p style="text-indent: 2em;">起初我用了一个自签名证书,发现特别不好使,虽然在firefox 跟 chrome 内核的浏览器上都能通过插件访问,但必须添加信任才行,特别是用Bitwarden Desktop 客户端连的时候,一直报错,可能是不认自签名证书。 于是琢磨了下怎么手动申请Let’s Encrypt的证书用于内网,对过程简单梳理如下。 </p> <h2>自签名证书 | Self-Signed Certificate</h2> <p><img alt="自签名证书浏览器状态" border="0" height="357" src="https://drive.google.com/uc?id=1BKB_vOReFMpRqC8YABAUW6IkldkXiUBm" style="background-image: none; display: inline;" title="自签名证书浏览器状态" width="644" /></p> <p style="text-indent: 2em;">如果你在内网环境只需要自签名证书,可以参考Bitwarden的帮助文档(<a href="https://bitwarden.com/help/article/install-on-premise/" target="_blank">Installing and depoying</a>)进行设置:</p> <p style="text-indent: 2em;">1、生成自签名证书: </p> <blockquote> <p>//一条命创建私钥和证书: <br />openssl req -x509 -newkey rsa:4096 -sha256 –nodes -days 10950 \ <br /> -keyout <font color="#0080ff">privite.key </font>-out<font color="#0080ff"> identity.crt</font> \ <br /> -subj "/C=<font color="#0080ff">US</font>/ST=<font color="#0080ff">New York</font>/L=<font color="#0080ff">New York</font>/O=<font color="#0080ff">Company Name</font>/OU=<font color="#0080ff">CREAST</font> /CN=<font color="#0080ff">bw.creast.win</font>" </p> </blockquote> <p style="text-indent: 2em;">将生成的文件放到 /ssl 目录下: </p> <blockquote> <p style="text-indent: 2em;"><font color="#0080ff">privite.key </font> ~/bwdata/ssl/<font color="#0080ff">bw.creast.win/private.key</font></p> <p style="text-indent: 2em;"><font color="#0080ff">identity.crt</font> ~/bwdata/ssl/<font color="#0080ff">bw.creast.win/identity.crt</font> </p> </blockquote> <p style="text-indent: 2em;">2、生成.pfx格式证书文件: </p> <blockquote> <p>//一条命令将前面生成的私钥和证书打包成需要的pfx格式文件 </p> <p>openssl pkcs12 -export -out ./<font color="#0080ff">identity.pfx</font> -inkey <font color="#0080ff">privite.key</font> \ <br /> -in <font color="#0080ff">identity.crt</font> -certfile <font color="#0080ff">identity.crt</font> -passout pass:<font color="#0080ff">IDENTITY_CERT_PASSWORD</font> </p> <p>注意 <font color="#0080ff">IDENTITY_CERT_PASSWORD</font> 跟./env/global.override.env 配置内容保持一致;</p> </blockquote> <p style="text-indent: 2em;">执行成功后将生成的 文件,放到 /identity 目录下:</p> <blockquote> <p style="text-indent: 2em;"><font color="#0080ff">identity.pfx</font> ~/bwdata/identity/<font color="#0080ff">identity.pfx <font color="#0080ff"> </font></font></p> </blockquote> <p style="text-indent: 2em;">以上操作直接在部署Bitwardern 的机器上执行。</p> <h2>证书申请前提条件</h2> <ol> <li> <div style="text-indent: 2em;">有公网DNS解析正常的域名 (用于通过Let's Encrypt 证书申请的验证)</div> </li> <li> <div style="text-indent: 2em;">该域名指向公网上的一台能正常响应的主机(能ping通,该主机上不要求运行任何服务)</div> </li> <li> <div style="text-indent: 2em;">一台能访问外网的机器,安装Let's Encrypt 的命令行工具Certbot ,以ubuntu 为例:</div> </li> <blockquote> <p>$ sudo apt-get install certbot </p> </blockquote> </ol> <p style="text-indent: 2em;"><font color="#0080ff"> <font color="#0080ff"> </font></font></p> <h2>申请 Let's Encrypt 证书</h2> <p><img alt="Let's Encrypt证书浏览器状态" border="0" height="365" src="https://drive.google.com/uc?id=1FVOa0qBGbWYOAcfpEYFDt-vh9JvceScp" style="background-image: none; display: inline;" title="Let's Encrypt证书浏览器状态" width="644" /></p> <p style="text-indent: 2em;">关于 certbot 的功能跟原理这里不多讲,这里重点说一下如何以 manual 方式申请证书。以下操作步骤在一台能访问外网的机器上执行,如果你部署bitwarden的机器可以访问外网可以直接在上面执行:</p> <p style="text-indent: 2em;">1、选择以dns认证方式申请证书:</p> <blockquote> <p>sudo certbot certonly --manual --preferred-challenges dns</p> </blockquote> <p>填写完基本信息后会要求你在dns记录里添加一条指定内容的的 TXT 记录,如:</p> <blockquote> <p>_acme-challenge.<font color="#0080ff">bw.creast.win.</font> 300 IN TXT "<font color="#0080ff">gfj9Xqa2gVbHjl….FtkDRg85nM</font>"</p> </blockquote> <p style="text-indent: 2em;">2、登录你的域名服务商后台,添加 TXT 记录:</p> <p><img alt="DNS TXT Record" border="0" height="124" src="https://drive.google.com/uc?id=1eOcn8sQyAB1IJrF5zC47u7q6WZbIGN0-" style="background-image: none; display: inline;" title="DNS TXT Record" width="809" /> </p> <p style="text-indent: 2em;">3、验证TXT DNS 记录</p> <p>在回车确认前,先用dig 检查一下dns 记录是否已经生效:</p> <blockquote> <p>dig -t txt _acme-challenge.<font color="#0080ff">bw.creast.win</font> <br />确认;; ANSWER SECTION: 部分跟验证指定的内容一致后再回车;</p> </blockquote> <p style="text-indent: 2em;">4、成功生成证书</p> <p>待 certbot 程序验证通过后会生成多个pem格式的证书相关文件,并存放在 /etc/letsencrypt/live/ 目录下,分别是:</p> <table border="1" cellpadding="1" cellspacing="0" style="width: 787px;"><tbody> <tr> <td valign="top" width="141">let’s Encrypt 文件</td> <td valign="top" width="464"> 文件说明</td> <td valign="top" width="180"> 对应nginx ssl 设置</td> </tr> <tr> <td valign="top" width="141">`privkey.pem`</td> <td valign="top" width="464">the private key for your certificate</td> <td valign="top" width="180">ssl_certificate_key</td> </tr> <tr> <td valign="top" width="141">`fullchain.pem`</td> <td valign="top" width="464">the certificate file used in most server software.</td> <td valign="top" width="180">ssl_certificate </td> </tr> <tr> <td valign="top" width="141">`chain.pem`</td> <td valign="top" width="464">used for OCSP stapling in Nginx >=1.3.7.</td> <td valign="top" width="180"> </td> </tr> <tr> <td valign="top" width="141">`cert.pem`</td> <td valign="top" width="464"> <p>will break many server configurations, and should not be used without reading further documentation</p> </td> <td valign="top" width="180"> </td> </tr> </tbody></table> <p>这里 privkey.pem 跟 fullchain.pem 是需要用到的私钥和证书文件,将他们复制一份到部署Bitwarden机器 bwdata/ssl/<font color="#0080ff">bw.creast.win</font> 目录下</p> <p style="text-indent: 2em;">5、生成.pfx格式证书文件:</p> <blockquote> <p>//同自签名证书一样,命令中的私钥和证书文件将使用通过 certbot 申请获得的: <br />openssl pkcs12 -export -out ./<font color="#0080ff">identity.pfx</font> -inkey <font color="#0080ff">privkey.pem</font> \ <br /> -in <font color="#0080ff">fullchain</font><font color="#0080ff">.pem</font> -certfile <font color="#0080ff">fullchain.pem</font> -passout pass:<font color="#0080ff">IDENTITY_CERT_PASSWORD</font> <br />注意 <font color="#0080ff">IDENTITY_CERT_PASSWORD</font> 跟./env/global.override.env 配置内容保持一致;</p> </blockquote> <p style="text-indent: 2em;">执行成功后将生成的 文件,放到部署Bitwarden机器的 bwdata/identity 目录下。至此,配置Bitwarden nginx SSL所需的证书文件都准备就绪。</p> <h2>更新SSL相关配置</h2> <p style="text-indent: 2em;">1、更新 bwdata/config.yml 配置文件,将ssl相关配置修改为实际环境值,例如:</p> <blockquote> <p>url: https://<font color="#0080ff">bw.creast.win</font> <br />ssl: true <br />ssl_managed_lets_encrypt: false <br />ssl_certificate_path: /etc/ssl/<font color="#0080ff">bw.creast.win</font>/<font color="#0080ff">fullchain.pem</font> <br />ssl_key_path: /etc/ssl/<font color="#0080ff">bw.creast.win</font>/<font color="#0080ff">privkey.pem</font></p> </blockquote> <p style="text-indent: 2em;">2、更新 Bitwarden docker 配置文件,重启bitwarden服务</p> <blockquote> <p>./bitwarden.sh stop</p> <p>./bitwarden.sh rebuild</p> <p>./bitwarden.sh start</p> </blockquote> <p>待所有服务重新起来后,在浏览器访问可以看到已经是绿标,显示证书来自“ Let's Encrypt”。用 Bitwarden Desktop 客户端连接也正常了。</p> <p><img alt="Let's Encrypt证书" border="0" height="484" src="https://drive.google.com/uc?id=1fZZj0yWPblWBEwjXmtTkoXuUpCzDtJvM" style="background-image: none; display: inline;" title="Let's Encrypt证书" width="612" /></p>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-59429931562003763852018-10-10T21:23:00.029+08:002021-03-10T21:49:01.447+08:00CSDN登不上去了<p>CSDN居然开启强制捆绑手机号码,强烈 ╭∩╮(︶︿︶)╭∩╮</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglGlkemaO3PlhyTwo0YpPcx8Bq2oLaUWMIA7VjeJQI3Vo1CKj7BbNHc0KmZUeyKRE3OzSXJXeO2p3a4Pd-8v1ukifMAV8B_HjdxkDtNvvHaiL0ps3daKwEeGNcvZ8BRalgXAsSm9stKIA/s1263/CSDN%25E5%25BC%25BA%25E5%2588%25B6%25E7%25BB%2591%25E5%25AE%259A.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="739" data-original-width="1263" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglGlkemaO3PlhyTwo0YpPcx8Bq2oLaUWMIA7VjeJQI3Vo1CKj7BbNHc0KmZUeyKRE3OzSXJXeO2p3a4Pd-8v1ukifMAV8B_HjdxkDtNvvHaiL0ps3daKwEeGNcvZ8BRalgXAsSm9stKIA/s16000/CSDN%25E5%25BC%25BA%25E5%2588%25B6%25E7%25BB%2591%25E5%25AE%259A.png" width="666" /></a></div>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-74104940733194663952017-08-12T21:16:00.000+08:002017-09-19T00:36:39.065+08:00MongoDB 3.4 Role-Based Access Control(MongoDB访问权限控制)<h2>本篇适用情形:</h2><ul><li>MongoDB 启用权限控制</li><li>MongoDB 用户管理</li><li>了解MongoDB 内置角色</li></ul><p><br></p><p>正文:</p><p>【1】内建角色介绍</p><h3>Built-In Roles </h3><ul><li><h4>Database User Roles (exist in <em>each</em> database.)</h4></li><ul><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#read">read</a></p></li><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#readWrite">readWrite</a></p></li></ul><li><h4>Database Administration Roles (exist in <em>each</em> database.)</h4></li><ul><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#dbAdmin">dbAdmin</a></p></li><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#userAdmin">userAdmin</a></p></li><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#dbOwner">dbOwner</a> (readWrite + dbAdmin + userAdmin)</p></li></ul><li><h4>All-Database Roles (in the admin database apply to all but the local and config databases)</h4></li><ul><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#readAnyDatabase">readAnyDatabase</a></p></li><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#readWriteAnyDatabase">readWriteAnyDatabase</a></p></li><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#userAdminAnyDatabase">userAdminAnyDatabase</a></p></li><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#dbAdminAnyDatabase">dbAdminAnyDatabase</a></p></li></ul><li><h4>Superuser Roles</h4></li><ul><li><p><a href="https://docs.mongodb.com/manual/reference/built-in-roles/#root">root</a> (readWriteAnyDatabase + dbAdminAnyDatabase + userAdminAnyDatabase + clusterAdmin)</p></li></ul></ul><p><br></p><p>【2】用户基本操作</p><p>//查看现有用户</p><p>> use admin<br>> db.system.users.find()</p><p>//删除用户dbuser</p><p>> use admin<br>> db.system.users.remove({user:"dbuser"});<br>
</p><p>//创建管理用户dbadmin</p><p>> use admin<br>> db.createUser({ user:"dbadmin", pwd:"passowrd", roles: [{ role: "root", db: "admin" }] });</p><p><br></p><p>//为db_files库创建读写权限用户dbuser</p><p>> use db_files<br>> db.createUser({ user:"dbuser", pwd:"passowrd", roles: [{ role: "dbOwner", db: "db_files" }] });</p><p>//为db_files库创建只写权限用户rduser</p><p>> use db_files<br>> db.createUser({ user:"rduser", pwd:"passowrd", roles: [{ role: "read", db: "db_files" }] });</p><p><br></p>
<p>【3】启用MongoDB用户访问控制</p>
<ol>
<li>导出keyfile(用于集群之间验证身份),并拷贝到各个机器中</li>
<p>导出认证key文件</p><p>openssl rand -base64 741 > mongo-key</p><p>chmod 600 mongo-key #这里必须是600</p>
<li>编辑配置文件</li>
<p>$ sudo vi /etc/mongod.conf<br>security:<br> authorization: enabled<br> keyFile: /home/soft/mongodb-3.2.1/mongo-key</p><p>clusterAuthMode: "keyFile"</p>
<li>重启服务</li>
<p>$ sudo service mongod restart</p>
</ol>
<p>【4】以授权方式访问MongoDB</p>
<ul>
<li>命令行</li>
<p>mongo -u dbuser -p passwd dbtest</p>
<li>URL</li>
<p>mongodb://user_rw:password@192.168.1.21:27017/db_test?replicaSet=???&authSource=db_test</p>
<li>GUI客户端</li>
<p><img width="556" height="392" title="image" style="display: inline; background-image: none;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjh3FDTUekN3h9Jj58G1LHvOLkPpVsc1QzvI2ftiEvhTpgaGCSG9eUt68LwsK-xF6eeyekLGvnWeY9w1gL8bq2TPt0zZ7sEgBx58svhG2ou095FZjSO7dFIQrjErj-d9nBIJ-27k1cftbA/?imgmax=800" border="0"></p><p><img width="556" height="392" title="image" style="display: inline; background-image: none;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEilnsjzR5IBkWjab1k-R7pU2BQkjdQ_eT8D8b8R99IgWShoXXDqvu2A7oIRVJhNLmFVVPKHccGdtrIYN3TqjU2uKMl3cOMc3YZlmevl_x6SAaUG-Zfl3ki3KgaIaBKQxhpyeheGjvdjPdc/?imgmax=800" border="0"></p>
</ul>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-3265845793323257942017-08-10T21:52:00.002+08:002017-09-19T22:38:06.051+08:00Python 3 学习笔记 - 时间相关<h3>时间的表示</h3><p>1、时间不在Python Build-in类型,Python 提供 datetime、time 和 calendar 等模块用于时间类型的处理。</p><p>2、时间戳(timestamp),表示自从1970年1月1日(00:00:00 GMT)到现在经历的秒数。</p><p>3、datetime格式:2016-07-17 16:31:00</p><p>4、时间元组,Python中大部分函数使用元组处理时间,由9个字段构成:</p><table width="600" border="1" cellspacing="0"><tbody><tr><td width="50" valign="top"><p><b>序号</b><b></b></p></td><td width="150" valign="top"><p><b>字段</b></p></td><td width="200" valign="top"><p><b>属性</b></p></td><td width="350" valign="top"><p><b>值</b><b></b></p></td></tr><tr><td width="50" valign="top"><p>0</p></td><td width="150" valign="top"><p>tm_year</p></td><td width="200" valign="top"><p>年,4位数</p></td><td width="350" valign="top"><p>2008</p></td></tr><tr><td width="50" valign="top"><p>1</p></td><td width="150" valign="top"><p>tm_mon</p></td><td width="200" valign="top"><p>月</p></td><td width="350" valign="top"><p>1 到 12</p></td></tr><tr><td width="50" valign="top"><p>2</p></td><td width="150" valign="top"><p>tm_mday</p></td><td width="200" valign="top"><p>日</p></td><td width="350" valign="top"><p>1 到 31</p></td></tr><tr><td width="50" valign="top"><p>3</p></td><td width="150" valign="top"><p>tm_hour</p></td><td width="200" valign="top"><p>时</p></td><td width="350" valign="top"><p>0 到 23</p></td></tr><tr><td width="50" valign="top"><p>4</p></td><td width="150" valign="top"><p>tm_min</p></td><td width="200" valign="top"><p>分</p></td><td width="350" valign="top"><p>0 到 59</p></td></tr><tr><td width="50" valign="top"><p>5</p></td><td width="150" valign="top"><p>tm_sec</p></td><td width="200" valign="top"><p>秒</p></td><td width="350" valign="top"><p>0 到 61 (60或61 是闰秒)</p></td></tr><tr><td width="50" valign="top"><p>6</p></td><td width="150" valign="top"><p>tm_wday</p></td><td width="200" valign="top"><p>一周的第几日</p></td><td width="350" valign="top"><p>0到6 (0是周一)</p></td></tr><tr><td width="50" valign="top"><p>7</p></td><td width="150" valign="top"><p>tm_yday</p></td><td width="200" valign="top"><p>一年的第几日</p></td><td width="350" valign="top"><p>一年中的第几天,1 到 366</p></td></tr><tr><td width="50" valign="top"><p>8</p></td><td width="150" valign="top"><p>tm_isdst</p></td><td width="200" valign="top"><p>夏令时</p></td><td width="350" valign="top"><p>是否为夏令时:1(夏令时)、0(不是夏令时)、-1(未知),默认 -1</p></td></tr></tbody></table><p><br></p><h3>获取时间</h3><blockquote><p>#获取系统时区<br>print (time.<strong>timezone</strong>)</p><p>#获取当前时间戳</p><p>
print ("timestamp:", time.<strong>time() </strong>)<br>timestamp: 1505619448.0100803</p><p>#获取本地时间当前时间元组<br>print (time.<strong>localtime()</strong> )<br>time.struct_time(tm_year=2016, tm_mon=7, tm_mday=17, tm_hour=3, tm_min=37, tm_sec=28, tm_wday=6, tm_yday=260, tm_isdst=0)</p>#获取治天文时间下当前时间元组<br>print (gmtime = time.<strong>gmtime(</strong><strong>) </strong>)<p>#获取当前时间datetime<br>print(datetime.<strong>datetime.now()</strong> )<br>
2016-07-17 16:04:42.246403</p></blockquote><h3>时间格式转换</h3><p>1、时间戳->时间元组:</p><blockquote><p>#当地时间转换<br>lctime = time.<strong>localtime(</strong>time.time()<strong>)</strong><br>
print ("localtime:",lctime)<br>localtime: time.struct_time(tm_year=2016, tm_mon=7, tm_mday=17, tm_hour=3, tm_min=37, tm_sec=28, tm_wday=6, tm_yday=260, tm_isdst=0)</p><p>#格林威治天文时间转换<br>gmtime = time.<strong>gmtime(</strong>time.time()<strong>)<br></strong>print ("GM time:",gmtime)</p></blockquote>2、时间元组->时间戳:<blockquote><p>#保留一位小数<br>print (“timestramp:”, time.<strong>mktime(</strong>lctime)<strong>)<br></strong>timestamp: 1505619448.0</p><p>#不含小数<br>print (“timestramp:”, <strong>calendar.timegm(</strong>gmtime<strong>)<br></strong>timestamp: 1505619448</p></blockquote><p>3、时间戳 –> datetime </p><blockquote><p>#当地时间转换<br>print(datetime.datetime.fromtimestamp(<strong></strong>time.time()))<br>dt = datetime.datetime(2016,7,17,12,31) <br>
print(dt.<strong>timestamp()</strong>)</p><p>#格林威治天文时间转换<br>print(datetime.datetime.utcfromtimestamp(<strong></strong>time.time()))<br>
</p></blockquote><p>4、datetime –> 时间戳</p><blockquote><p>#以指定日期创建datetime<br>dt = datetime.datetime(2016,7,17,12,31) <br> print(dt.<strong>timestamp()</strong>)</p></blockquote><p>5、可读性时间字符串格式转换:</p><blockquote><p>#时间元组格式化为可读形式<br>lctime = time.<strong>asctime(</strong> time.localtime([ts]) <strong>)</strong><br> print ("localtime:",lctime)</p>localtime: Sun Sep 17 03:37:28 2017</blockquote><blockquote><p>#时间元组格式化为datetime形式
<br>print (time.<strong>strftime</strong>("%Y-%m-%d %H:%M:%S", time.localtime()))<br>2016-04-07 10:29:46<br></p></blockquote><blockquote><p>#datetime 格式化为可读形式<br>dt = dtat.time.datetime.now() <br> print(now.strftime('%a, %b %d %H:%M'))<br><code>Wed, Mar </code><code>29</code> <code>16</code><code>:</code><code>18</code></p><p>#字符串-> datetime<br><code>b = <code></code><code>"21/11/06 16:30"</code><br>print (</code><code>datetime.datetime.<strong>strptime</strong>(b</code><code>, </code><code>"%d/%m/%y %H:%M"</code><code>))<br></code>2006-11-21 16:30:00</p>#字符串->时间元组->时间戳 <br>a = "Sat Mar 28 22:24:24 2016" <br>print (time.<strong>mktime</strong>(time.<strong>strptime</strong>(a,"%a %b %d %H:%M:%S %Y")))<br>1459175064.0</blockquote><p>6、时间日期格式化符号:</p><ol><li>%y :两位数的年份表示(00-99)</li><li>%Y :四位数的年份表示(000-9999)</li><li>%m :月份(01-12)</li><li>%d :月内中的一天(0-31)</li><li>%H :24小时制小时数(0-23)</li><li>%I :12小时制小时数(01-12)</li><li>%M :分钟数(00=59)</li><li>%S :秒(00-59)</li><li>%a :本地简化星期名称</li><li>%A :本地完整星期名称</li><li>%b :本地简化的月份名称</li><li>%B :本地完整的月份名称</li><li>%c :本地相应的日期表示和时间表示</li><li>%j :年内的一天(001-366)</li><li>%p :本地A.M.或P.M.的等价符</li><li>%U :一年中的星期数(00-53)星期天为星期的开始</li><li>%w :星期(0-6),星期天为星期的开始</li><li>%W :一年中的星期数(00-53)星期一为星期的开始</li><li>%x :本地相应的日期表示</li><li>%X :本地相应的时间表示</li><li>%Z :当前时区的名称</li><li>%% :%号本身</li></ol><h3>Calendar模块</h3>1、Python默认周一为每周的第一天,可通过调用 calendar.setfirstweekday()函数进行修改:<blockquote><p>#设置每周的起始日期,0(周一)到6(周日)<p>calendar.setfirstweekday(weekday)</p></blockquote>2、常用calendar函数列举<br><blockquote><p>#获取某月日历<br>print (calendar.<strong>month</strong>(2017, 7<b>,w=2,l=1</b>))</p> <pre> July 2017<br>Mo Tu We Th Fr Sa Su<br> 1 2<br> 3 4 5 6 7 8 9<br>10 11 12 13 14 15 16<br>17 18 19 20 21 22 23<br>24 25 26 27 28 29 30<br>31</pre>#获取2017年全年日历<br>print (calendar.<strong>calendar</strong>(2017<b>,w=2,l=1,c=6</b>))<p>#以列表形式输出某月日历<br>print (calendar.<b>monthcalendar</b>(2017, 7))<br>[[0, 0, 0, 0, 0, 1, 2], [3, 4, 5, 6, 7, 8, 9], [10, 11, 12, 13, 14, 15, 16], [17, 18, 19, 20, 21, 22, 23], [24, 25, 26, 27, 28, 29, 30], [31, 0, 0, 0, 0, 0, 0]]</p><p>#闰年判断,是返回True,否则为false<br>calendar.isleap(year)</p><p>#返回给定日期周几,0(周一)到6(周日)<br>calendar.weekday(year,month,day)</p></blockquote>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-41334210095858034742017-08-08T21:05:00.000+08:002017-09-16T22:10:43.041+08:00Python 3 学习笔记 - 数据类型<h2>Python变量</h2><p>1、Python是动态语言,不需要声明变量,但变量在使用前必须赋值,变量赋值以后该变量才会被创建。变量的类型由所赋予的值的类型来决定通。<p>2、查询变量所指向的对象数据类型: <blockquote><p>type() //返回数据类型,如class 'int'<p>isinstance(a, int) //返回 True / False</p></blockquote><p>3、一个变量可以通过赋值指向不同类型的对象,只保留最后一次赋值<p>4、可以同时为多个变量赋值,例:<blockquote><p>a, b, c , d = 100, ‘bob’, 3e+26J, True</p></blockquote><p><br><h2>Python数据类型</h2><p>Python3 中有六个标准的数据类型:<table width="700" border="1" cellspacing="0" cellpadding="1"><tbody><tr><td width="53" valign="top"><p><b>序号</b><b></b></p></td><td width="141" valign="top"><p><b>数据类型</b></p></td><td width="208" valign="top"><p><b>说明</b></p></td><td width="207" valign="top"><p><b>赋值示例</b></p></td></tr><tr><td width="53" valign="top"><p>01</p></td><td width="141" valign="top"><p>Number(数字)<br><b><font color="#c0504d">不可变类型</font></b></p></td><td width="208" valign="top"><p>4种数字类型,Python3支持布尔型</p></td><td width="207" valign="top"><p>num = 1</p></td></tr><tr><td width="53" valign="top"><p>02</p></td><td width="141" valign="top"><p>String(字符串)<br><b><font color="#c0504d">不可变类型</font></b></p></td><td width="208" valign="top"><p>由1个或多个字符组成</p></td><td width="207" valign="top"><p>str = ‘abcd’</p></td></tr><tr><td width="53" valign="top"><p>03</p></td><td width="141" valign="top"><p>List(列表/数组)<br><b><font color="#4bacc6">可变类型</font></b></p></td><td width="208" valign="top"><p>列表是有序的对象结合</p></td><td width="207" valign="top"><p>list = [1,2,3,4,5,’six’]<br>lists = [<br> [‘a’,’A’,1], [‘b’,’B’,2]<br>]</p></td></tr><tr><td width="53" valign="top"><p>04</p></td><td width="141" valign="top"><p>Tuple(元组)<br><b><font color="#c0504d">不可变类型</font></b></p></td><td width="208" valign="top"><p>数组的简化,最大特点是不可修改,作常量使用</p></td><td width="207" valign="top"><p>tup = (1,2,3,4,5,’six’)</p></td></tr><tr><td width="53" valign="top"><p>05</p></td><td width="141" valign="top"><p>Sets(集合)</p></td><td width="208" valign="top"><p>集合是无序不重复元素的序列</p></td><td width="207" valign="top">set1 = {'aa', 'bb', 'cc'}<br>set2 = set('abcada')</td></tr><tr><td width="53" valign="top"><p>06</p></td><td width="141" valign="top"><p>Dictionary(字典)<br><b><font color="#4bacc6">可变类型</font></b></p></td><td width="208" valign="top"><p>字典是一种映射类型,是无序的对象集合</p><p><br></p></td><td width="207" valign="top"><p>dict = {<br> ‘k1’:’a’,<br> ‘k2’:’1’ ,<br> ‘k3’:{k31:31}}</p></td></tr></tbody></table><p><br></p>
<h3>Number(数字)</h3><p>1、Python3 数字类型支持: <b>int(整数)、float(浮点)、bool(布尔)、complex(复数)</b><p>2、在混合计算时,Python会把整型转换成为浮点数。<p><br><h3>String(字符串)</h3><p>1、字符串用单引号<strong>'</strong>xxx<strong>'</strong>或双引号<strong>"</strong>xxx<strong>"</strong></p><p>2、三引号<strong>"""</strong>xxx<strong>"""</strong> 或者 <strong>'''</strong>xxx<strong>'''</strong>,表示多行的字符串,三引号中可嵌套单引号和双引号</p><p>3、反斜杠<strong>\ </strong>为转义字符 或 换行,字符串前加 <strong>r </strong>或<strong> R</strong>可以让反斜杠不发生转义</p><p>4、字符串的索引值以 0 为开始值,变量[上标:尾下标]<br></p>
<p>5、加号 (+) 是字符串的连接符, 星号 (*) 表示复制当前字符串</p><blockquote><p>print (str + "ABC") # 连接字符串<br>print (str * 2) # 输出字符串两次<br></p></blockquote><p><br></p><h3>List(列表)</h3><p>1、列表用方括号<strong>[</strong>xxx<strong>]</strong>表示,元素之间用逗号 <strong>, </strong>隔开</p><p>2、列表的元素可以多种类型,可嵌套列表,例:</p><blockquote><p>list = [ 'abcd', 123, 3.1415, 'efg', 70.2 ]</p></blockquote><p>3、列表可以使用 + 操作符进行拼接</p><p>4、列表中的元素是可变的</p><p>5、列表常用操作:</p><p>insert(n,v),在数组索引n的位置,插入值v</p><blockquote><p>num.insert(1,9)<br>[1, 9 , 2, 3 ,4 ,5]<br></p></blockquote><p>append(),在数组末尾添加上元素</p><blockquote><p>num.append(6)<br>[1, 9 , 2, 3 ,4 ,5, 6]<br></p></blockquote><p>pop(),默认删除最后一个元素,pop(n)删除索引n对应的元素</p><blockquote><p>num.pop(1)<br>[1, 2, 3, 4, 5, 6]</p>num.pop()<br>[1, 2, 3, 4, 5]</blockquote><p><br></p><h3>Tuple(元组)</h3><p>1、元组用括号<strong>(</strong>xxx<strong>)</strong>表示,元素之间用逗号 <strong>, </strong>隔开</p><p>2、元组特性与列表一样,但元组的元素不能修改</p><p>3、元组可以使用 + 操作符进行拼接</p><p>4、包含 一个元素的元组,需要在元素后添加逗号</p><blockquote><p>tup1 = (20,) </p></blockquote><p>5、Python函数返回多个值的时候,是以元组的方式返回</p><p><br></p><h3>序列切片</h3><p>1、String、List、Tuple 都属于序列。</p><p>2、list[x:y] 表示从索引x开始到索引y的元素集合</p><p>3、方括号左边是闭区间,右边是开区间,所以不包含list[y]</p><p>4、特殊索引位说明:</p><p>[0:]表示从索引0开始,直到最后一个元素<br>[:n]表示从第一个元素开始,直到索引n<br>[:-1]表示从第一个元素开始,到<font color="#ff0000">倒数第二</font>个元素</p><p>以字符串为例:</p><blockquote><p>str = 'abcdefg'<br> <br>print (str) # 输出字符串<br>print (str[0]) # 输出字符串第一个字符<br>print (str[0:-1]) # 输出第一个到<font color="#ff0000">倒数第二个</font>的所有字符<br>print (str[2:]) # 输出从第三个开始的后的所有字符</p></blockquote><p><br></p><h3>Set(集合)</h3><p>1、集合使用大括号 { } 或者 set() 函数创建</p><blockquote><p>set_stu1 = {'aa', 'bb', 'cc', 'bb', 'cc', 'aa'}<br>set_stu2 = set('abcadcdabaca')</p></blockquote><p>2、创建一个空集合必须用 set() ,而不能用 { }</p><p>3、集合的基本功能是进行成员关系测试和删除重复元素,例:</p><blockquote><p>print (set1)<br> {'aa', 'bb', 'cc'}</p>print (set2)<br>{'b', 'a', 'd', 'c'}</blockquote><p><br></p><h3>Dictionary(字典)</h3><p>1、字典用"{ }"标识,它是一个无序的<b>键(key) : 值(value)</b>对集合</p><p>2、字典的<strong>键(key)</strong>必须为不可变类型,且必须唯一</p><p>3、创建空字典使用 <b>{ }</b></p><p>4、字典元素的输出顺序与定义顺序无关,因为定义时key的顺序和放在内存的key顺序没有关系,例:</p><blockquote><p>dict = {‘bb’:20, ‘aa’: 10 , ‘cc’:30}<br>print(dict)<br>{‘aa’:10, ‘bb’: 20 , ‘cc’:30}<br></p></blockquote><p>5、字典采用散列表(hashtable)的算法,不管字典中有多少项,in操作符花费的时间都差不多。</p><p>6、字典的内置的函数</p><blockquote><p>dict.keys() 返回所有的key值<br>dict.values() 返回所有的value值<br>dict.items() 返回key,value组成的元组</p><p>list(dict.items() ) 将返回结果转换成列表/数组</p></blockquote><p><br></p><h2>数据结构的遍历</h2><p>1、列表/元组的遍历</p><blockquote><p>for v in list:<br> print(v)</p></blockquote><p>2、列表+循环用法:</p><blockquote><p>[i*2 for i in range(10) if i%2==0]</p></blockquote><p>3、字典的遍历,输出key</p><blockquote><p>for k in dict:<br> print(k)</p></blockquote><p>4、字典的遍历,输出value值</p><blockquote><p>for v in dict.values():<br> print(v)</p></blockquote><p>3、字典的遍历,同时输出key:value</p><blockquote><p>for k,v in dict.items():<br> print(k,v)</p><p>或者</p><p>for c in dict: <br> print(c,':',dict[c])</p></blockquote><p>***dict循环输出不是按赋值时的顺序</p><p><br></p><h2>数据类型转换</h2><p>常用类型转换内置的函数列表:</p><table width="700" border="1" cellspacing="0" cellpadding="1"><tbody><tr><td width="217" valign="top"><p><b>函数</b><b></b></p></td><td width="511" valign="top"><p><b>描述</b><b></b></p></td></tr><tr><td width="217" valign="top"><p>int(x [,base])</p></td><td width="511" valign="top"><p>将x转换为一个整数</p></td></tr><tr><td width="217" valign="top"><p>long(x [,base] )</p></td><td width="511" valign="top"><p>将x转换为一个长整数</p></td></tr><tr><td width="217" valign="top"><p>float(x)</p></td><td width="511" valign="top"><p>将x转换到一个浮点数</p></td></tr><tr><td width="217" valign="top"><p>complex(real [,imag])</p></td><td width="511" valign="top"><p>创建一个复数</p></td></tr><tr><td width="217" valign="top"><p>str(x)</p></td><td width="511" valign="top"><p>将对象 x 转换为字符串</p></td></tr><tr><td width="217" valign="top"><p>repr(x)</p></td><td width="511" valign="top"><p>将对象 x 转换为表达式字符串</p></td></tr><tr><td width="217" valign="top"><p>eval(str)</p></td><td width="511" valign="top"><p>用来计算在字符串中的有效Python表达式,并返回一个对象</p></td></tr><tr><td width="217" valign="top"><p>tuple(s)</p></td><td width="511" valign="top"><p>将序列 s 转换为一个元组</p></td></tr><tr><td width="217" valign="top"><p>list(s)</p></td><td width="511" valign="top"><p>将序列 s 转换为一个列表</p></td></tr><tr><td width="217" valign="top"><p>set(s)</p></td><td width="511" valign="top"><p>转换为可变集合</p></td></tr><tr><td width="217" valign="top"><p>dict(d)</p></td><td width="511" valign="top"><p>创建一个字典。d 必须是一个序列 (key,value)元组。</p></td></tr><tr><td width="217" valign="top"><p>frozenset(s)</p></td><td width="511" valign="top"><p>转换为不可变集合</p></td></tr><tr><td width="217" valign="top"><p>chr(x)</p></td><td width="511" valign="top"><p>将一个整数转换为一个字符</p></td></tr><tr><td width="217" valign="top"><p>unichr(x)</p></td><td width="511" valign="top"><p>将一个整数转换为Unicode字符</p></td></tr><tr><td width="217" valign="top"><p>ord(x)</p></td><td width="511" valign="top"><p>将一个字符转换为它的整数值</p></td></tr><tr><td width="217" valign="top"><p>hex(x)</p></td><td width="511" valign="top"><p>将一个整数转换为一个十六进制字符串</p></td></tr><tr><td width="217" valign="top"><p>oct(x)</p></td><td width="511" valign="top"><p>将一个整数转换为一个八进制字符串</p></td></tr></tbody></table>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-82487239578156983742017-07-20T22:15:00.001+08:002017-08-31T22:14:04.761+08:00Windows 10 WinSxS Cleanup<p style="text-indent: 2em;">笔记本用的Windows 10 从Windows 7升级来的,自装win7到现在有好几年了,系统盘也变得异常臃肿,不得不对WinSxS等目录做个清理。</p><ol>
<li>清理WinSxS,以管理员权限运行以下命令 (cmd with administrator rights): type</li>
<p>Dism.exe /online /Cleanup-Image /StartComponentCleanup</p>
<p>Dism.exe /online /Cleanup-Image /StartComponentCleanup /ResetBase</p>
<p><br></p>
<li>删除历史更新下载的备份文件:</li>
<p>delete files in %SystemDrive%\Windows\SoftwareDistribution\Download</p><p><br></p>
<li>删除积攒已久的驱动文件,以管理员权限运行以下命令(cmd with administrator rights): and type:</li>
<p>TAKEOWN /F "C:\Windows\System32\DriverStore\FileRepository" /R<br>
ICACLS "C:\Windows\System32\DriverStore\FileRepository" /T /L /GRANT *S-1-1-0:F<br>
RD /s /Q "C:\Windows\System32\DriverStore\FileRepository\"</p></ol>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-19918422778135732622017-07-17T20:47:00.001+08:002017-09-16T21:13:09.349+08:00Gradle 与 Maven 项目转换<h2>【gradle --> maven】</h2>
<p>修改build.gradle,增加以下内容:(group,version可自行修改,artifactId默认为目录名称)</p><blockquote>
<p>apply plugin: 'java'<br>
apply plugin: 'maven'</p><p>group = 'com.creast'<br>version = '0.7-dev'<br>sourceCompatibility = 1.6</p></blockquote>
<p>
然后./gradlew build ,成功后将在build\poms目录下生成pom-default.xml文件,把它复制到根目录下,改名成pom.xml即可</p><p><br></p>
<h2>【maven --> gradle】</h2>
<p>先保证本机安装了gradle 2.0以上的版本</p>
<p>然后在maven根目录下运行:</p>
<blockquote><p>gradle init --type pom</p></blockquote>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-69475355656541553612017-07-16T21:05:00.002+08:002017-09-16T23:42:30.279+08:00Python 3 学习笔记 - 函数与类<h2>Python函数</h2><h3>1、函数代码块以 <b>def</b> 关键词开头进行定义</h3><blockquote><pre>def func(arg1[,arg2,arg3,…,argN]):
[expression]</pre><p>#示例(为什么每次def一个函数都是一个不开心的开始呢):<br>def func(x,y,z=0):<br> if x > y:<br> print(“>”)<br> else:<br> print(“<”)<br> if z==1: <br> return x-y</p></blockquote><h3>2、函数参数</h3><p>必需参数 </p><ul><li>调用函数,必需参数须以正确的顺序传入函数,调用时的数量必须和声明时的一样,不然会出现语法错误:<br>TypeError: func() missing 1 required positional argument: 'arg'</li></ul><p>关键字参数 </p><ul><li>函数调用使用关键字参数来确定传入的参数值,不需要按定义时的顺序</li></ul><p>默认参数 </p><ul><li>定义函数时为参数赋值,如果没有传递参数值,则会使用默认值</li><li>默认参数必须放在最后面</li></ul><p>不定长参数</p><ul><li>函数定义时 *变量名 来存放所有未命名的变量参数,调用时可为空:<br>def func( arg1, *vartuple ):</li></ul><h3>3、return语句</h3><p>不带参数值的return语句返回None</p><p>函数返回多个值的时候,是以元组的方式返回,可以对返回的元组进行转换后使用</p><h3>4、匿名函数</h3><p>使用 lambda 表达式创建匿名函数:</p><blockquote><pre>变量名 = lambda [arg1[,arg2,arg3,…,argN]]:expression</pre></blockquote><p>匿名函数拥有自己的命名空间,且不能访问自有参数列表之外的或全局命名空间里的参数。</p><p><br></p><h2>类(Class)</h2><h3>1、Python中类的概念跟C++ 中基本一样,基本功能:</h3><ul><li><strong>继承:</strong>即一个派生类(derived class)继承父类/基类(base class)的字段和方法。<li><strong>方法:</strong>类中定义的函数。 <li><strong>方法重写:</strong>子类对从父类继承的方法进行改写/重构<li><strong>实例化:</strong>创建一个类的实例,类的具体对象<li><strong>对象:</strong>通过类定义的数据结构实例。对象包括两个数据成员(类变量和实例变量)和方法。</li></ul><h3>2、关键字 <font style="font-weight: bold;">class</font> 定义类</h3><blockquote><p><strong>class</strong> classname: <br> def<strong> __init__(self)</strong>: #类的构造函数 __init__()<br> [expression]</p><p> def func(<strong>self</strong>): #类的方法必须有第一个参数 <strong>self</strong>,代表类的实例<br> [expression]</p><p> <statement-1> <br> . <br> . <br> <statement-N></p></blockquote><h3>3、类对象调用</h3><blockquote><h3>instance<strong>.objname </strong>或<strong> </strong>instance<strong>.func()</strong></h3></blockquote><h3>4、类的继承</h3><blockquote><p>class SonClassName(BaseClassName1):</p><p>#父类在另一个模块中时:<br>class SonClassName(<strong>modname</strong>.BaseClassName):</p><p>#多父类继承:<br>class SonClassName(Base1, Base2, Base3):</p></blockquote><h3>5、类的私有属性</h3><ul><li><strong>__</strong>private_attrs:__开头声明该属性为私有,不能在类的外部被使用或直接访问,在类内部的方法中使用时 <strong>self.__private_attrs</strong></li><li><strong>__</strong>private_method:__开头声明该方法为私有方法,不能在类的外部调用,在类的内部调用 <strong>self.__private_methods</strong></li></ul><h3>6、类的专有方法:</h3><ul><li><strong>__init__ :</strong> 构造函数,在生成对象时调用
<li><strong>__del__ :</strong> 析构函数,释放对象时使用
<li><strong>__call__:</strong> 函数调用 <li><strong>__repr__ :</strong> 打印,转换
<li><strong>__str__:</strong> 被打印时调用<li><strong>__setitem__ :</strong> 按照索引赋值
<li><strong>__getitem__:</strong> 按照索引获取值
<li><strong>__len__:</strong> 获得长度
<li><strong>__cmp__:</strong> 比较运算
<li><strong>__add__:</strong> 加运算 <strong>+</strong> <li><strong>__sub__:</strong> 减运算 <strong>-</strong><li><strong>__mul__:</strong> 乘运算 <strong>×</strong>
<li><strong>__div__:</strong> 除运算
<strong>÷</strong><li><strong>__mod__:</strong> 求余运算
<li><strong>__pow__:</strong> 乘方</li></ul><p><br></p><h2>变量的作用域</h2><h3>1、Python 变量的作用域一共有4种,按查找顺序如下:</h3><ul><li>L (Local) 局部作用域 ,定义在函数内部的变量<li>E (Enclosing) 闭包函数外的函数中 <li>G (Global) 全局作用域 ,定义在函数外的变量<li>B (Built-in) 内建作用域</li></ul><h3>2、作用域只对模块(module),类(class)以及函数(def、lambda)有效</h3><blockquote><p>其它代码块(如 if/elif/else/、try/except、for/while等)没有作用域定义,这这些语句内定义的变量,外部也可以访问</p><p>if True:
<br> msg = 'in if block' </p><p>def func():<br> msg_func = 'in func block'</p><p>print(msg)<br>>>>in if block</p><p>print(msg_func)<br>>>>NameError: name 'msg_inner' is not defined</p></blockquote><h3>3、修改变量作用域</h3><p>修改函数内部的变量,使用 <strong>global</strong> 关键字声明</p><blockquote><p>def func():<br> global variate</p></blockquote><p>修改嵌套函数内的变量,使用<strong>nonlocal</strong>关键字声明</p><blockquote><p>def func():<br> i = 1<br> def innerfun():<br> nonlocal variate<br> print(variate)<br> innerfun()</p></blockquote>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-19868150934068408012017-06-17T23:04:00.000+08:002017-09-16T23:47:59.747+08:00Python 3 学习笔记 - 基本语法<h2>Python标识符</h2><p>1、标识符第一个字符必须是字母表中字母或下划线'_'</p><p>2、标识符的其它部分由字母、数字和下划线'_'组成</p><p>3、标识符大小写敏感</p><p>4、标识符不可与Python保留字冲突,查询保留字:</p><blockquote><p>>>> import keyword<br>>>> keyword.kwlist<br>['False', 'None', 'True', 'and', 'as', 'assert', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']</p></blockquote><p><br></p><h2>代码注释</h2><p>1、单行注释以 <b>#</b> 开头</p><p>2、多行注释可以用多个 # 号,或者使用三个单引号<strong> '''</strong> 或三个双引号 <strong>"""</strong> 将注释括起来</p><p>3、py文件第一行注释标用于指向 python 解释器的路径,例:</p><blockquote><pre>#!/usr/bin/python3</pre><pre>#!/usr/bin/env python3</pre></blockquote><p><br></p><h2>导入模块import</h2><p>1、Python中用 import 或者 from...import… 来导入相应的模块,类似于C语言中的include头文件</p><p>2、从某个模块中导入某个函数,格式为: </p><blockquote>from moduleX import func</blockquote><p>3、从某个模块中导入多个函数,格式为: </p><blockquote>from moduleX import funA, funB, funC<!--EndFragment--></blockquote><p>4、将整个模块导入,格式为:</p><blockquote>from moduleX import *<p>或者<p>import moduleX </p></blockquote><p>5、通过import moduleX 导入,引用时要加module名前缀,例:<font color="#ff0000">sys.</font></p><blockquote><p>import sys<br>def example1():<p>print <font color="#ff0000">sys.</font>path</p></blockquote><p>6、Python常用的包</p><blockquote><p>数据分析<br>numpy、scipy、pandas</p><p>绘图包<br>matplotlib、seaborn</p><p>机器学习<br>scikit-learn、Gensim、NLTK</p><p>网络爬虫<br>urllib、BeautifulSoup</p></blockquote><ol>
</ol>
<p><br></p>
<h2>代码块格式</h2><p>1、Python使用缩进来表示代码块,不需要使用大括号<strong>{ }</strong></p><p>2、同一个代码块的语句必须包含相同的缩进量(空格 或 tab)</p><p>3、函数之间或类的方法之间用空行分隔,区分两段不同功能或含义的代码</p><p>4、代码语句过长时,使用反斜杠 <strong>\</strong> 实现语句换行</p><blockquote><pre>total = item_one + \
item_two + \
item_three</pre></blockquote><p>5、Python也可在同一行中使用多条语句,语句之间使用分号 <strong>;</strong> 隔开</p><blockquote><pre>#!/usr/bin/python3
import sys; x = 'creast'; sys.stdout.write(x + '\n')</pre></blockquote><p>6、多个语句构成代码组,首行以关键字开始,以冒号 <strong>:</strong> 结束</p><blockquote><p>if expression : <br> case1<br>
elif expression : <br> case2 <br>
else : <br> case3</p></blockquote><p><br></p><h2>循环与控制流</h2><h3>1、循环判断常用操作符</h3><blockquote><p>< 小于 <br><= 小于或等于 <br>> 大于 <br>>= 大于或等于 <br>== 等于,比较对象是否相等 <br>!= 不等于</p></blockquote><h3>2、if 条件控制</h3><blockquote><p><strong>if</strong> a>b or b < 0<strong>:</strong><br> print(“>”)<br><strong>elif</strong> a == b<strong>:</strong><br> print(”=”)<br><strong>else:</strong><br> print(“<”)</p></blockquote><h3>3、while 循环</h3><blockquote><p><strong>while</strong> i < n<strong>:</strong><br> i+=1<br> if i%2 == 1:continue<br> print(“the num is:”,i)<br> if i == 10: break<br>print(”end while”)</p><p><strong>while</strong> i< n<strong>:</strong> <br> print (i, " <") <br> count += 1<br><strong> else</strong>: <br> print (">=")<br></p></blockquote><h3>4、for 循环</h3><blockquote><p><strong>for</strong> i <strong>in</strong> range(n)<strong>:</strong><br> print(“the num is:”,i)<br><strong>else:</strong><br> print("只在 for 循环正常终止时执行")</p></blockquote><h4><br></h4><h2>常用函数 </h2><h3>range()函数</h3><p>1、range(n) 默认从0开始,执行n次。</p><p>2、range(x,y,z) 参数说明:</p><blockquote><p>x: 起始值,包含<br>y: 终止值,不含<br>z: 步长,可以为负数</p></blockquote><h3>print()函数</h3><p>1、sep 参数用于将结果输出到同行,在每个输出元素之间添加分隔符:</p><blockquote><pre>>>> print(‘aa’,’bb’,’cc’,sep=' : ')<br>aa : bb : cc
</pre></blockquote><p>2、end 参数用于在输出的结尾添加分隔符,且下一个输出不换行,:</p><blockquote><p>>>> print(‘aa’,end=' : ')<br>>>> print(‘bb’,end=' : ')<br>>>> print(‘cc’,end=' : ')</p><p>aa : bb : cc</p></blockquote>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-50917846571539071612017-03-23T21:07:00.001+08:002017-08-30T15:24:08.912+08:00CDH安装Phoenix<p style="text-indent: 2em">目前Phoenix并没有作为一个标准集成的service被CDH支持,所以不能通过添加服务的方式进行添加,但通过Cloudera labs提供的parcel也可以很方便的进行集群部署。</p>
<h4>1、下载安装包</h4>
从以下地址获取phoenix parcel文件:<br />
<a href="http://archive.cloudera.com/cloudera-labs/phoenix/parcels/latest/" target="_blank"><u>http://archive.cloudera.com/cloudera-labs/phoenix/parcels/latest/</u></a><br />
选择响应的OS版本,以ubuntu 14.04为例选择trusty的版本:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXNqGfkI7GO9z4Y4N5DwJAUAIEKjJ23YYhJJz1CPRO6otwi4oTCo1sNV-6KumaXRkIW7dZvvkd5Gr_Y4SV-YjHlgtOyn8LmWiQeLP6z0h2qiO4jPHgHf4NwwDvawxwcScGBBPecf1nf4k/s1600-h/wpsD925.tmp%255B5%255D"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghs6FwX7fTpmLUAdbGneHKbV5wVf4u-78fo6JqxfMq-0zRbn2fGiqzcRVLN5ZNbzInoDsQa21ubUGqi0WJnIi6wXNbP3vIEGOD4p-5BwrhL6M7K3dEd_lhJyGhvkPCSUnVD7KXTa5MSZI/?imgmax=800" style="background-image: none; border: 0px; display: inline;" title="" /></a><br />
注意:CDH不支持直接部署Apach phoenix (<a href="http://archive.apache.org/dist/phoenix/" target="_blank">http://archive.apache.org/dist/phoenix/</a>),包依赖关系不同。<br />
下载后上传到CDH hadoop集群主节点parcel目录:/opt/cloudera/parcel-repo<br />
将parcel文件的 ownership 改为:cloudera-scm:cloudera-scm<br />
<h4>
2、分发部署parcel</h4>
登录CM管理后台,点击右上角<img alt="" border="0" src="https://lh3.googleusercontent.com/-dCwAq9qFQDU/WaYVmgV59OI/AAAAAAAAOqE/GOr2FuUaGYw80Kkf8jC6qXEg8TS-eZtwACHMYCw/wpsD926.tmp%255B4%255D?imgmax=800" style="background-image: none; display: inline;" title="" />,在Parcel 配置窗口新增一行url:<br />
<u>http://archive.cloudera.com/cloudera-labs/phoenix/parcels/latest/</u><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9QzdChyphenhyphen2tOGniYS92W2e3gcVRbdkp4JKzc0MnhRRrql5lt84RtBAM-fyP0yEAzXzU1I8Zvt72mExTvIXvdWrzEGAiXzmTvh91gJl_d4OthcvH1u4tDrcfQo1tt7V04XygJfyCMDvii7A/s1600-h/wpsD936.tmp%255B5%255D"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYIwZ3WFesTPuijozzDb8QIPY3OtuiLh3zexKunBaiWMmMTBh21EBokHiGuFlKSZVEsXMSKhjDVoHcVf0Wl80ZnpsA_Do5mDOc_O_XznYvW1NLwwnyxwDjPfzZAEutGCsbUK1owfDDE7g/?imgmax=800" style="background-image: none; border: 0px; display: inline;" title="" /></a><br />
完成后可以看到新增了一个CLABS_PHOENIX的Parcel,<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRCSRU6zlxLKpJAaHtb1JOVj_HAZmHXsjIeGWIT1h4ShCVmLxfQ2uhjpzQWg2Bmydm6IBloGKycMS9nhV9M4HMISNJHv5ue3KRa8s46cEoGIT_iL4ySjTZ25b6Mv5tKMnsZkWYv2w_4B4/s1600-h/wpsD937.tmp%255B5%255D"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhH9vGCN-qI31QIa5Gnrwqv59QwDUkC91Gc411vypATQ_O3LpRgvjbV6zQ7sjl5gAR-AQNJJzGS4JbrlLOugBKSoane_uuHMy-GuNn7DmE74_gFffxFCR7T0YDFIRiwAsCbGwNttomiLC0/?imgmax=800" style="background-image: none; border: 0px; display: inline;" title="" /></a><br />
将parcel分发到所有节点:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPXfIb9eYRw7b7YZoJ_DYNp0ABTbW5z9CbG9dXzzwZ_pXqyUQf4eIvsLdtKW0ivWDEGRKBzGZ96jpIMUCL0BmHSo8tCFUiDg5k9YW0Pd6d12ooTTUpk6FI1nYjknR5u6kuUQlFoEgCVHY/s1600-h/wpsD938.tmp%255B5%255D"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhpsa1mahD2_LTTJdl74XXWlSPpn-_9vD1r8CfaNrzSJBxDPhTQ9K0_WT2uVfkxPB6Em4WxzXpidp2YgimaiRQCwETblke5vdxCxXOdZgpt9P1qm7vfrdt9vh2qT8r5GvluXO2NdMdpDk/?imgmax=800" style="background-image: none; border: 0px; display: inline;" title="" /></a><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbqomrNYkzM5q2HjlKoGg21bVaVC-DMn77XJlqIa0T9qrN9-k_dYxP2LXOxjZBZsvmS-fMLHZbRZL5uWwA6-J8VZ9uzjT44aQk3d-D-YAMOM4cZ6oIdCfWZ0AhZVtGIu3tNOssSePktx8/s1600-h/wpsD939.tmp%255B5%255D"><img alt="" border="0" height="287" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuZKAI9A1mADjprqFEFmLScU6REejghPpodru8R96Tu0WQN8BSyJLnxS7j8x-wsRUKrfPa0eDCgN49QMrVj5C2AOf-Z_HO4MVVcJ6sUIf-pxVCEyEZVKRxOtMQI2WUv0HCO0i1CeAPumk/?imgmax=800" style="background-image: none; border: 0px; display: inline;" title="" width="640" /></a><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGg6JSeywJf7V3Ue0eCL1yK_AsnhgA8baNtKUVa2xJFY4F5IamamhXIdOt_TXUB7ja3Ec4oGL9jjrZXpRhDd0u8iWDGxLicL9UinJAbrwMAUjBGJtELBePr8o5Zhpv8wD-B81GLlQyCao/s1600-h/wpsD94A.tmp%255B5%255D"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxDsKod6RpGKaH31RuP5qI6Lbms_Msu7idFWdb7gzQ4MRPvqmnTNI1AIRp6BlWX4w8iZgsq-nm4kKzb6ee1_RQTWNxzECFDyPppqM41ROVLi_OInuFxJF1W52_2BlfTm_URLi3Y2VxtKI/?imgmax=800" style="background-image: none; border: 0px; display: inline;" title="" /></a><br />
激活Phoenix parcel 之后,需要重启HBase及集群相关服务:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNpRuXCMzCp7LqtmOz4RZmJC8iiXmgX7nwz6lJ7yclswntr5z1gObkfhhgzA7I1f0rnEBtHs_TZCJjZXOOAVvyWjfBA1CvEO4qa-yneugjWrJIw_84yNHRfFWTOEAaNmO_pRzy9V2GfYE/s1600-h/wpsD94B.tmp%255B5%255D"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqm0-UU4abc_eyjTgJ5qxn5lk_yfZPS3vZE78VF6gy1QD799z2EUqqlFRm7g8bDkY9Ub-W9CURTGFg1VsSOpw02RLDgxYbY3ylMxf9vpYQ_8eLZfGsz_R-r6UuKlYFQO_0JnrpKXQh8bI/?imgmax=800" style="background-image: none; border: 0px; display: inline;" title="" /></a><br />
<h4>
3、功能测试</h4>
重启完成后,在任一节点(ZooKeeper节点)上简单测试一下,可以正常work了:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNro4fbezqT6gZtubPACnGZY7_vzJBsHoo6Ez3GIPD1zgItf9teU8VnqfcvWXjZTbcs05IBTnNlq8eW_6t0gshl8n7DlxEbD4BxjzD3caejQGCqhovmiyRDE2QLNsZv8Dspsq_mgh_msQ/s1600-h/image%255B5%255D"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgX_9U8_XdSe7fVyzV7ujxF14-tsvagACoy18CCCTASlMcAJoio14fZbgCKYZu38kLTqKcW3amKOKRrJmwcqgehL9vZZp57DZpUQQbLRXBZvnSo2BCbWFrtReTz_-Il5_akRxl1MTStPp4/?imgmax=800" style="background-image: none; border: 0px; display: inline;" title="" /></a><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAQyXp8X4d4RFG9zwXZkZGEI1l1h46_qkdWDFgWBHRqa4dpc-C0TrP66GO9awMlazqPUz4tCeHwLJlc4QCo-nBbAqZFpHVm-qQ8MIjSJuDgS0P6n8oh_TbCm-fKQrBjHBtUyLkKbHQtdQ/s1600-h/wpsD95D.tmp%255B5%255D"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQKOlyTEhJA0RDZa6vz3Gmf5INaiUHt0I31Ktgvjqt0nwZ0B5tmfhF2yo8Do2hfudQq1gWaOdXN5biQg8o2BGOQjR6-X7bmS5zxG7tC85MOOPtjaxXGorT7G8sucYtz_6TBCaO8g65S-o/?imgmax=800" style="background-image: none; border: 0px; display: inline;" title="" /></a>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-79445036048710463552017-03-23T20:35:00.000+08:002017-09-02T17:06:38.456+08:00Cloudera Hadoop 集群安装(ubuntu + CDH5.10)<h2>前言</h2><p>CDH是Cloudera公司的Hadoop发行版,基于稳定版本的Apache Hadoop构建,包含Hadoop,Spark,Hive,Hbase和一些工具等,并集成了很多补丁,可直接用于生产环境。通过Cloudera Manager可以简化hadoop各组件的部署和配置过程。<p>目前CDH对ubuntu支持的最高版本为14.04,因此不建议使用更高版本的ubuntu来部署CDH。<h2><b>环境准备:</b><b></b></h2><h4>1.配置网络(所有节点)</h4><p>修改hostname:<blockquote><p>$ sudo vi /etc/hostname<p>cdh01</p></blockquote><p>修改IP地址:<blockquote><p>$ sudo vi /etc/network/interfaces<p>auto eth0<br>iface eth0 inet static<br> address 172.16.13.11<br> netmask 255.255.255.0<br> gateway 172.16.13.254</p></blockquote><p>重启网络服务生效:<blockquote><p>$ sudo ifdown eth0 && sudo ifup eth0</p></blockquote><p>修改ip与主机名的对应关系:<blockquote><p>$ sudo vi /etc/hosts<p>172.16.13.11 cdh01<br>172.16.13.12 cdh02<br>172.16.13.13 cdh03</p></blockquote><h4>2.配置SSH(所有节点)</h4><p>启用root登陆(CDH5.10需要进行此步操作)<blockquote><p>$ sudo vi /etc/ssh/sshd_config<p>#PermitRootLogin without-password<br>PermitRootLogin yes<p>$ sudo service ssh restart<p>$ sudo passwd root</p></blockquote><p>设置ssh无密码登陆<p>//在主节点上执行一路回车,生成无密码的密钥对<p>$ ssh-keygen -t rsa<p>//将生成的密钥文件复制到其它节点<blockquote><p>$ ssh-copy-id cdh02<br>$ ssh-copy-id cdh03</p></blockquote><p><b>测试:</b>在主节点上ssh hadoop2,正常情况下,不需要密码就能直接登陆进去了。<h4>3.安装Oracle JDK(所有节点)</h4><p>运行CDH5必须使用Oracle的Jdk,需要Java 7及以上版本支持。<p>在Oracle的官网下载jdk包,解压到相应目录,例如/usr/java/jdk1.8.0_121:<blockquote><p>$ sudo tar -zxvf /home/user/jdk.xx.tar.gz -C /usr/java/</p></blockquote><p>配置环境变量,配置一个全局的JAVA_HOME变量:<blockquote><p>$ sudo vi /etc/profile<p>export JAVA_HOME=/usr/java/jdk1.8.0_121<br>export JRE_HOME=${JAVA_HOME}/jre<br>export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib<br>export PATH=.:${JAVA_HOME}/bin:$PATH<p>$ source /etc/profile</p></blockquote><h4>4.安装配置MySql(主节点)</h4><p>安装mysql服务器:<blockquote><p>$ sudo apt-get install mysql-server</p></blockquote><p>根据提示设置root的初始密码:<blockquote><p>$ sudo mysqladmin -u root password 'szgwnet'</p></blockquote><p>启动mysql服务:<blockquote><p>$ sudo service mysql start</p></blockquote><p>创建mysql数据库:<blockquote><p>$ mysql -uroot -pxxxxxx<p>#hive<br>create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;<p>#Hue<br>create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;<p>#Oozie Server<br>create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;</p></blockquote><p>设置root授权访问以上所有的数据库:<p>//授权root用户在主节点拥有所有数据库的访问权限<blockquote><p>grant all privileges on *.* to 'root'@'cdh01' identified by 'password' with grant option;<p>flush privileges;</p></blockquote><p>配置 mysql connector 驱动<p>MySQL Connector官方JDBC驱动地址:<p><a href="https://dev.mysql.com/downloads/connector/j/"><u>https://dev.mysql.com/downloads/connector/j/</u></a><blockquote><p>$ cd /usr/share/java<br>$ sudo ln -s mysql-connector-java-5.1.41.jar mysql-connector-java.jar</p></blockquote><p><img width="640" height="127" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5-jn-pPPE1HXJoAii0c7OJADJPc18GwMro2nNQa23s1LhV9uQnwOnL3Jda5Qt2wRHHFjvj7HZ31aJ1BVcDGnTfvJop9fgRYQbFlA9rj6yTsyI_lMGVLW61oTa4CjxBviCNyIQk5tR1HI/s1600/image%255B15%255D" border="0"><br>//也可以将jar包重命名为 “mysql-connector-java.jar”,多个版本时建议用软链。</p><h4>5.配置NTP服务(所有节点)</h4><p>集群中所有主机必须保持时间同步,如果时间相差较大会引起各种问题。这里以namenode节点作为ntp服务器与外界对时中心同步时间,随后对其它所有节点提供时间同步服务。Cloudera建议所有CDH节点都需要启动ntpd服务,要不然会报“时钟偏差”的错误。<p>安装NTP组件<blockquote><p>$ sudo apt-get install ntp</p></blockquote><p>配置NTP服务(管理节点)<p>在配置之前,先使用ntpdate手动同步一下时间,避免本机时间与标准时间差距太大:<p>//跟内网时间服务器或Internet时间<blockquote><p>$ sudo ntpdate -u 172.16.10.1<br>or<br>$ sudo ntpdate -u time.nist.gov</p></blockquote><p>NTP主要的配置文件为:/etc/ntp.conf,<p>$ sudo vi /etc/ntp.conf<blockquote><p># 设置用于校时的上层时间服务器<br>server 172.16.10.1 prefer<br>server time.nist.gov<p># 外部时间服务器不可用时,以本地时间作为时间服务<br>server 127.127.1.0<br>fudge 127.127.1.0 stratum 10<br>restrict 192.168.13.0 mask 255.255.255.0 nomodify<br>broadcast 172.16.13.255</p></blockquote><p>重启NTP时间服务:<blockquote><p>$ sudo service ntpd restart</p></blockquote><p>配置ntp客户端<p>NTP客户端配置主要将ntp server指向管理节点:<blockquote><p>//设置cdh01 为ntp server<br>$ sudo vi /etc/ntp.conf<br>server cdh01 prefer</p><p>//先手动同步一下时间,一般需要等待5-10分钟才可以正常同步。<br>$ sudo ntpdate -u cdh1</p></blockquote><p>重启NTP服务自动同步时间:<blockquote><p>$ sudo ervice ntpd restart<br>$ sudo cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime </p></blockquote><h4>6.配置本地硬盘(datanode)</h4><p>Datanode节点一般会配置多块本地硬盘(非RAID方式),这里需要提前对每块硬盘进行格式化和挂载。<p>格式化所有本地数据盘:<blockquote><p>//此处为实验环境新建一个lv卷模拟一块硬盘<br>$ sudo lvcreate -L 20G -n disk1 ubt1404-vg<p>//格式化为ext4<br>$ sudo mkfs.ext4 /dev/mapper/ubt1404--vg-disk1</p></blockquote><p>将所有磁盘挂载到指定目录:<blockquote><p>//尝试手工挂载是否正常<br>sudo mount /dev/mapper/ubt1404--vg-disk1 /ldisk/disk1<p>//编辑fstable,将所有数据盘分别挂载到ldisk目录下diskx<br>$ sudo vi /etc/fstable<br>/dev/mapper/ubt1404--vg-disk1 /ldisk/disk1 ext4 rw 0 0<br>/dev/mapper/ubt1404--vg-disk2 /ldisk/disk2 ext4 rw 0 0<p>//将所有盘进行挂载<br>$ sudo mount -a<p>//查看正常挂载后的文件系统<br>$ sudo df -ah<br>/dev/mapper/ubt1404--vg-disk1 20G 44M 19G 1% /ldisk/disk1<br>/dev/mapper/ubt1404--vg-disk2 20G 44M 19G 1% /ldisk/disk2<br>...<p>//如果data.dir 配置失败可以尝试手动创建dn数据目录。在每个挂载点下新建dn目录,并修改合适的目录权限<br>$ sudo mkdir /ldisk/disk1/dn<br>$ sudo chown hdfs.hadoop /ldisk/disk1/dn</p></blockquote><h2><b>安装CM(Server/Agent):</b><b></b></h2><p>CDH5.10之前版本可能需要手动创建cloudera-scm用户(所有节点)<blockquote><p>$ sudo useradd --system --home=/var/lib/cloudera-scm-server --no-create-home --shell=/bin/nologin --comment "Cloudera Manager" cloudera-scm<p>$ cat /etc/passwd<br>cloudera-scm:x:105:112:Cloudera Manager,,,:/var/lib/cloudera-scm-server:/bin/nologin</p></blockquote><h4>1.安装Cloudera Manager Server(主节点)</h4><blockquote><p>$ sudo apt-get install cloudera-manager-daemons<br>$ sudo apt-get install cloudera-manager-server</p></blockquote><p>***在线安装过程较长,可将apt下载好的deb包(/var/cache/apt/archives目录下)copy出来,通过 dpkg -i –R <deb_dir> 进行离线安装。<p>完成安装后文件路径:<p>/usr/sbin/cmf-server<br>/usr/sbin/cmf-agent<p>初始化Cloudera Manager的数据库(手动)<blockquote><p><i>//mark: configure clouder manager metadata to saved in the MySQL.<br></i>$ sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql cmf -hlocalhost -uroot -pxxxx --scm-host localhost scm scm scm</p></blockquote><p><img width="640" height="92" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFfvKD1NUmJX6FD_1kbT6_YEOsJv4diLvYmWdACTvT0jxhm5pWlCq4TCo5ayoqdKwGxHr64ofvwU19QxmZb7QFLQI4O5wPRppbb_ivgWYVb5ikwNzJyHoWGwdR6yfpwvSIOnDTeIjVu7Q/s1600/image%255B25%255D" border="0"><br>***执行scm_prepare_database.sh 初始化数据库后不需要手动修改cm数据库配置文件。</p><p>修改cm数据库配置文件(手动)<blockquote><p>$ sudo /etc/cloudera-scm-server/db.properties</p></blockquote><p><img width="640" height="167" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6kvCtfIsHL3gJJC-65GnDXHLpO-gHY1pdGRr1OsZeeZig1cpDIyJNCiiHQ2xRUPn4NvsuWSm9tbfSXYe1ouD8w0yPsxQ7HNH4TSqa56sdQK977dbsknXnXA-HmvDvGrdllLZyfyRLSr0/s1600/image%255B20%255D" border="0"></p><h4>2.启动CM服务(主节点)</h4><blockquote><p>$ sudo service cloudera-scm-server start</p></blockquote><p>第一次启动服务时间较长,cmf数据库初始化,直到7180端口起来才启动完成:<p><img width="640" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhleaZBOLYqmsNDfDmCtThfzEFGyhUFwRWd5BrA3vMZoS0k_K4fCIv2zSXJ-ixjzCV6lFq7beAF1zPVXC8QS6ilOD5oTy30fBzaRcbP0P9IplYHAvOcEK0uvdDb_mXk6zuVYKWI2_N8OaE/s1600/wpsACF3.tmp5" border="0"><h4>3.安装Cloudera Manager Agent(所有节点)</h4><blockquote><p>//此步骤在cm添加节点时自动部署,这里可提前手动安装缩短部署时间。<br>$ sudo apt-get install cloudera-manager-daemons<br>$ sudo apt-get install cloudera-manager-agent</p></blockquote><p>***在线安装过程较长,可将apt下载好的deb包(/var/cache/apt/archives目录下)copy出来,通过 dpkg -i –R <deb_dir> 进行离线安装。<p>修改Agent配置文件<blockquote><p>$ sudo vi /etc/cloudera-scm-agent/config.ini <br>server_host= cdh01 //主节点的主机名</p></blockquote><h4>4.准备Parcels(离线部署CDH5)</h4><p>CDH5 parcel最新版下载地址:<p><a href="http://archive-primary.cloudera.com/cdh5/parcels/latest/"><u>http://archive-primary.cloudera.com/cdh5/parcels/latest/</u></a><p>相关的parcel文件如下(这里为CDH 5.10.1 for ubuntu 14.04版本):<blockquote><p>CDH-5.10.1-1.cdh5.10.1.p0.10-trusty.parcel<br>CDH-5.10.1-1.cdh5.10.1.p0.10-trusty.parcel.sha1<p>//注意将CDH-5.x.x-x.xxx.xx.parcel.sha1,重命名为CDH-5.x.x-x.xxx.xx.parcel.sha,否则,系统会重新下载CDH-5.1.3-1.cdh5.1.3.p0.12-el6.parcel文件。</p></blockquote><p>将下载的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中(parcel-repo需要手动创建):<p><img width="640" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOPr3tQst797pY_QoQeWmtB4EK4GDRGV249IwpevAsy2Kipyqj7Bpfe4zTU16oOsGVcpVE5dNTZI0wMbf7JALiYY6RGVoyFBvU2bjLVsOv7eOvJOtmw_GO7szxFB3BOszvrIAdLkiHkrc/s1600/wpsACF4.tmp5" border="0"><h2><b>通过CM部署Hadoop集群:</b><b></b></h2><h3>1.配置集群主机</h3><p>Cloudera Manager 浏览器认管理账号:<p><a href="http://172.16.13.11:7180"><u>http://172.16.1</u><u>3</u><u>.</u><u>11</u><u>:7180</u></a></p><blockquote><p><b>U</b><b>ser</b>:admin <br> <b>P</b><b>assword</b>:admin</p></blockquote><p><img width="640" height="322" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfM-1FuO8Az9VFGfhhA_3AuPWMZ9Xb_HtV3KXF2snYys6rOWWfmDBlsimRkwpGoL5HRJxreYN3yJUjJzvcyerJ3nXxcvd-sicOzs1AaHIeEuoFCzpU2B-fzgNe3S17rp1zGwqsjtMhwPc/s1600/wpsACF5.tmp5" border="0"></p><p>首次登陆进入集群安装向导:<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvRqZh1yDc0fEyS5z7RKicWIbZcOczcLbLd4XuLGYDQoP53umMLxUbB1N2LPp5uVJxTzHmXF8VNLuKrQx69ztyF_r_9UDyz4lSCjhlVHCoZjaozd7ahUleRQUjOPmx2PAgfP9n-nVXKi4/s1600/wpsAD06.tmp5"><img width="640" height="352" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvRqZh1yDc0fEyS5z7RKicWIbZcOczcLbLd4XuLGYDQoP53umMLxUbB1N2LPp5uVJxTzHmXF8VNLuKrQx69ztyF_r_9UDyz4lSCjhlVHCoZjaozd7ahUleRQUjOPmx2PAgfP9n-nVXKi4/s1600/wpsAD06.tmp5" border="0"></a><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrgcw0wJQwgg3NkN-Zrb7p77yoG7lOHV0PZnNd0Lsx343c-bEBlhyphenhyphenWb95s2q2tOWRnYZXE5tv2KfnFoV1O6VqZZlA9d3G8vmjjghE1eJvwqVlte6tj4HwloUZEhpPfDvMsp4_XLqWDr5U/s1600/wpsAD07.tmp5"><img width="640" height="344" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrgcw0wJQwgg3NkN-Zrb7p77yoG7lOHV0PZnNd0Lsx343c-bEBlhyphenhyphenWb95s2q2tOWRnYZXE5tv2KfnFoV1O6VqZZlA9d3G8vmjjghE1eJvwqVlte6tj4HwloUZEhpPfDvMsp4_XLqWDr5U/s1600/wpsAD07.tmp5" border="0"></a><h3>2.Parcels部署前的准备</h3><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinr87dbrjznhunaCYOOkna-kgtODtRrHpeSHt2MGf46wkfeFAWhRz_z8RM0dlBqfXsFiyN5sJc6tM0EfnC-Mv5KbNCVOHgvlBt9Qa3dFsjJNeVvrYT4aggWy9PQnYZpCSzj22eu3M6FLs/s1600/wpsAD08.tmp5"><img width="640" height="453" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinr87dbrjznhunaCYOOkna-kgtODtRrHpeSHt2MGf46wkfeFAWhRz_z8RM0dlBqfXsFiyN5sJc6tM0EfnC-Mv5KbNCVOHgvlBt9Qa3dFsjJNeVvrYT4aggWy9PQnYZpCSzj22eu3M6FLs/s1600/wpsAD08.tmp5" border="0"></a><p>//这里不用再选择安装JDK<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaJm1vrkAULDRVye1JILPcCMMBMWyZcX9qRIKyOhTVBalpwTKt7vUG78yfCBVZvFzBCqxLm1_aOorAFCm26dQvlb03VBIRar5TSHcsayfadNh0RgEKr1dOmn7HwgrWzNT8MsemOuHykvc/s1600/wpsAD09.tmp5"><img width="640" height="441" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaJm1vrkAULDRVye1JILPcCMMBMWyZcX9qRIKyOhTVBalpwTKt7vUG78yfCBVZvFzBCqxLm1_aOorAFCm26dQvlb03VBIRar5TSHcsayfadNh0RgEKr1dOmn7HwgrWzNT8MsemOuHykvc/s1600/wpsAD09.tmp5" border="0"></a><p>//选择传统模式安装,不启用单用户模式<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwr8w9Aai1If01eE_6wxF5r2d_-gCE6jwEiqy84MrHLmt-LvA7TEGdigVmwF5i79GdD1GKerTXC_2PeR_6ctlK7uCaWelSWaMjYrmK9MbIMeN4iYd7DVKBiV8wMcvkROrUg-Wbg9PXwDY/s1600/wpsAD19.tmp5"><img width="640" height="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwr8w9Aai1If01eE_6wxF5r2d_-gCE6jwEiqy84MrHLmt-LvA7TEGdigVmwF5i79GdD1GKerTXC_2PeR_6ctlK7uCaWelSWaMjYrmK9MbIMeN4iYd7DVKBiV8wMcvkROrUg-Wbg9PXwDY/s1600/wpsAD19.tmp5" border="0"></a><p>//CDH5.10这里请直接选择以root 用户(CDH会自动为不同的服务建立不同的用户):<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMAokBab3XMV5JvHQE1Klttfka5UdGdzbKc7PA2YuXTSsp4YGCNGd_sgVNe586ODnCNZk3Ho1tlXQQGsqQm9xSy75uvHvZgs2wWEQAGX_0oxKDUUWUf7Ci63yanJtU-PT8CCD6h7YSYXI/s1600/wpsAD1A.tmp5"><img width="640" height="446" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMAokBab3XMV5JvHQE1Klttfka5UdGdzbKc7PA2YuXTSsp4YGCNGd_sgVNe586ODnCNZk3Ho1tlXQQGsqQm9xSy75uvHvZgs2wWEQAGX_0oxKDUUWUf7Ci63yanJtU-PT8CCD6h7YSYXI/s1600/wpsAD1A.tmp5" border="0"></a><p>//这一步安装是为每个节点安装cloudera-manager-daemons 和 cloudera-manager-agent,前面已经手动安装了,所以很快完成:<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQ0hKGdL45gua6Wr21ji0rpKeTaU6G0THMEWHamXeiuleWiVOTpVjkzugn_vPaEepz5MQiYERc1quk1w9Dji2o7jarKGm2nzrbkHKvmjKSH2H1lhD-bteubGKn57SsHscuf7gH3Llz31s/s1600/wpsAD1B.tmp5"><img width="640" height="387" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQ0hKGdL45gua6Wr21ji0rpKeTaU6G0THMEWHamXeiuleWiVOTpVjkzugn_vPaEepz5MQiYERc1quk1w9Dji2o7jarKGm2nzrbkHKvmjKSH2H1lhD-bteubGKn57SsHscuf7gH3Llz31s/s1600/wpsAD1B.tmp5" border="0"></a><p>//将CDH parcels分发到各个点等待安装:<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUlh2ZzkOQtCJl3BZ3kuihrFMgzV6UOshyN0UR8E3ulZUS-hu-_Plz16ppaBEbdy21MMWdGmrToty2Q3fvgYYwiCUUSk2ycY3xODiPEn14a_S5bYi2ImOzoFGGVr-UusPFSOFLgw1ECOs/s1600/wpsAD2C.tmp5"><img width="640" height="366" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUlh2ZzkOQtCJl3BZ3kuihrFMgzV6UOshyN0UR8E3ulZUS-hu-_Plz16ppaBEbdy21MMWdGmrToty2Q3fvgYYwiCUUSk2ycY3xODiPEn14a_S5bYi2ImOzoFGGVr-UusPFSOFLgw1ECOs/s1600/wpsAD2C.tmp5" border="0"></a><p>//主机配置条件兼容性检查,满足所有条件后再开始下一步的安装:<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAcXaK17fQkezDVsFctVOGi6hWjmFmMOUfTYHBuCLhRVQ3hObeIeYqAPGcAPbGmW6V2SegpK0BTRTKgvq7MdxYNkyGH0HVgOEvvD3WNGB5kTGXW-4XEm_M5NygBPtG6SfA3e7625YfdeM/s1600/wpsAD2D.tmp5"><img width="640" height="445" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAcXaK17fQkezDVsFctVOGi6hWjmFmMOUfTYHBuCLhRVQ3hObeIeYqAPGcAPbGmW6V2SegpK0BTRTKgvq7MdxYNkyGH0HVgOEvvD3WNGB5kTGXW-4XEm_M5NygBPtG6SfA3e7625YfdeM/s1600/wpsAD2D.tmp5" border="0"></a><h3>3.CDH集群安装</h3><p>//选择Hadoop集群安装组件<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmSitXQiUOQdoIkjodsXSzYkGYQllO19oTAs7aTvwcOZKTKjSKjODbo4_YSt8h4V6qeK9sHwxsl0jxGe_xcgXhJBWMvVpwtORIsEJ8O0RKICndvjTfEz-dILzxzrJXZKnejf-YkF7HSuA/s1600/wpsAD2E.tmp5"><img width="640" height="329" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmSitXQiUOQdoIkjodsXSzYkGYQllO19oTAs7aTvwcOZKTKjSKjODbo4_YSt8h4V6qeK9sHwxsl0jxGe_xcgXhJBWMvVpwtORIsEJ8O0RKICndvjTfEz-dILzxzrJXZKnejf-YkF7HSuA/s1600/wpsAD2E.tmp5" border="0"></a><p>//将个组件/角色合理的分配到不同的主机上:<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDHwBJYcl_GrV3mP_-oJxYsLIuwKRAgk4JRJ7gBG9Cur6S4LH8xR8IYBDcI3VmnyIbTsbH-bxCfKocSA9-8brBLkoiROMfDg12eFxMydvTLCZg6OsnmLGO1mxDYCvnJ-C9PadqUzIhRK0/s1600/wpsAD2F.tmp5"><img width="640" height="432" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDHwBJYcl_GrV3mP_-oJxYsLIuwKRAgk4JRJ7gBG9Cur6S4LH8xR8IYBDcI3VmnyIbTsbH-bxCfKocSA9-8brBLkoiROMfDg12eFxMydvTLCZg6OsnmLGO1mxDYCvnJ-C9PadqUzIhRK0/s1600/wpsAD2F.tmp5" border="0"></a><p>//管理节点承担了较多的角色,建议内存配大一些(角色后续也可以根据需要分配到其它节点):<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaoyYou3tTAmME_OLTCpqqiebFWLj2gRBcWMIBwngpYiDP61603fU-8vAec_SADvGpVlD1EmBf9ZX2cbZ0aa4V6sD9qhjUkQqr_LRrmojDjtsce9Fbf60VSZEdRL4R06pZz3FGWxek2jA/s1600/wpsAD40.tmp5"><img width="640" height="216" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaoyYou3tTAmME_OLTCpqqiebFWLj2gRBcWMIBwngpYiDP61603fU-8vAec_SADvGpVlD1EmBf9ZX2cbZ0aa4V6sD9qhjUkQqr_LRrmojDjtsce9Fbf60VSZEdRL4R06pZz3FGWxek2jA/s1600/wpsAD40.tmp5" border="0"></a><p>//这里database主机名填 localhost 验证通过:<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGsuVtyNH0Ji0XKO5HdeJ1F9PduvShO6Dc6sSTzvlfk3bIiWxLlADeIlfJXvy6SrvrScrKlYxmbQegWJlSGFNtel6sew3kkIrLlbVs1ELd1uz0WIe0CJzk-s4yicrwPGA4ZDZcoKn8oU8/s1600/wpsAD41.tmp5"><img width="640" height="446" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGsuVtyNH0Ji0XKO5HdeJ1F9PduvShO6Dc6sSTzvlfk3bIiWxLlADeIlfJXvy6SrvrScrKlYxmbQegWJlSGFNtel6sew3kkIrLlbVs1ELd1uz0WIe0CJzk-s4yicrwPGA4ZDZcoKn8oU8/s1600/wpsAD41.tmp5" border="0"></a><p>//预览所有配置信息,规划好DataNode和NameNode的数据存储路径(后续也可以进行调整):<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg98I88odaWEup1vPNZQi90UfRt0mBTz5rDV3vCCYsqmpcb_u7uQO54gD2oOe4LUwD52pcjkI95MI6n7TkHoIZaBIHkgRXNCMd3bjV_8O45M9O60ylqnXCqoT-d_osUtxw9GmQnZk6Mho4/s1600/wpsAD42.tmp5"><img width="640" height="441" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg98I88odaWEup1vPNZQi90UfRt0mBTz5rDV3vCCYsqmpcb_u7uQO54gD2oOe4LUwD52pcjkI95MI6n7TkHoIZaBIHkgRXNCMd3bjV_8O45M9O60ylqnXCqoT-d_osUtxw9GmQnZk6Mho4/s1600/wpsAD42.tmp5" border="0"></a><p>//确认配置后,开始集群的安装:<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEzRG79h64JC00yWwsrRyGe10HbAVHm9nyEMng6gtjK6O6p3ceTRCnkiYmyT1VDlM9kJZd_duerw5k7wCHXDQ0GoxYHMUBrYNIJVgnCrr213CA2N7Z_Z7ZLRMapORfRK7QbeVA3sLZpZQ/s1600/wpsAD52.tmp5"><img width="640" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEzRG79h64JC00yWwsrRyGe10HbAVHm9nyEMng6gtjK6O6p3ceTRCnkiYmyT1VDlM9kJZd_duerw5k7wCHXDQ0GoxYHMUBrYNIJVgnCrr213CA2N7Z_Z7ZLRMapORfRK7QbeVA3sLZpZQ/s1600/wpsAD52.tmp5" border="0"></a><p>//这个过程时间相对较长,遇到报错根据log的内容排查问题,然后在retry会继续进行,如果不小心关掉了这个页面可以在Running Commands 里找到First Run 这个任务继续:<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcABNgOdXaXhyphenhyphenwBMp32cmUUrj3KlMKfIj6ARsHhtjNr3t3Ws0RhixTFccoNYeBxud6lEsQrkwDfW-_l29FcWm5-kH-dKr5h9EjK2Gf3Z8XQynX8rWtbqWYkvbW7pP92zjbvnA1tuvUc8o/s1600/wpsAD53.tmp5"><img width="640" height="412" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcABNgOdXaXhyphenhyphenwBMp32cmUUrj3KlMKfIj6ARsHhtjNr3t3Ws0RhixTFccoNYeBxud6lEsQrkwDfW-_l29FcWm5-kH-dKr5h9EjK2Gf3Z8XQynX8rWtbqWYkvbW7pP92zjbvnA1tuvUc8o/s1600/wpsAD53.tmp5" border="0"></a><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW1bCU6gRypMQ3s8SYBlIu9HH9SNSMOSYvcHbAsm8xM8rytbU6TEgyhnEoQR26Z9rwYUjiJY6bFfCzJx6v2mT4GP8UYksYC1exNigrwp3az_QK0ec9f40mQFGGfuhd9k87xUS92GAAtlQ/s1600/wpsAD54.tmp5"><img width="640" height="449" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW1bCU6gRypMQ3s8SYBlIu9HH9SNSMOSYvcHbAsm8xM8rytbU6TEgyhnEoQR26Z9rwYUjiJY6bFfCzJx6v2mT4GP8UYksYC1exNigrwp3az_QK0ec9f40mQFGGfuhd9k87xUS92GAAtlQ/s1600/wpsAD54.tmp5" border="0"></a><p>完成Hadoop集群的安装:<br><img title="wpsAD55.tmp_thumb2" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="wpsAD55.tmp_thumb2" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2m1Hi7F1XhyphenhyphenL9y1sy_8AsJfQFU3Ec-7s3wf37AAwgHfjp5rodhOZ9aiOIU8wMZu00TJdY0SW9pu5YVbIbFsrcM4SLDGuCNAYZUBNog_ZASJ0DMyVaiPUaY0QIu4u6kgmx5fl4dl37vYA/?imgmax=800" border="0"><h2><b>Hadoop集群配置相关</b><b></b></h2><h3>Hadoop集群计算测试</h3><p>登陆到集群中任意节点,以hdfs用户执行测试任务(用Hadoop计算PI值,圆周率)<blockquote><p>$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 10000</p></blockquote><p><a style="margin-right: 1em; margin-left: 1em;" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIXO21WlDIE0SuHScyvZZmC7kgeJDaAM9vDbFJErTPdUN1tMLRzJIDVJUbVp3LRJBA5jHl6KTGCE0Zj_mKRAemcbJt6DAUzJslu-JJemC8vB2vX0G_CNxBwUy4H9BJCIlzVPWZyU5iaOo/s1600/image%255B10%255D" imageanchor="1"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1f0OA-UT0iQ9VwnXN0lpUI3pq3_-ldDJDer3HymZigy3loe24hMhGk5P0QR4Aj-U0HM00nMxx6EBOZF_R6zURYq6VWCHx2JxpZhiZMmQk0WIYOozHUrHZ0zaVK8sxQ7nDN719QYclpIY/s1600/image_thumb%255B4%255D" border="0" data-original-width="640" data-original-height="390"></a><br><a style="margin-right: 1em; margin-left: 1em;" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbAnMtdQcKpPnBJ-0MF3p2XPCx2rR3CMEbLPv5uwqU-f0iibXgaH3htlY-uYf0FJZSZOa8FjJZW5cbjOs0ufLQ7FI-8cVo1PBty_LpkFzZjIQ-C8ua7rg4VL9Ssiut8KYcGDeBqqB6U3Y/s1600/image%255B27%255D" imageanchor="1"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgr7gkLUq2DnnCQc4zY4M-r6BrtNThID8NT0yAhOr8lJixou-mVPRphb7uDq8HTGoCzY5KaJbsPfTT07lBxkm1vt5226EmtmuXVcTUnmvOeybKC_-673lFZGkIN86XHwDSKmGRrmRBDHVw/s1600/image_thumb%255B11%255D" border="0" data-original-width="640" data-original-height="352"></a><h3>配置HDFS数据目录</h3><p>在HDFS配置页面找到HDFS DataNode 存储目录(dfs.data.dir/dfs.datanode.data.dir)<p>前面已经为每个磁盘挂载到一个目录下,这里将这些目录都添加上,需要注意这里不能直接添加挂载点的目录,需要指向挂载点下的一个子目录,例如 /ldisk/disk1/dn<p><img width="640" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggx54YvSzVzP-R00ZJ1G6o6BRXLnrd2mX3x7VUP2T7jjf9LXhRmeNd5cz8PtjbvaxU1gw8fJmu3dz4aeSxZTYcZoGJuH4P3gHp6ygjpeyuYe9OZXLZgKlHn6DsegFztQXbVe8cystAMYs/s1600/image%255B29%255D" border="0" data-original-width="823" data-original-height="309"><p>集群重新配置后,可以看到HDFS容量已经得到增加:</p><p><a style="margin-right: 1em; margin-left: 1em;" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZCU2EkLiFCZwuDwP3opbXMoFWZEMCdBXSeeo3ZAz1WKw1udR0UdCcfBA0Y-rVupZQsZ9ZgTuZIZ5xO3vSzzqkLJJ75iB7pweoSv6coJeLOtgDTIn2L7g31LC93lN7-FVYr6fEbYyJMTA/s1600/image%255B33%255D" imageanchor="1"><img width="640" height="428" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZCU2EkLiFCZwuDwP3opbXMoFWZEMCdBXSeeo3ZAz1WKw1udR0UdCcfBA0Y-rVupZQsZ9ZgTuZIZ5xO3vSzzqkLJJ75iB7pweoSv6coJeLOtgDTIn2L7g31LC93lN7-FVYr6fEbYyJMTA/s1600/image%255B33%255D" border="0" data-original-width="823" data-original-height="551"></a></p><h3>配置HDFS副本数</h3><p>在HDFS配置页面,通过dfs.replication 筛选出副本数配置项目,默认为[3]:<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjM2Nfq1iL9Nk1ury4ndJU6bUumsrIHMLyfLgIWkM6VkxrHZsuil-2k9NQdqtlInfESJ3yeBEBn5ZMbCEmCCncybx_8Pv6N9GLIfwO-AnsVrvf9YCL1O2DvBUr3JHMo5fNnxCf1GLqtBfs/s1600-h/image%255B10%255D"><img width="640" height="354" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuxvCjlhEb_1d7LehvkxZqAgSFCH0VL4IXon90cqh7yrZvyLkkMdcMwpERzv00BPa8lVqr459_gNQPS2FwTVwjfft7f1DTpYs0JEDEaTE06Gn8IUI5AyFyndVQdnHxxjJSEZeZTyB4aPw/?imgmax=800" border="0"></a><h3>新增服务到集群</h3><p>选择集群,操作,添加服务<p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3fdVenKVBA4kuxBNy2cDUG14i6om4N_2NnJKZcA4c9IlbCZJdGfBmJP118vq9ixxS25pwuRb5cLrCRpaOgW4FzHAAT7l6TUg-Q2Wqws9G-Ej1esbp2L4YiEyS5596wEspOQD3xPm5vxE/s1600-h/image%255B8%255D"><img width="640" height="355" title="image" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgv3DL5FvHTh9VcJ3EM2jjQ5WDqO8KE6cj0ie0Y-lgLaU5YAj6y4LYMHWjzbwUSHhvByUa7xXJhy7I7ASzyXST_y-qa8iFI1YzVW7pFj9GopNJp8J_A-iLILUrOiLGICAuGOJuG5C2-EqQ/?imgmax=800" border="0"></a></p>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-49274731914207840822016-07-16T21:48:00.001+08:002017-09-22T23:47:16.738+08:00Python爬虫相关笔记<p>前段时间尝试用爬虫做个差评信息平台“MUSEE”,对折腾Python爬虫的一些过程记录下。</p>
<h2>资源调研</h2><p style="text-indent: 2em;">资源调研主要是做对于要爬的线上数据进行调研,弄清楚站点的URL、数据链接、数据标签等,便于后续的数据抓取。</p><p style="text-indent: 2em;">日常用google chrome比较多,通过F12 Development Tool 分析页面请求过程。以爬tmall的用户评价为例,打开任何一个商品链接,在页面加载过程通过监控 network 刷出来的内容,通常一大堆的东西很难区分出哪个内容加载的评论数据,这里有个小技巧,点击下一页在页面加载下一页评论的过程中,在新加载的数据部分更容易发现目标。如下图,就是前两页评论的数据:</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAjftQH7wQhkKTmEHxZS80Cb41JNISCDfvqWCZ31Zrz3geLj-u6QVZ1blaELTzdDZCv1U8RkpW6hbnE1YFOm0SSmX0uF3ZJCND1NJ0AaDk4em9GZcLHCArHMetQkBKA7JTj6gLCw4woGI/s1600-h/image%255B31%255D"><img width="684" height="269" title="image" style="display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfGxPEMkLRfuB3veGQk3QVW3fHBqGBtCSupt2mBVULZPa2h6CrZbhCtusxCRlVWKbUxSVQkKpHWhxQatPGY6hmt7M7Wry4j2S6dqGk2jZ2xQEPPrdMwGO8u_t-EBmgV8s0HJ2bFmnvQq8/?imgmax=800" border="0"></a></p><p>多刷几页会发现商品的评论数据链接地址格则比较简单,通过指定“currentPage=x”进行翻页,tmall的评论数据格式比较友好,json内的所有数据清晰可见,例如;itemId是商品id,sellerid是卖家id,currentPageNum是页面号,conten是商品评价内容 …</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEic1rypbqIrjT8THkZ-yFWrvEvjNF1V5WzD91J7MxSOLGPGz9lRWIejcQXrkdib7109lzguPS9CZn3GVmMiUtmfSlnIN0lICqtAa5wpcPZb55LABBkCrm7R7tqPCXX_0V8xNJMs9NGzYNo/s1600/json1081.png"><img width="684" height="303" title="image" style="display: inline; background-image: none;" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEir1neER8Uila_g3_2n4wnMJU0HWyQQiqsf-CY-jYnc1CdIcgMma0-Fmmed-Fa1VKUppVbo5cT2p-QcqsCvVctltO6plCaR3jj6pJpKr0ton2nmzcTW0N93aLpn2iVY1bVSct7Ut9lhGyw/s1600/json1081_thumb.png" border="0"></a><br><br></p><h2>爬页面</h2><p style="text-indent: 2em;">爬页面主要是通过模拟页面请求将需要的web内容抓出来(主要是html数据),用到Python的urllib2 模块。urllib2 用一个request对象来映射web访问请求,通过urlopen函数将请求转成可操作的文件对象。</p><h4>http请求</h4><p>大部分用到的是http请求。</p><h4> ftp 请求</h4><p>主要是爬图片或其它文件用。</p><h4>异常的处理,常见HTTP状态码:</h4><blockquote><p>"100" : Continue <br>"200" : OK<br>"403" : Forbidden <br>"404" : Not Found <br>"502" : Bad Gateway </p></blockquote><p><strong>Timeout 设置</strong></p><p>爬的时候根据实际情况,注意合理设置timeout,避免网站响应太慢造成的影响</p><h2>扣内容</h2><p style="text-indent: 2em;">类似ps的抠图一样,也需要从整个页面html文件中扣出所需的有效数据。扣内容主要用到re模块以及恰当的匹配规则。</p><h3>re模块</h3><p>使用re模块(re.findall ),的一般步骤是:</p><ol><li>将正则表达式的字符串形式编译为Pattern实例</li><li>用Pattern实例处理文本并获得匹配结果(一个Match实例)</li><li>用Match实例获得信息,再进行其他的操作</li></ol><h3>规则匹配</h3><ul><li>正则表达式,是做规则匹配的重要方法</li><li>CSS选择器是圈定范围的必备供手段。</li></ul><h2>页面验证</h2><p>对于http请求,有些网站内容需要登录或者传cookie才能访问,注意正确的处理。</p><h3>Cookie 的处理</h3><p>需与cookielib模块配合,urllib2 对 cookie 处理是自动的。(urllib2.build_opener)</p><blockquote><p>import cookielib </p><p>cookie = cookielib.CookieJar() </p></blockquote><h3>用户登录</h3><p>urllib2.Request的时候加入data参数,将账户信息定义为字典通过data传入,POST给服务器。<br></p><h3>反防盗链</h3><p>通过设置Headers属性进行伪装,常用设置项:</p><blockquote><p>headers = { <br>'User-Agent' : 'Mozilla/4.0',<br>'Referer':'http://www.xxxx.com/mall',<br>Content-Type : xxx,<br>application/json: xxx, }</p></blockquote><h2>数据存储</h2><p>如果是图片等文件,直接存文件系统上;</p><p>像评论等文本内容,做格式化后存成csv文本文件或者json格式</p><p><br></p><p>++++++++++++++++++++++++++++++++++++++++++</p><h2>爬虫框架Scrapy</h2><p>对于大量数据爬取可以用爬虫框架,在数据格式化、数据保存等方面比urllib省事。大体过程差不多:</p><h3>1、爬URL</h3><p>通过定义Spiders实行对web内容的抓取,将html内容保存文本文件。</p><h3>2、目标数据筛选</h3><p>Scrapy里,可以使用XPath的selectors,基于 XPath表达式。Selectors 有四种基础的方法:<a href="http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector.xpath"><br></a></p><ul><li><a href="http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector.xpath" target="_blank">xpath()</a>:返回一系列的selectors,每一个select表示一个xpath参数表达式选择的节点<a href="http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector.css"><br></a><li><a href="http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector.css" target="_blank">css()</a>:返回一系列的selectors,每一个select表示一个css参数表达式选择的节点<a href="http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector.extract"><br></a><li><a href="http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector.extract" target="_blank">extract()</a>:返回一个unicode字符串,为选中的数据<a href="http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector.re"><br></a><li><a href="http://doc.scrapy.org/en/latest/topics/selectors.html#scrapy.selector.Selector.re" target="_blank">re()</a>:返回一串一个unicode字符串,为使用正则表达式抓取出来的内容</li></ul><p>获取文本:sel.xpath('//ul/li/text()').extract() <br>获取超链接:sel.xpath(<a href="mailto:‘//ul/li/a/@href').extract">‘//ul/li/a/@href').extract</a>() <br>结合css选择器:sites = sel.xpath('//ul[@class="main-post"]/li')</p><h3>3、数据格式整理</h3><p>对爬到的数据存放到Item对象中,方面后续做各种数据操作:</p><pre><code>for site in sites:
item = DmozItem()
item['title'] = site.xpath('a/text()').extract()
item['link'] = site.xpath('a/@href').extract()
item['desc'] = site.xpath('text()').extract()
items.append(item)
return items</code></pre><h3>4、数据保存<strong>Pipeline</strong></h3><p><a href="http://doc.scrapy.org/en/0.14/topics/feed-exports.html#topics-feed-exports" target="_blank">Feed exports</a>,主要有四种:JSON,JSON lines,CSV,XML。</p><p>保存为json: scrapy crawl dmoz -o items.json -t json </p><h3><br></h3><p>++++++++++++++++++++++++++++++++++++++++++++++++++</p><h2>其它</h2><h3>Pandas模块</h3><p>Pandas也是常用的一个模块,主要两个用途:</p><p>1、用于数据整理,可以直接读取JSON,转成规范的DataFrame</p><p>2、作为数据分析工具</p><h3>Selenium + Web Browser</h3><p>Selenium模拟操作的方式爬web虽然效率比较低下,但不容易遭到封杀。有些不容易爬到又很重要的数据,可以试试 Selenium 。</p>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-79441180353369081652016-06-12T20:51:00.001+08:002017-08-31T22:11:51.549+08:00关于VMware虚拟网卡网络问题的处理<h2>本篇适用情形:</h2>
<ol>
<li>VMware虚机拷贝或移动后网络问题</li>
<li>其它虚机格式转换VMware虚机后网络问题</li>
<li>VMware虚机虚拟网卡设置问题</li>
<li>修改VMware虚机虚拟网卡类型</li>
<li>Ubuntu 16.04虚拟机网卡设置问题</li>
</ol>
Highlight:<br>
<ul><ul><!--StartFragment-->
</ul>
<li>/etc/network/interface //网卡配置文件(ubuntu)</li>
<li>lspci -vk</li>
<li>lsmod</li>
<li>ifconfig –a</li>
<li>ethtool</li>
<li>resolvconf -u</li>
</ul><h2>
正文:</h2><p>
【<strong>Step1】</strong></p><p>
在着手解决问题之前,先查看一下VMware虚机的网卡配置是否正常。打开虚机目录下的 xxx.vmx 文件,找到虚拟网卡的配置信息(第1块网卡通常为ethernet0),例如:<br>
<br>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRhSguDnkSBwwFxHYHfDiklSdjaIcHqH0fgqYzzuMwIcyOBNNg-9AvYcAYGuWqm42FyR-Sk9o6gfdGcVsfjtKow_ZbSyW_P6dqpcenNu2fv6BHOECqOr15yRAhyMkDJKVGijPVWM5lH70/s1600/image64" border="0" data-original-width="614" data-original-height="139"><br>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIxXKwe_Tx-I5WqHLnZXqlFy1DBb8lhiq-zWzge_9WLE30SISeAj4sCB-9HLUBP0usBiV88a9_8DJ9Ngi6mQTQ2cYGStXm9qtGrtLxJjruDmHKNhMpNZPeXhocuP14xjy-wMr9SoMZhBM/s1600/image4" border="0" data-original-width="617" data-original-height="110"><br>
<br>
VMware虚拟网卡有以下几种设备类型:<br>
</p><ul>
<li>
ethernet0.virtualDev = "vlance" //AMD PCNet AM79C970A 10/100Mbps</li>
<li>ethernet0.virtualDev = "e1000" //Intel(R) 82545EM PRO/1000 PCI</li>
<li>ethernet0.virtualDev = "e1000e" //Intel(R) 82545L PRO/1000 PCIe</li>
<li>ethernet0.virtualDev = "vmxnet" //VMware PCI Ethernet Adapter</li>
</ul>
<blockquote>
VMware Workstation / Player 建的虚机默认“vlance” 是一块比较老的百兆网卡,可以修改为e1000或e1000e(这两者的区别可以看做一块是pci网卡一块是pcie接口网卡);没有正常安装vmware tools的情况下,不建议改为“vmxnet”。</blockquote><p>
以下内容在Ubuntu 16.04 LTS虚机下进行。</p><p>
【<strong>Step2】</strong></p><p>
执行ifconfig查看网口配置信息,可能只有环回口的信息:<br>
<img width="617" height="183" title="image" style="display: inline; background-image: none;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwpRTsU_yF_si-_Am0XE-62y2V-I6CtJ_m5iIZbpt3iPJsSoShYdmQcjDVng-r5NBFj37l69r8BvWS1XY1gBcsxCXbZePnh0eJjpuO0hAmpRQfmlZPZ7ONJL-BnS7EIFmHxj4OfKOS30c/" border="0"><br>
执行ifconfig –a 查看系统识别到的所有网口,如果能查看到其它网口信息,例如:<br>
<img width="618" height="128" title="image" style="display: inline; background-image: none;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0fAYw4Gy7etzrB1VqJQgGp6hcz2iZoEGF3fP6smIisA7_WEcuA2DZZag9d_p46MNxB4p1Q_GUcuZ2u_cqBkosnyu2I61jT9L5cGQ3Q4LT9M3l8X19mpEuCtg-EmxFn2kiLBYZwDHckBw/" border="0"><br>
说明虚拟网卡的驱动是ok的,继续<strong>Step3</strong>。<br>
这是一块e1000e的虚拟网卡,根据Step1里的pci插槽配置信息 ethernet0.pciSlotNumber = "160" ,所以网口名称为 ens160。<br>
</p><blockquote>
ubuntu 16.04 网口命名会以网口的设备信息命名,如果希望恢复eth0、eth1的传统命名方式,可修改系统启动参数: <br>
<code># vi </code>/etc/default/grub <br>
<code>RUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"</code><br>
<pre><code># update-grub</code></pre>
<code>重启后即会回到eth0、eth1的网口名称。</code></blockquote><p>
如果ifconfig –a 依然只有lo环回口信息,确保<strong>Step1</strong>里的虚拟网卡配置正确,跳到<strong>Step4</strong>。</p><p>
【<strong>Step3】</strong></p><p>
检查网口的网络连接状态:<br>
$ sudo ethtool ens160<br>
<img width="616" height="387" title="image" style="display: inline; background-image: none;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLwNhgTIeA0yK_fIZXVWY0AiALoljTir_6iGeHZfq23InaxUfACL0TYxPzx7IFysmw7iLeM5PEsBU_NSgxpBmlqHT9et-TJZATmITKsFkw_6JUgPaRNS7q0WHen-5McUEustN5zBNtTtU/" border="0"><br>
最后一行如果是 Link detected: yes ,跳到<strong>Step5</strong>;<br>
如果是 Link detected: no,请检查虚机配置网卡设备状态是否为连接状态,如果勾选后检测不到连接再重启一下虚机。<br>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2zWpWRMSTZSje-zQFOIaeprsJZUm1CyAB4THxuSK_BsJK4Up_3iijBbThFGdEu00zDfxvNUrVyBv9An-bRFunZ6M7IWC4WB-QD85xxOQzr7XiHduNDr9vLtAiAQGm6GcckQyuzIM62XM/s1600/image87" border="0" data-original-width="669" data-original-height="373"></p><p>
【<strong>Step4】</strong></p><strong></strong><p><br>
查看虚拟网卡的硬件识别情况:<br>
$ sudo lspci –vk</p><p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLAAazad90WJRoMBmh_UFvALqwXHY_W33_aF9xx-PI3JZt4C-mPcFhRZXa7pyjl_ioIvm7StSGowm-w-xw1bNIVfgq6XoCVNLOQ2hmP9DGhxp9Zp0LLrbPn9aSE26CctRlgq-ulJotlhM/s1600/image112" border="0" data-original-width="694" data-original-height="273"><br>
</p><div class="separator" style="text-align: center; clear: both;">
</div>
<div class="separator" style="text-align: center; clear: both;">
<br></div><p>
上图是网卡驱动加载正常的情况,如果没有driver in use的信息,查看系统是否有驱动模块:<br>
$ sudo lsmod |grep e1000e</p><p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_YelY0LO-lQjLVKgEQ1QXut7By39ViK-5c2rhdl0xcX5rJrlIGtOHQ6DA14IMi7RxVjT4zdh9KtKqBHgLPCBMBmkJmxoUeRtF4X82xuY4vBFL19-3uq0dGiRS6uhTG8UIb9FU6FtBid4/s1600/image_thumb27" border="0" data-original-width="694" data-original-height="55"></p><p>
如果没有虚拟网卡的驱动模块,需要手动安装驱动(关于linux驱动安装步骤Google一下);或按<strong>Step1</strong>将虚拟网卡类型修改为其它类型进行尝试。<br>
解决网卡驱动问题后,通过dmesg可查看系统启动过程虚拟网卡的加载过程都是ok的:<br>
$ sudo dmesg |grep e1000e</p><p><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgShaxQ3LlKfSsggZIhn7rlBS8QFK-JCFtzQ7qv7DNjjkbS_aINT6gFS0fZK2y9LtJYY9K79-1LwdHHtzThyh5aPLGGpgA_qHM5-4VKDUGAoOpspKldLbad5III-idVjzdHptF7X08_des/s1600/image129" border="0" data-original-width="807" data-original-height="165">
</p><p>
【<strong>Step5】</strong></p><p>
检查网口配置文件,为系统识别的网口ens160配置ip地址(以配置静态IP为例):<br>
$ sudo vi /network/interface<br>
<img width="498" height="140" title="image" style="display: inline; background-image: none;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrM4bclIBgr0D4GmBNPwvo7aH1LRXiseInP6wtuKuKEzb-eHUiT5ViE5noQVljXX7KT4Sqf2FYDc74HDNyiA_fi9NGTeRhaWSAyFDPxT5cH7ESJP3d8vxLxGHhsI6rejKk7GOP5A2nFSA/" border="0"><br>
保存后,重启网络服务:<br>
$ sudo service networking restart<br>
$ ifconfig<br>
<img width="679" height="297" title="image" style="display: inline; background-image: none;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgItN5jWGCW-XlcxZMUZEmB5x4oL8YDSekZ7frr00ZUNZMdvWEaOMF8O5Jd31R_niTg1kVxirovGb4E32qDJOpPlEoHzMvrdCNQPCTpXYAOq2V0bVGfe-fG98nD0jxQv44kPiN_EDAE7UA/?imgmax=800" border="0"><br>
如上图所示,网口已工作正常。</p><p>
【<strong>Step6】</strong></p><p>
如果访问外网或域名地址不通,检查一下DNS的配置,对于ubuntu 14.x之后的版本,手动编辑/etc/resolv.conf 配置文件重启后会失效,改为修改配置文件:<br>
/etc/resolvconf/resolv.conf.d/base<br>
添加dns server地址,类似:<br>
nameserver 8.8.8.8<br>
nameserver xx.xx.xx.xx<br>
修改后命令更新一下即可:<br>
$ sudo resolvconf -u</p>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-21038540689754062512016-01-05T21:02:00.002+08:002020-05-23T23:40:25.470+08:00OpenSSL 申请证书<p>SSL 证书会涉及到的几类文件:</p> <p>privkey.key the private key for your certificate. <br /> identity.csr the Certificate Signing Request. <br /> certificate.crt the certificate file used in most server software. <br /> certificate.pfx the PKCS#12 format is an archival file that stores both the certificate and the private key. </p> <p>By default, OpenSSL generates keys and CSRs using the PEM format (the raw). </p> <p> </p> <h2>1,Generating a new Private Key</h2> <blockquote> <p>openssl genrsa -out private.key 2048</p> </blockquote> <p> </p> <h2>2, Creating Your domain CSR</h2> <h3>Creating Your domain CSR for an existing private key</h3> <blockquote> <p>openssl req -new -key private.key -out identity.csr \ <br /> -subj "/C=US/ST=Utah/L=Lehi/O=CREAST, Inc./OU=IT/CN=yourdomain.com"</p> </blockquote> <h3>or Creating CSR with One command:</h3> <blockquote> <p>openssl req --new \ <br /> -newkey rsa:4096 -sha256 –nodes --keyout private.key \ <br /> -out identity.csr \ <br /> -subj "/CN=Bitwarden IdentityServer" -days 10950 </p> </blockquote> <h3>Generate a certificate signing request based on an existing certificate</h3> <blockquote> <p>openssl x509 -x509toreq -in CAcertif.crt -out identity.csr -signkey private.key</p> </blockquote> <h3>Verifying CSR info</h3> <blockquote> <p>openssl req -text -in identity.csr -noout --verify</p> <p>verify OK <br /> Certificate Request: <br /> Data: <br /> Version: 1 (0x0) <br /> Subject: CN = hostone.myners.net <br /> Subject Public Key Info: <br /> Public Key Algorithm: rsaEncryption <br /> RSA Public-Key: (4096 bit) <br /> Modulus: <br /></p> </blockquote> <h2>3, <b>Generate a self-signed certificate</b></h2> <p>自签名证书主要在内网环境下使用:</p> <blockquote> <p>openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout private.key -out certificate.crt</p> <p>openssl req -x509 -sha256 -nodes -newkey rsa:4096 -keyout private.key \ <br /> -out certificate.crt -subj "/CN=Bitwarden IdentityServer" -days 10950 </p> </blockquote> <p> </p> <h2>4, Sending the CSR to the CA <br /></h2> <h4>Viewing Certificate Information</h4> <blockquote> <p>openssl x509 -text -in CAcertif.crt -noout</p> <p>Certificate: <br /> Data: <br /> Version: 3 (0x2) <br /> Serial Number: <br /> 04:be:c0:3c:1a:6e:c2:79:56:f6:3d:a5:26:63:51:fe:68:2c <br /> Signature Algorithm: sha256WithRSAEncryption <br /> Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 <br /> Validity <br /> Not Before: May 22 14:11:15 2020 GMT <br /> Not After : Aug 20 14:11:15 2020 GMT <br /> Subject: CN = hostone.myners.net <br /> Subject Public Key Info: <br /> Public Key Algorithm: rsaEncryption <br /> RSA Public-Key: (2048 bit) <br /> Modulus:</p> </blockquote> <p> </p>
<h2>5,Verifying Your Keys Match</h2> <p>To verify that your public and private keys match, use the -modulus switch to generate a hash of the output for all three files (private key, CSR, and certificate).</p> <div>Use the following commands to generate a hash of each file's modulus:</div> <div> </div> <p><code>// Verifying private key</code></p> <blockquote> <p><code>openssl rsa -modulus -in private.key -noout | openssl sha256</code></p>
<p>(stdin)= 5a07ccd5d1208a7f4e445414b841d100b85ac3138d47554be333fbd18d42a89c</p>
</blockquote> <p><code><code>// Verifying CSR</code></code></p> <code><code> <blockquote> <pre>openssl req -modulus -in identify.csr -noout | openssl sha256</pre>
<p>(stdin)= 5a07ccd5d1208a7f4e445414b841d100b85ac3138d47554be333fbd18d42a89c </p>
</blockquote>
</code></code><code>
<pre>// Verifying certificate</pre>
<blockquote>
<pre>openssl x509 -modulus -in certificate.crt -noout | openssl sha256</pre>
<p><font color="#ff0000">(stdin)= 81c6258063b5d2a7403f5ad6d3c490c79261402f9269f6a304e95f1ffde1eb04</font> </p>
</blockquote>
<pre> </pre>
<p>If the output of each command matches, then the keys for each file are the same. </p>
<p>if there is any mismatch, then the keys are not the same and the certificate cannot be installed.</p>
<p>Key mismatch errors are typically caused by installing a certificate on a machine different from the one used to generate the CSR.</p>
<p> </p>
<h2>6,Converting Certificate Formats </h2>
<p><b>Convert a PEM certificate file and a private key to PKCS#12 (.pfx .p12)</b></p>
<pre>openssl pkcs12 -export -out certificate.pfx -name "yourdomain-digicert-(expiration date)" \<br /> -inkey privateKey.key -in certificate.crt -certfile CAcertifi.crt \<br /> <code>-passout pass:IDENTITY_CERT_PASSWORD</code></pre>
<p><code></code> </p>
<p><code>openssl pkcs12 -export -name "yourdomain-digicert-(expiration date)" \
<br /> -out yourdomain.pfx -inkey yourdomain.key -in yourdomain.crt</code></p>
<p><b></b> </p>
<pre>// Verifying <b>PKCS#12 file</b></pre>
<pre>openssl pkcs12 -info -in keyStore.pfx</pre></code>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-57911490239342275502015-11-22T00:27:00.000+08:002017-09-22T20:57:01.074+08:00关于 Blogger Template 的一点基础知识<h2>模板布局</h2><h3></h3><p>曾经从blogspot转到WordPress,体验过WP的丰富和灵活型,但折腾下来也遇到很多问题,缺乏靠谱的WP托管商,自己维护web主机也花了不少精力,最后发现还是Blogger以不变应万变的姿态保持着它的简单和稳定,包括一直稳定地被和谐。。。</p><p>Blogger的一大痛点是内置模板都比较朴素,定制性不如WP等blog系统强大,这里对Blogger的模板文件相关内容进行说明。</p><p>整个Blogger的内容主要由板块section(下图中蓝色标签)和小部件widget(下图中绿色标签)组成,板块主要负责对页面划分区域,widget负责在每个区域里填充内容,所有的页面内容如站头、博客正文、侧栏等都放在对应的widget里面。</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwfPAwTPCjdQ4TSXE6jrYK0b_OXe-UHnLwbCdOKep-tjpwehsd19pbyLzR6g4NaxonnzctXQqRYeO1CS4PO9hBbaikW456g87-5f5FV2i6grN4UDfQRr9k3YiaELQy8wKVJwt8tc2fUGk/s1600-h/image3"><img width="604" height="500" title="Blogger Template" style="display: inline; background-image: none;" alt="Blogger Template" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiK69m5d9ikGEMUuxI6nFI_VYK4wLE0bQQ4et9f29Nw_Ny7BZrdmH1bpRFVJP7h2B640V289DD6ujU2Na-A_hK-0fofR_terA_UC34R6OXxsqeff5WrbFQP-SF54eHWrtIp8FHq6y7Ga7M/?imgmax=800" border="0"></a></p><h2>布局元素</h2><p>Blogger template 是一个 XHTML 文件,包含了常规的html、css、js 以及Blogger的专属元素标签(XML Namespace) ,上面用于布局的<b:section>和<b:widget> 就是属于其中。</p><h3>b:section 版块</h3><p>section版块用于划分网页的各个区域,如边栏、页脚。在blogger模板中,至少要有一个section标签,section标签内只能包含小部件<b:widget>标签,不能包含其它html元素,</p><h4>格式示例:</h4>
<pre><code><b:section id='header' class='header' maxwidgets="1" showaddelement="false"><br> <b:widget id='Header1' locked='true' title='Blogger' type='Header' visible='true'> … </b:widget-settings><br></b:section></code></pre>
<h4>section属性:</h4><table width="600" border="1" cellspacing="0"><tbody><tr><td width="238" valign="top"><p><b>属性</b></p></td><td width="477" valign="top"><p><b>说明</b></p></td></tr><tr><td width="238" valign="top"><p>id (必需)</p></td><td width="477" valign="top"><p>不能重复,只能包含字母和数字</p></td></tr><tr><td width="238" valign="top"><p>class (可选)</p></td><td width="477" valign="top"><p>常用class值:navbar、header、main、sidebar、footer。</p><p>更换模板时Blogger能根据class匹配转移块里的内容</p></td></tr><tr><td width="238" valign="top"><p>maxwidgets -(可选)</p></td><td width="477" valign="top"><p>设置版块中<b:widget>的数量上限</p><p>如果未指定,则表示没有限制。</p></td></tr><tr><td width="238" valign="top"><p>showaddelement -(可选)</p></td><td width="477" valign="top"><p>“true”/“false”,设置版块是否可以“添加页面元素”</p><p>默认为“true”。</p></td></tr><tr><td width="238" valign="top"><p>growth -(可选)</p></td><td width="477" valign="top"><p>horizontal: 小部件并排布置<p>vertical: 小部件堆叠布置,默认</p></td></tr></tbody></table>
<h3>b:widget 小部件</h3><p>widget小部件是blogger主要的内容载体,Blogger规定了几种widget类型不需要重新定义。<b:widget>里面一般会包含<b:widget-settings> 标签,用于对widget的设置项进行定义;widget中至少有一个Includable且id=mian。这个Includable为此widget的主体内容,将会被显示出来,而其余的Includable则不会被显示,但可以通过include调用。<br></p>
<h4>格式示例:</h4><pre><code><b:widget id='Blog1' locked='true' title='Blog Posts' type='Blog' version='2' visible='true'><br> <b:widget-settings><br> <b:widget-setting name='showDateHeader'>false</b:widget-setting><br> <b:widget-setting name='style.textcolor'>#009a61</b:widget-setting><br> </b:widget-settings><br> <b:includable id='main' var='this'>…</b:includable><br> <b:includable id='aboutPostAuthor'>…</b:includable><br><b:widget/></code></pre><h4>Widget属性:</h4><table width="600" border="1" cellspacing="0"><tbody><tr><td width="195" valign="top"><p><b>属性</b></p></td><td width="549" valign="top"><p><b>说明</b></p></td></tr><tr><td width="195" valign="top"><p>id (必需)</p></td><td width="549" valign="top"><p>每个小部件 ID 都应是唯一的,只能包含字母和数字</p></td></tr><tr><td width="195" valign="top"><p>type (必需)</p></td><td width="549" valign="top"><p>只能使用以下几种类型值:BlogArchive, Blog, Feed, Header, HTML, SingleImage, LinkList, List, Logo, BlogProfile,<p>Navbar, VideoBar, NewsBar</p></td></tr><tr><td width="195" valign="top"><p>locked (可选)</p></td><td width="549" valign="top"><p>“yes”或“no”,默认为“no”。锁定后小部件不能被移动或删除。 </p></td></tr><tr><td width="195" valign="top"><p>title (可选)</p></td><td width="549" valign="top"><p>如果未指定显示标题,Blogger会使用“List1”之类的默认标题。</p></td></tr><tr><td width="195" valign="top"><p>pageType (可选)</p></td><td width="549" valign="top"><p>取值“all”、“archive”、"main”或“item”,默认为“all”。<br></p></td></tr><tr><td width="195" valign="top"><p>mobile (可选)</p></td><td width="549" valign="top"><p>“yes”、“no”或“only”,默认为“default”。<p>该属性决定了小部件是否在移动设备上显示,当属性为“default”时,只有标题、博客、个人资料、页面列表、AdSense 和归属会显示在移动设备上。</p></td></tr></tbody></table><h3>b:include / b:includable</h3><p>b:include 和 b:includable 标配搭配使用,通过b:includable 定义代码内容,可包含html内容,在要用到该代码的地方使用 b:include 调用,可以在多个不同的地方重复引用。<p>widget中 id='main' 的Includable 会直接显示出来,不需要通过b:include 调用。<h4>格式示例:</h4><pre><code><b:includable id='postBody' var='post'><br> <div class='post-body entry-content container' expr:id='&quot;post-body-&quot; + data:post.id'><br> <data:post.body/><br> </div>
</b:includable></code><code><b:include data='post' name='postBody'/></code></pre><h4>b:includable属性</h4><ul><li><strong><font color="#ff0000">id</font></strong>(必选):由字母和数字构成的唯一标识符。每个小部件都必须有一个包含“id='main'”的 includable。
<li><strong>var</strong>(可选):由数字和字母构成的标识符,用于引用此部分的数据。</li></ul><h4>b:include属性:</h4><ul><li><strong><font color="#ff0000">name</font></strong>(必选):由字母和数字构成的标识符,必须与同一小部件中现有 b:includable 的 ID 相匹配。
<li><strong>data</strong>(可选):将传递到 includable 的表达式或数据块。这将成为 includable 中 var 属性的值。
<li><strong>cond</strong>(可选):只有结果为真时才执行 include 的表达式。该表达式与“b:if”的 cond 属性相同。</li></ul>
<h3>b:loop 循环</h3><p>利用 b:loop 标记,您可以多次重复某部分内容。此标记最常用于针对指定页面输出博文列表中的各篇博文,或者输出各条评论或各个标记等。<h4>格式示例:</h4><p>以下为使用循环的一般格式:<pre><code><b:loop var='i' values='data:posts'>
<h2><data:i.title/></h2>
</b:loop></code></pre><h3>b:if、b:elseif、b:else 条件判断</h3><p>利用 <strong>b:if</strong>、<strong>b:elseif</strong> 和 <strong>b:else</strong> 标记,可以在特定情况下展示特定内容,并在其他情况下展示其他内容。例如,通过条件判断实现在首页上显示特定文本,但在查看单独的博文时显示其它内容。<h4>格式示例:</h4>
<pre><code><b:if cond='condition'>
[条件为 true 时显示的内容]
<b:elseif cond='another condition'/>
[之前没有 if 或 elseif 条件为 true,但此 elseif 条件为 true 时显示的内容]
<b:else/>
[未满足 if 或 elseif 条件时显示的内容]
</b:if></code></pre><h3>b:switch, b:case 条件判断</h3><p><strong>b:switch</strong> 标记配合<strong>b:case </strong>使用,功能与使用包含多个b:elseif 的 b:if 标记大致相同,使用“switch”分支时,不需要重复变量名称。<h4>格式示例:</h4><p>以下举例说明了如何根据所呈现页面的类型输出不同的标头。</p><pre><code><b:switch var='data:blog.pageType'>
<b:case value="static_page" />
<h1>页面</h1><
<b:case value="item" />
<h1>博文</h1>
<b:default />
<h2>博文</h2>
</b:switch></code></pre>
<h3>b:with 变量</h3><p>利用 <strong>b:with</strong> 标记可以申明一个变量,用于暂时存储已计算表达式的值,避免复杂的内联表达式。<h4>格式示例:</h4><p>将bg-image 样式值暂存在style变量,通过 expr:style 进行调用。</p><pre><code><b:with <strong>var='style</strong>' value='"bg-image: url(\"" + data:sourceUrl "\"); " + " width: " + data:width + "px; " '>
<div id='header-outer'>
<div id='header-inner' <strong>expr:style='data:style'</strong>>
<h1>我的标题</h1>
</div>
</div>
</b:with></code>
<p>注意,只有<strong>b:with</strong>标记的子节点才能访问style变量。</p></pre><h3>data: 数据调用</h3><p>data:标记用于在模板中对Blogger内的变量数据进行调用,调用格式为:</p>
<pre><code><data: name /><br><data: name1.name2 /></code><p>“Name”是要使用的数据的名称;“name2”是名为“name1”的数据集合中的一项内容。</p></pre>
<p>Blogger中定义了很多数据集合及数据标记,这些标记拥有不同的有效范围,介绍一些常用的数据调用:<p>通用数据:可在Blogger任何位置调用<ul><li><data:pageTitle/>:当前页面的标题。通常是指博客标题,但可能包含文件或会话页面上的其他信息。<li><data:pageType />:当前页面的类型,可以选择“item”、“file”或“index”。 <li> <data:homepageUrl />:Blog首页地址<li><data:blog.title />:Blog的标题。</li><!--EndFragment--><li><data:url />:当前页面的网址。 </li></ul><p>Header小部件中使用的数据:</p><ul><li><data:title />:blog的标题
<li><data:description />:blog的描述。</li></ul><p>Blog小部件中使用的数据:</p><ul><li><data:posts />:此页面的博文列表<li><data:posts.body/>:博文正文内容<li><data:posts.id/>:博文ID<li><data:posts.title/> :博文的标题</li></ul><p>BlogProfile小部件中使用的数据:</p><ul><li><data:userUrl />:作者个人资料的网址。</li><li><data:aboutme />:“我的简介”个人资料信息</li><li><data:photo.url/> - Size: <data.photo.width /> × <data.photo.height /> :用户个人资料照片地址、宽度和高度</li></ul><blockquote><p>更多关于数据标签的内容参见Blogger的支持文件:</p><p><a title="https://support.google.com/blogger/answer/47270" href="https://support.google.com/blogger/answer/47270">https://support.google.com/blogger/answer/47270</a></p></blockquote>
<h2>模板样式</h2><p>Blogger模板样式包含 css 样式声明 和 样式相关的变量,存放在<head> </head>部分。</p><h3>CSS样式</h3><p>所有的css样式代码 都存放在 <code><b:skin> </b:skin></code> 标记内。</p><p>css代码支持 /* … */ 标记进行注释。</p><h3>样式变量</h3><p>在b:skin标签内可以为字体、颜色、长度等值定义变量,这些变量的值可通过Blogger后台模板设置界面进行修改。<h4>定义变量:</h4><pre><code>//example:
<Variable name="loraNormal20" description="Item body font" type="font" default="400 20px Lora, serif" value="400 20px Lora, serif"/>
<Variable name="attribution.icon.color" description="Attribution icon color" type="color" default="rgba(0,0,0,0.54)" hideEditor="true" value="rgba(0, 0, 0, 0.54)"/>
<Variable name="sidebar.width" description="Sidebar width" type="length" min="100px" max="1000px" default="320px" value="280px"/></code></pre>
<p>变量属性说明:<ul><li>name - 每个变量名称必须唯一,可包含字母或数字。。
<li>description - 变量说明,用于在“字体和颜色”设置界面显示。
<li>type – “font”, “color”, “length” 三种类型。颜色变量会在模板设计器中提供可供选择的调色板,字体变量会在模板设计器中提供字体选择。<li>default - 默认值。对于颜色,此变量应是十六进制颜色代码,如 #FF0066。对于字体,此变量会是 font-style font-weight font-size font-family 格式的列表。</li><li>value - 当通过模板设置界面修改后,值存在value里面。</li></ul><h4>使用变量:</h4><p>定义好变量后,直接在 <strong>b:skin</strong> css 代码中通过 $variable_name 进行调用,例如:</p>
<pre><code>.sidebar-container {
background-color: $(sidebar.backgroundColor);
position:relative;
max-width: $(sidebar.width);
}</code></pre>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-77091611747386661402013-05-05T11:35:00.004+08:002017-10-06T12:12:59.504+08:00Ubuntu iptables端口转发<h2>端口转发设置</h2>
<h3>本地端口转发</h3><pre><p>//localhost:8080 转发到 localhost:80</p><code class="Bash">sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080</code></pre>
<h3>跨机端口转发</h3>
<pre><p>//启用ip转发功能</p><code class="Bash">sudo vi /etc/sysctl.conf
net.ipv4.ip_forward=1</code>
<code class="Bash">sudo sysctl –p</code></pre>
<pre><p>//172.16.1.8:80 转发到 localhost:1234</p><code class="Bash">sudo iptables -t nat -A PREROUTING -p tcp --dport 1234 -j DNAT --to-destination 172.16.1.8:80
sudo iptables -t nat -A POSTROUTING -d 172.16.1.8 -p tcp --dport 80 -j MASQUERADE</code>
<code class="Bash">sudo iptables -I FORWARD -d 172.16.1.8 -p tcp –dport 80 -j ACCEPT
sudo iptables -I FORWARD -s 172.16.1.8 -p tcp –sport 80 -j ACCEPT</code></pre><h2>iptables基础知识</h2><h3>Filter表:过滤数据包</h3><ol><li>INPUT 链:过滤所有目标地址是本机的数据包(对进入本机数据包的过滤)</li><li>OUTPUT 链:过滤所有本机产生的数据包(对源地址得数据包的过滤)</li><li>FORWARD 链:过滤所有路过本机的数据包(源地址和目标地址都不是本机的数据包。</li></ol><h3>NAT表:网络地址转换</h3><ol><li>PREROUTING 链:数据包到达防火墙时改变包的目的地址。</li><li>OUTPUT 链:改变本地产生数据包的目标地址。</li><li>POSTROUTING 链:在数据包离开防火墙时改变数据包的源地址。</li></ol><h3>iptables常用命令</h3><blockquote><p>//查看iptables链表规则(不指定 –t 默认为Filter表)<br>sudo iptables -L –n</p><p>//清除Filter表某条规则链中所有规则<br>sudo iptables –F <font color="#0080c0">INPUT</font></p><p>//清除Filter表所有规则链的规则<br>sudo iptables –F</p><p>//清除Filter表用户自定链中的规则<br>sudo iptables –X</p><p>//在某个规则链末尾新增一条规则<br>sudo iptables -A <font color="#0080c0">INPUT …</font></p><p>//从某个规则链中删除一条规则<br>sudo iptables -D <font color="#0080c0">INPUT …</font></p><p>//从某个表链插入一条规则,原本该位置上的规则往后顺移<br>sudo iptables -I <font color="#0080c0">INPUT </font>1 --dport 80 -j ACCEPT<p>//查看NAT表规则信息<br>sudo iptables -t nat –L</p><p>//清除NAT表所有规则链的规则<br>sudo iptables -F -t nat<br>sudo iptables -X -t nat</p><p>//在NAT表某个规则链末尾新增一条规则<br>sudo iptables -t nat -A <font color="#0080c0">PREROUTING …</font></p><p>//从NAT表某个规则链中删除一条规则<br>sudo iptables -t nat -D <font color="#0080c0">PREROUTING …</font></p></blockquote><h3>常用匹配参数</h3><ul><li>参数 -p, –protocol<br>指定通讯协议类型,如tcp、udp、icmp 等,使用 all 关键词指定所有类型</li><li>参数 -s, --src, --source<br>来源 IP地址,可以是单个IP或网络(-s 192.168.1.0/24)</li><li>参数 -d, --dst, –destination<br>目的 IP地址,同上</li><li>参数 --sport, --source-port<br>数据包的来源端口号,可以是单个端口或端口范围(--sport 8010:8080)</li><li>参数 --dport, --destination-port<br>数据包的目的端口号,同上</li><li>参数 -i, --in-interface<br>指定数据包从哪个网口进入,支持通配字符 + (-i eth+ 表示所有的 ethernet 网卡)</li><li>参数 -o, --out-interface<br>指定数据包从哪个网口发出,同上</li></ul><h3>常用处理参数</h3><p>参数 –j :用来指定要进行的处理动作</p><ul><li>ACCEPT 将封包放行,进行完此处理动作后,将不再比对其它规则</li><li>REJECT 拦阻该封包,并传送封包通知对方</li><li>DROP 丢弃封包不予处理,进行完此处理动作后,将不再比对其它规则</li><li>REDIRECT 将封包重新导向到另一个端口(PNAT),进行完此处理动作后,将 会继续比对其它规则</li><li>MASQUERADE 改写封包来源 IP 为防火墙 NIC IP,可以指定 port 对应的范围,进行完此处理动作后,直接跳往下一个规则炼(mangle:postrouting)</li><li>SNAT 改写封包来源 IP 为某特定 IP 或 IP 范围,可以指定 port 对应的范围,进行完此处理动作后,将直接跳往下一个规则炼(mangle:postrouting)</li><li>DNAT 改写封包目的地 IP 为某特定 IP 或 IP 范围,可以指定 port 对应的范围,进行完此处理动作后,将会直接跳往下一个规则炼(filter:input 或 filter:forward)</li><li>MIRROR 镜射封包,也就是将来源 IP 与目的地 IP 对调后,将封包送回,进行完此处理动作后,将会中断过滤程序</li><li>QUEUE 中断过滤程序,将封包放入队列,交给其它程序处理</li></ul>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-37326466396896100602013-05-04T20:35:00.000+08:002017-10-02T11:39:18.503+08:00ubuntu tips<h2>启动相关 </h2><h3>// boot repair</h3><blockquote><p>sudo add-apt-repository ppa:yannubuntu/boot-repair && sudo apt-get update<br>sudo apt-get install -y boot-repair<br>sudo boot-repair<br></p></blockquote><h3>// edit grub boot menu</h3><blockquote><p>$ sudo add-apt-repository ppa:danielrichter2007/grub-customizer<br> $ sudo apt-get update<br> $ sudo apt-get install grub-customizer</p></blockquote><h3>// Change Plymouth Graphical boot animation Themes in Ubuntu:</h3><blockquote><p>sudo update-alternatives --config default.plymouth //select a theme number<br>sudo update-initramfs -u</p></blockquote><h2>包管理</h2><h3>// manage ppa repository</h3><blockquote><p>Install: <br>sudo apt-add-repository ppa:user/repository<br>Uninstall: <br>sudo apt-add-repository -r ppa:user/repository</p></blockquote><h3>// unpack deb </h3><blockquote><p>dpkg-deb -R ibm-notes-9.0.i586.deb ./tmp<br>vi ./tmp/DEBIAN/control<br>dpkg-deb --build ./notes_rebuild</p></blockquote><h2>GUI相关</h2><h3><font style="font-weight: normal;">// gnome desktop</font></h3><blockquote><p>sudo add-apt-repository ppa:gnome3-team/gnome3</p><p>sudo apt-get install ubuntu-gnome-desktop ubuntu-gnome-default-settings<br>sudo apt-get install gnome-documents gnome-boxes<br>
sudo apt-get install full-gnome3-experience // option for gnome3<br>
sudo apt-get remove ubuntu-settings // this will remove 'ubuntu-desktop'<br>
sudo dpkg-reconfigure gdm<br>
sudo service gdm/lightdm status<br></p></blockquote><h3>
// install MS TrueType core fonts</h3><blockquote><p>
sudo apt-get install ttf-mscorefonts-installer</p></blockquote><h3><font style="font-weight: normal;">//Firewall gui in Ubuntu </font></h3><blockquote><p>sudo apt-get install gufw</p></blockquote>
<h2>文件系统<br></h2><h3>//fs mount in fstab</h3><blockquote><p># <file system> <mount point> <type> <options> <dump> <pass><br> /dev/mapper/DevClient--vg-root / ext4 errors=remount-ro 0 1<br>/dev/sda3 /media/other ext3 rw, default 0 0</p><p># mount ntfs<br>/dev/sda5 /media/USER ntfs rw,nosuid,nodev,default_permissions 0 0<br> /dev/sdb6 /media/WORKS ntfs-3g silent,umask=0 0 0 </p></blockquote><p><br></p><h3>//ext4 filesystem file restore</h3><blockquote><p>文件恢复:<br>extundelete /dev/sdaX --restore-file /*/*....*/file<br>目录恢复:<br>extundelete /dev/sdaX --restore-directory /*/*....*/dir<br>恢复近期被删的所有:<br>extundelete /dev/sdaX --restore-all</p></blockquote><h2>VMware虚拟化</h2><h3>// start VMWare Workstation in Ubuntu 13.04</h3><blockquote><p>$sudo apt-get install build-essential linux-headers-$(uname -r)<br>$sudo ln -s /usr/src/linux-headers-$(uname -r)/include/generated/uapi/linux/version.h /usr/src/linux-headers-$(uname -r)/include/linux/version.h</p></blockquote><h3>
// change vmware S/N under cli</h3><blockquote><p>
/usr/lib/vmware/bin/vmware-vmx --new-sn NY0K2-2ULDJ-LZUP1-MV154-8363F</p></blockquote><h2>设置优化</h2><h3>// edit dns server</h3><blockquote><p>sudo gedit /etc/resolvconf/resolv.conf.d/head<br>sudo gedit /etc/resolvconf/resolv.conf.d/tail<br>sudo resolvconf –u</p></blockquote><h3><font style="font-weight: normal;">// add rar support</font></h3><blockquote><p>sudo apt-get install p7zip-rar<br></p></blockquote><h3><font style="font-weight: normal;">// unzip</font></h3><blockquote><p> unzip -O CP936 xxx.zip ( -O GBK / GB18030 / gb2312)<br></p></blockquote><h3><font style="font-weight: normal;">// powersave for laptop</font></h3><blockquote><p>sudo apt-get install laptop-mode-tools<br></p></blockquote><h3><font style="font-weight: normal;">// Enable hibernate</font></h3><blockquote><p>sudo apt-get install hibernate</p><p>or<br>sudo gedit /var/lib/polkit-1/localauthority/50-local.d/com.ubuntu.enable-hibernate.pkla<br> [Re-enable hibernate by default in upower]<br> Identity=unix-user:*<br> Action=org.freedesktop.upower.hibernate<br>ResultActive=yes</p><p>[Re-enable hibernate by default in logind]<br> Identity=unix-user:*<br> Action=org.freedesktop.login1.hibernate<br>ResultActive=yes</p></blockquote><h3><font style="font-weight: normal;">// remove old kernel ubuntu</font></h3><blockquote><p>uname -a<br>dpkg --get-selections|grep linux<br>sudo apt-get remove linux-image-2.6.15-23-386</p><p>rpm -q kernel<br> rpm -e kernel-xxx</p></blockquote>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-31779145912800623792013-04-29T00:00:00.000+08:002017-08-26T18:18:02.266+08:00Install IBM Notes 9.0 on Ubuntu 13.04<p style="text-indent: 2em">用了半年的ubuntu 12.10,感觉电源管理有些糟糕,电池基本上都管不了3个小时。所以在第一时间更新了13.04,虽然不是LTS版本从用上的几天感觉还不错,升级相对平滑也没有太多软件兼容性问题。下面将在ubuntu 13.04上安装Notes 9.0的过程分享如下,主要是去掉.deb中的部分包依赖即可正常安装。</p> <p># ***************************************************************** </p> <p>1) Download the debian version of IBM Notes 9 at:</p> <blockquote> <p><a href="http://www14.software.ibm.com/webapp/download/brand.jsp?b=Lotus">http://www14.software.ibm.com/webapp/download/brand.jsp?b=Lotus</a></p></blockquote> <p>2) Extract files from the archived tar:</p> <blockquote> <p>$tar -xvf NOTES_9.0_LINUX_DI_EN_Evaluation.tar<br><font color="#4f81bd">ibm-activities-9.0.i586.deb<br>ibm-cae-9.0.i586.deb<br>ibm-feedreader-9.0.i586.deb<br>ibm-notes-9.0.i586.deb<br>ibm-opensocial-9.0.i586.deb<br>ibm-sametime-9.0.i586.deb<br>license.tar<br>smartupgrade.sh</font></p></blockquote> <p>3) Removing dependencies from notes deb file:</p> <blockquote> <p>a. Extract the deb file into "notes_rebuild" folder<br>$dpkg-deb -R ibm-notes-9.0.i586.deb ../notes_rebuild/<br>$ls notes_rebuild/*<br><font color="#4f81bd">/DEBIAN/<br>/opt<br>/usr</font></p> <p>b. Edit the control file under "DEBIAN", find the line that starts "Depends:", delete "libcupsys2" and all of the "libgnome-desktop..." </p> <p>c. Build a new .deb file base on the "notes_rebuild" folder, it'll take a few minutes:<br>$dpkg-deb --build ./notes_rebuild</p></blockquote> <p>4) Replace the ibm-notes-9.0.i586.deb by new .deb file.</p> <p>5) Okay, let's begin to do the installation of Notes 9:</p> <blockquote> <p>$sudo apt-get update<br>$sudo apt-get install ia32-libs libgnomeprint2.2-0:i386 libgnomeprintui2.2-0:i386 libgnomevfs2-bin:i386 libgnome2-0:i386 libgnomeui-0:i386 libjpeg62:i386 grep:i386 libgconf2-4:i386 libpam0g:i386 libxkbfile1:i386 ttf-xfree86-nonfree<br>$sudo ./smartupgrade.sh</p></blockquote> <p>6) when finish all steps above, you'll see the new look of Notes 9, enjoy ~</p><br /><p style="text-indent: 2em"><img style="background-image: none; border-right-width: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px; padding-top: 0px" title="IBM_notes_9" border="0" alt="IBM_notes_9" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFVDLCxURbfxQ1uMhv47sx6E-B4r8s4riUGOA8i2FTN2iJ-mmLhTWB5_RkPQympcuL7gN_hGnTOKK4NgOLAeC0puBF6xsLyptcejwSytdwKEvbN8qeO-u2cb6oChaoeiee9bjeDO3-F7A/?imgmax=800" width="543" height="400"></p> iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-83640747987740279152010-03-15T11:49:00.000+08:002010-09-05T01:29:03.040+08:00This blog has moved<p>This's a private&off-line blog .</p>
<br />
<br />
This blog is now located at __FTP_MIGRATION_NEW_URL__.<br />
You will be automatically redirected in 30 seconds, or you may click <a href="http://myners.net/__FTP_MIGRATION_NEW_URL__">here</a>.<br />
<br />
For feed subscribers, please update your feed subscriptions to<br />
__FTP_MIGRATION_FEED_URL__.iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-84232909952011665902010-02-19T21:22:00.000+08:002017-08-31T22:13:39.804+08:00LVM在线扩容及相关知识汇总<p>关于LVM的逻辑结构:</p><p>【物理磁盘分区】-创建->【PV(Physical volume)】-组成->【VG(Volume group)】-划分->【LV(Logical volume)】-挂载->【文件系统挂载点】</br><br />
<img alt="logical volume manager" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGXHar_NilVHBvecKlsS-OZ3lpLjzxcb_e365q8HAqKHLfON0LRv-WQ3RVTCZ2GW7eCEZTy4kdw-Q6Pf5fTijQLZY2yPNwZEHYGAJSwzQs4jJtRWwzwAr-aPmi3vuS6KqJ4Lv3gRRZ5Bo/s1600/500px-LVM1.svg.png" title="logical volume manager" /></p><p>新增物理盘对LV卷在线扩容:</p><ul><li> 对新增硬盘分区</li>
</ul><blockquote><p>df -ah<br />
fdisk -l<br />
fdisk /dev/sda<br />
n -->创建分区<br />
t -->修改分区类型为8e<br />
w -->保存结果</p></blockquote><ul><li>将分区容量加入VG</li>
</ul><blockquote><p>pvcreate /dev/sda3 -->创建PV<br />
lvs -->查看确认待扩容的VG名<br />
vgextend VolGroup00 /dev/sda3 -->将sda3扩入VG00</p></blockquote><ul><li>对LV进行扩容</li>
</ul><blockquote><p>lvextend -L +2G /dev/mapper/VolGroup00-LogVol00 -->扩展LV00,增加2G<br />
e2fsck –f /dev/VolGroup00/LogVol00 -->强行检查data逻辑卷<br />
resize2fs /dev/VolGroup00/LogVol00 -->重定义文件系统大小<br />
**以上三步要按顺序进行,用以下一条命令可以达到上面三条命令相同的效果:<br />
e2fsadm –L +2G /dev/mapper/VolGroup00-LogVol00</p><p>df -ah -->最后查看调整后的分区挂载情况<br />
lvdisply -->查看LV更改后的状态<br />
lvscan -->扫描逻辑卷,看它们的活动情况</p></blockquote><p>如果新增LV卷:</p><blockquote><p>lvcreate –n LogVol01 –L 5G VolGroup00 -->从VG00中创建一个新的LV01,大小5G<br />
mkfs.ext3 /dev/VolGroup00/LogVol01 -->格式化LV01<br />
mke2fs /dev/VolGroup00/LogVol01 -->在LV上建立文件系统</p></blockquote><p>**有些操作需强制卸载挂载点<br />
umount /dev/mapper/VolGroup00-LogVol00 -l -->卸载下LV00,即/挂载点</p><p>====================================================================================</p><ul><li>调整根分区/</li>
</ul><p>如果要使根分区 / 使用LVM设备,需先创建好LVM设备,再用lvmcreate-initrd命令在boot分区创建支持lvm功能的虚拟磁盘镜像文件,最后修改grub.conf加载此镜像文件[修改fstab文件将/指向LV]</p><p>如果不能通过lvmcreate-initrd生成img,则可以按照下面方法手工修改init文件:</p><p>If your initrd script is called /boot/initrd-xxxxxxxxx.img, then you can unpack it with:</p><p>mkdir /tmp/initrd<br />
cd /tmp/initrd<br />
gzip -cd /boot/initrd-xxxxxxxxx.img | cpio -imd --quiet</p><p>编辑 /tmp/initrd/init ,修改对应的LV信息,然后再重新打包:</p><p>cd /tmp/initrd<br />
find . | cpio -co | gzip -9 > /boot/initrd-xxxxxxxxx.img </p><ul><li>调整Swap</li>
</ul><p>vgdisplay -->查看VG里面是否有剩余空间</p><p>swapoff -a -->关闭正在用的swap分区</p><p>lvextend -L +512M /dev/mapper/VolGroup00-LogVol01 -->扩展用于交换分区的LV</p><p>mkswap /dev/ubuntu/swap_1 -->建立swap的文件系统</p><p>swapon -a -->启用新的swap系统</p><p>====================================================================================<br />
LVM常用操作参考:</p><ul><li> 查看:<br />
pvdisplay vgdisplay lvdisplay pvs vgs lvs</li>
</ul><ul><li> 建立:<br />
pvcreate /dev/{sda,sdb} <br />
vgcreate vg01 /dev/sda /dev/sdb<br />
lvcreate --size 40G -n LVname VGname</li>
</ul><ul><li> 增加LV:<br />
lvresize 60G /dev/vg/lv<br />
resize2fs 60G /dev/vg/lv</li>
</ul><ul><li> 减小LV:<br />
umount /dev/vg/lv<br />
resize2fs 20G /dev/vg/lv<br />
lvresize 20G /dev/vg/lv</li>
</ul><ul><li> 删除磁盘:<br />
pvmove /dev/sda1 /dev/sda2<br />
vgreduce uplooking /dev/sda1<br />
pvremove /dev/sda1</li>
</ul><ul><li> 扩充磁盘:<br />
pvcreate /dev/sdb<br />
vgextend VGname /dev/sdb</li>
</ul><p>====================================================================================</p><p>LVM命令集合参考:<br />
</p><ul><li>PV:<br />
pvchange pvcreate pvmove pvresize pvscan <br />
pvck pvdisplay pvremove pvs </li>
<li> VG:<br />
vgcfgbackup vgconvert vgextend vgreduce vgscan<br />
vgcfgrestore vgcreate vgimport vgremove vgsplit<br />
vgchange vgdisplay vgmerge vgrename <br />
vgck vgexport vgmknodes vgs</li>
<li> LV:<br />
lvconvert lvdisplay lvresize lvchange lvextend lvreduce <br />
lvremove lvscan lvcreate lvrename lvs </li>
<li>LVM:<br />
lvm lvmchange lvmdump lvmdiskscan lvmsar lvmsadc </li>
</ul><p>====================================================================================</p><p>LVM高级应用:</p><p>减小PV:# 风险较大,不建议应用于生产环境</p><blockquote><p>pvresize -t --setphysicalvolumesize 45G -v /dev/sda6 可以再继续<br />
pvresize --setphysicalvolumesize 45G -v /dev/sda6<br />
fdisk /dev/sda6<br />
d掉6 再n +45G t 8e w<br />
partprobe<br />
pvscan</p></blockquote><p>如果在删除某LV后,没有在fstab里将该挂载点去掉,可能会导致重启后无法进入系统,并提示输入root密码进入维护模式或者按Ctrl+D重启。<br />
修复方法如下:</p><p>由于该模式下是以read-only挂载分区的,所以需要重新挂载/使其可读写:<br />
(Repair Filesystem)#mount -o remount,rw /<br />
(Repair Filesystem)#vi /etc/fstab 编辑fstab文件<br />
#/dev/mapper/asr_dbp1 /db ext3 defaults 1 2 去掉无效的LV或者磁盘挂载语句<br />
<<保存退出>><br />
(Repair Filesystem)#reboot 重启系统</p><p>对于其它问题引起的进入维护模式也可以尝试逐个检查磁盘分区<br />
(Repair Filesystem)#fsck /dev/hdaX</p>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-19097559199003825622009-10-06T07:37:00.000+08:002021-03-20T19:40:50.116+08:00关于Google Apps域名的转出<p style="text-indent: 2em">起初为了比较方便地开通Google Apps 服务就直接通过GG申请了域名,而实际上这个域名也是Google 通过Godaddy 管理的,并且设置了Private Registration 保护域名的注册信息。但是通过Google Apps 获得的Godaddy 账户不能直接在Godaddy 登陆(他们称之为“GoDaddy Google Sign-In Name”,只能通过Google Apps的专有入口登陆进行管理),因此就不能享受到Godaddy的一些免费服务,如:共享主机、邮箱、相册、统计 等等。</p> <p style="text-indent: 2em">Google Apps 的高级DNS设置页面提供了域名管理系统的入口链接: <br /><img style="display: inline" title="Advanced DNS setting" border="0" alt="Advanced DNS setting" src="http://days.myners.net/wp-content/uploads/163/16382/2009/10/apps-dns.png" width="534" height="229" /></p> <p style="text-indent: 2em">下面两幅图分别是Google Apps 与 GoDaddy 的管理界面,可以看出一些区别: <br /><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="at_gg_apps" border="0" alt="at_gg_apps" src="http://days.myners.net/wp-content/uploads/163/16382/2009/10/at-gg-apps1.png" width="248" height="144" /><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="at_godaddy" border="0" alt="at_godaddy" src="http://days.myners.net/wp-content/uploads/163/16382/2009/10/at-godaddy1.png" width="284" height="144" /></p> <p style="text-indent: 2em">于是打算将域名从Google Apps托管转出,直接放在Godaddy上管理。虽然本质上讲,这种transfer就是将域名从Godaddy转到Godaddy,但由于是属于不同子系统,所以只好按一般的transfer流程实现。不过整个流程非常方便(不像国内有些域名服务商转个域名还要给它提交身份证明,最后由它解锁,而且还要以发函的形式索取转移授权密码),需要确认的环节都是由用户自行审批(这其实也反映出用户对自己所购域名拥有绝对所有权),顺利的话一个小时就能搞定。  </p> <p style="text-indent: 2em">下面简单说一下transfer的流程(最好将当前的NS设置导出进行备份或者截图):</p> <p>1、通过<a title="Google Apps" href="http://www.google.com/a/" rel="nofollow" target="_blank">Google Apps</a> 入口登陆域名管理系统(相当于原域名注册商),将要转移的域名状态改为<strong><font color="#0080ff">unlock</font></strong>,并将 <font color="#0080ff"><strong>Authorization code</strong></font> 发一份到自己邮箱;</p> <p>2、登陆<a title="Godaddy" href="http://www.godaddy.com" rel="nofollow" target="_blank">Godaddy</a> 的帐户,点击Domain Related 下的 Domains Transfer,输入要转移的域名,如 <u>baidu.com</u>;</p> <p>3、如果前面做的域名解锁已经生效,这里会显示Available For Transfer,接下来开始Checkout(Godaddy的推销广告真是多),如果只是转移域名,最终的费用在USD$7左右(不知道是否是当前优惠,我的从09年直接续到了2011年),至于支付方式,个人感觉用双币信用卡是最方便的(域名转移没有可用优惠码的);</p> <p>4、在完成Godaddy 的购买后,Godaddy 会向原域名所有者的邮箱发一封邮件通知域名转移的<font color="#0080ff"><strong>Transaction ID </strong></font>和 <strong><font color="#0080ff">Security Code</font></strong>,最终会转发到你的邮箱的;</p> <p>5、与其它域名转移不同的,将Google Apps域名transfer out之前还需要取消前面提到的Private Registration 服务,否则会收到类似这样的原因 “Express written objection to the transfer from the Transfer Contact”而被拒绝;取消的方法如下:</p> <blockquote> <p>登陆 <a href="http://domainsbyproxy.com" rel="nofollow" target="_blank">domainsbyproxy.com</a>(由于private registration是由Google代理申请的,所以这里的customer number和密码都需要通过'Forgot your password'找回),<strong>My Domains</strong>: <br /><img border="0" alt="" src="http://forums.daxp.com/teaching/img_domain/private_domainsbyproxy004.png" /> <br />选择上要取消Private Registration的域名,按<strong>Continue</strong>继续 <br /><img border="0" alt="" src="http://forums.daxp.com/teaching/img_domain/private_domainsbyproxy007.png" /> <br />最后选择”Yes - Cancel Domains by Proxy® for the domain name(s) listed above.” <br /><img border="0" alt="" src="http://forums.daxp.com/teaching/img_domain/private_domainsbyproxy008.png" /> <br />完成后会显示如下信息,表示已经成功取消了该域名的Private Registration: <br />Your request has been processed but may take a few minutes to appear. Please be patient.</p> <p>原文参考:<a title="http://forums.daxp.com/showthread.php?p=6135" href="http://forums.daxp.com/showthread.php?p=6135" rel="nofollow" target="_blank">http://forums.daxp.com/showthread.php?p=6135</a></p> </blockquote> <p>6、再次进入<a title="Godaddy" href="http://www.godaddy.com" rel="nofollow" target="_blank">Godaddy</a> 的帐户,在Domain Manager的Pending Transfers选择要转移的域名信息,在Authorization点Begin Transfer Authorization,输入邮箱收到的Transaction ID和Security Code,最后要求输入域名转移密码(EPP Key),即通过邮箱接收的授权码。</p> <p><img style="border-right-width: 0px; margin: 0px 0px 0px 15px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" align="right" src="http://days.myners.net/wp-content/uploads/163/16382/2009/10/image1.png" width="204" height="112" />7、至此在<a title="Godaddy" href="http://www.godaddy.com" rel="nofollow" target="_blank">Godaddy</a> 上的转移步骤基本上完成了,同时也会收到一堆有关transfer进度的通知邮件。</p> <p>但为了能更快地完成转移,接下来再回到Google Apps的域名管理系统,选择Domains菜单下的Pending Transfers,如右图所示;可以看到正在等待审核通过的域名,选中它点 Accept/Decline,选择Accept 。</p> <p>8、完成后会收到一封来自<a href="mailto:service@secureserver.net">service@secureserver.net</a> 的邮件告知: <br />This is to confirm that the following domain name(s) has been successfully transferred away from Google Apps to another registrar . <br />同时<a title="Godaddy" href="http://www.godaddy.com" rel="nofollow" target="_blank">Godaddy</a> 也会发一封完成通知: <br />Congratulations! The transfer request for the following domain name(s) has been completed .</p> <p>至此整个转移结束了,最快一个小时搞定。但不好的消息是我发现域名转移到Godaddy后,原来Google Apps的相关NS设置都被没了,幸好前面做过备份,将Godaddy 的默认设置删除后导入之前的设置就行了,目前还没感觉到 Google Apps 服务因此而受到什么影响。 </p> <p>配图说明可参考这里: <a title="http://www.koryi.net/net/649.html" href="http://www.koryi.net/net/649.html" rel="nofollow" target="_blank">http://www.koryi.net/net/649.html</a>;附上官方的操作说明,非常详细:</p> <blockquote> <p><u><strong>Cancel the Private Registration (取消私有注册)</strong></u></p> <p>The private registration for your domain is managed through Domains By Proxy, while the domain registration is managed by us. </p> <p>You will need to log into your account with Domains By Proxy in order to cancel the privacy service. We have received the following instructions from Domains By Proxy for canceling private registration service for a domain: </p> <p>• Go to <a href="http://domainsbyproxy.com" rel="nofollow" target="_blank">domainsbyproxy.com</a> and click on 'My Account'. <br />• Log in to your Domains by Proxy account. </p> <p>-If you do not remember your Domains by Proxy login information, click on the 'Forgot your password?' link to retrieve your customer number and reset your password </p> <p>• Once logged in, click on 'My Domains'. <br />• Check the box next to the domain(s) you wish to cancel private registration for. <br />• Click on 'Continue'. <br />• Confirm that you wish to cancel the registration by selecting YES. <br />• Click the 'Submit' button. </p> <p>You will then receive an email regarding the cancellation. The domain will no longer appear in your Domains by Proxy account, and the cancellation will not affect any other services on the domain such as DNS or email. Please remember that cancellation of the private registration is not refundable. </p> <p>If you have any other questions regarding the cancellation of your private registration, please contact the Domains By Proxy support staff using the following URL: <a href="http://domainsbyproxy.com/Support.aspx"> <br />http://domainsbyproxy.com/Support.aspx</a></p> <p><strong><u>Unlock the Domain (域名解锁) </u></strong></p> <p>To Unlock your domain name, you will need to follow the below instructions. </p> <p>1. Go to <a href="http://www.google.com/a">http://www.google.com/a</a> and click on 'Returning user, sign inhere' on the top right. </p> <p>2. Enter your domain and select "Manage this Domain" from the drop down and click 'Go.' </p> <p>3. Now you will need to login to your Google account utilizing a username and password you previously created specifically for Google Apps for your domain. If you do not know this information, click 'Forgot your administrator username or password?' </p> <p>4. Once logged in, select 'Domain Names' from the blue box that lists your domain. </p> <p>5. Click on "Advanced DNS Settings". </p> <p>6. You will now see listed your GoDaddy Google Sign-In Name, Password, and PIN. </p> <p>7. Select "Sign in to DNS Console" in order to manage your DNS settings. </p> <p>8. Now enter your Google Sign-In Name and Password listed on the previous screen. </p> <p>9. You should now be in the Domain Control Center </p> <p>10. Click on the domain name you need to unlock. </p> <p>11. Click on the "Locking" button in the menu bar. </p> <p>12. Select the radial button next to "Unlock" and click the "OK" button. </p> <p>13. Click the "OK" button again on the "Processing" screen. </p> <p><u><strong>Request the Authorization Code (获取授权码) </strong></u></p> <p>To request the authorization code for your domain name, you will need to follow the below instructions. </p> <p>1. Go to <a href="http://www.google.com/a">http://www.google.com/a</a> and click on 'Returning user, sign inhere' on the top right. </p> <p>2. Enter your domain and select "Manage this Domain" from the drop down and click 'Go.' </p> <p>3. Now you will need to login to your Google account utilizing a username and password you previously created specifically for Google Apps for your domain. If you do not know this information, click 'Forgot your administrator username or password?' </p> <p>4. Once logged in, select 'Domain Names' from the blue box that lists your domain. </p> <p>5. Click on "Advanced DNS Settings". </p> <p>6. You will now see listed your GoDaddy Google Sign-In Name, Password, and PIN. </p> <p>7. Select "Sign in to DNS Console" in order to manage your DNS settings. </p> <p>8. Now enter your Google Sign-In Name and Password listed on the previous screen. </p> <p>9. You should now be in the Domain Control Center </p> <p>10. Click on the domain name you need to request the authorization code for. </p> <p>11. Find and click on the link "Send by Email" next to "Authorization Code" to the left of the domain details screen. </p> <p>12. The Authorization code will now be sent to the Administrative Email address. </p></blockquote>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-91272510418252296142009-07-19T05:18:00.000+08:002021-03-20T20:06:59.240+08:00Cisco顶级认证 = CCIE+MBA ?<p style="text-indent: 2em">时至今日,曾经被捧为至尊神话般的CCIE在现在已经得到了极大普及,在一般的中小型企业(如代理商、小型SI)经常能见到他们的身影,这主要得益于诸多优秀的培训机构啊<img style="margin: 0px 0px 0px 10px; display: inline" align="right" src="http://www.globalknowledge.de/images/4-Layer_Pyramid_m.gif" width="210" height="184" />。 </p> <p style="text-indent: 2em">物极必反,多起来也就意味着贬下去。Cisco在过去也采取了一些列的措施,用于维持其行业技术认证的价值,比如:规范缩减了Cisco认证考场、全面提升了认证考试的费用(CCNA的802直接价格翻倍;CCNP的901,812,825,845每门涨了20%;CCNP的642-892价格涨了60% ;CCIE笔试涨了11.11%)、并且调整了Cisco认证的层次(增加了比CCNA还低的CCENT)等等。 但这一切仍然不可避免一项专业化认证的同质化,于是Cisco最近又以『架构师』的名义推出了新的顶级认证。具体内容见以下文摘:</p> <blockquote> <p style="text-indent: 2em">多年以来,CCIE(Cisco Certified Internetwork Expert)认证一直是网络职业人士梦寐以求的最高级别认证,目前全球获得该认证殊荣的人也很少(<font color="#0080c0">过去时态</font>)。不过,现在这种情况出现了变化。思科在周一又发布了一种比CCIE级别更高的新认证 —— CCA(思科认证架构师)。思科试图通过CCA认证将CCIE的网络工程技能与MBA的商业才干结合起来。</p> <p style="text-indent: 2em">在思科的CCA认证中,CCA将训练报考者与首席商务运营主管们一起工作,以将商业需求转换成有效的IT系统。 思科发言人Fred Wieller表示:“我们的研究显示,如果一家公司购买了错误的技术和服务,那么他们可能会浪费1000亿美元。”产生这种浪费的原因是主管人员对商业需求有着深刻的认识,但是对IT架构缺乏认识,或者是对IT架构有着深刻的认识,但是对商业需求缺乏认识。CCA将训练CCIE如何沟通以上两个群体。</p> <p style="text-indent: 2em">思科新公布的CCA认证不仅花费昂贵,而且也极不容易通过。CCA认证需要花费15000美元。报考者不仅必须拥有一个CCIE或CCDE(思科认证设计专家)认证,并且需要有至少10年的相关行业的工作经验。报考者必须提出申请,然而再经过思科的严格筛选才能参加培训。一旦被允许参加培训课程,报考者必须为一系列商业需求提出一个架构解决方案。届时,考试委员会将会对培训者进行质询,培训者则需要对方案进行答辩,并进行修改。考试委员会由全球知名工程师和其它领域内的相关专家组成,成员不仅仅来自思科。</p> <p style="text-indent: 2em">对此,Wieller表示:“这一认证将鼓励企业将网络架构视为工作职能而不是工作的一部分。”一旦被允许参加培训课程,认证程序可在数周内完成。</p> <p style="text-indent: 2em">目前行业分析人士对于这种新认证十分赞成。市场研究机构Yankee Group的高级副总裁Zeus Kerravala表示:“CCIE目前已经是网络工程师的最高级别认证。然而新推出的CCA认证在级别上将超过CCIE认证,这意味着一名拥有CCIE认证的网络工程师将具有MBA的商业才干。许多公司都会经常购买一些技术,然后再盘算如何利用这些技术。这种现象十分普遍。以商业标准看待这些技术目前是一种十分抢手的技能。”</p> <p style="text-indent: 2em">Kerravala相信,CCA认证很快将与CCIE认证一样具有价值。由于许多系统整合公司如今必须出示其雇员中获得CCIE认证的比例,因此Kerravala相信系统整合公司将会率先派遣他们的雇员参加CCA认证。此外,Kerravala还认为大型跨国公司的网络工程师将也会热衷于这种新的认证。他认为未来这种认证甚至可能会代替某些商业研究生学位,今后一些合作复杂网络的公司将能会专门寻求这种人才。</p> <p style="text-indent: 2em">思科表示,未来CCA认证将会成为一种精英认证,每年通过这种认证的人数也将会非常少。思科项目经理Sanjay Mehta称,通过CCA认证的人数在头两年可能仅有两位数水平。思科在今年秋季将开始接受CCA认证的申请,在明年一月份首批参加培训人员将对他们的架构解决方案进行答辩。</p></blockquote>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-20928405118610052452009-05-16T02:31:00.000+08:002021-03-20T20:05:21.532+08:00高亮插件测试<p>练习搜索算法时的一些代码,试一下代码高亮的效果,可结果去如此糟糕 ... <br />怎么是这副模样? 跟模板不兼容 ?? <br /></p> <pre><code class="html"><br/>// Sorting.cpp : by iX.<br/>//<br/>#include <stdio.h><br/>#include <string><br/>#include <iomanip> //支持cout格式化输出<br/>#include <iostream><br/>static string ss;<br/>class rnda //定义一个随机数组类<br/>{<br/>public:<br/> int num; //数组元素个数<br/> double *p; //指针p指向数组空间首地址<br/>public:<br/> rnda(int);<br/> void print();<br/>};<br/>rnda::rnda(int n) //由指定值生成数组<br/>{<br/> int i,j;<br/> num=n;<br/> p=new double[num];<br/> double r=1.0;<br/> for(i=0;i<num ;i++) //产生num个0-1之间的随机数<br/> {<br/> r=2053.0*r+13849.0;<br/> j=r/65536.0;<br/> r=r-j*65536.0;<br/> p[i]=r/65536.0;<br/> }<br/> for(i=0;i<num;i++) //产生num个0-1000之间的随机数<br/> p[i]=0+1000.0*p[i];<br/> return;<br/>}<br/>void rnda::print() //打印出当前数组<br/>{<br/> int i;<br/> for(i=0;i<num;i++)<br/> cout<<setw(10)<<p[i];<br/> cout<<endl;<br/>}<br/>//*****************定义各类排序算法函数**********************<br/>// A、双向扫描冒泡算法<br/>void bubble(double a[],int n) <br/>{<br/> int up,dn,i,j; //定义i,j两个临时变量<br/> double t; //t用于数值交换的临时变量<br/> up=0; dn=n-1; //up作为数组扫描的上界标识,dn下界标识<br/> while(up<dn) <br/> {j=dn-1;dn=0;<br/> for(i=up;i<=j;i++)<br/> //从前(up)往后(dn)扫描,将较大值往后排<br/> if(a[i]>a[i+1])<br/> //最后一次交换后的i值作为下界(dn)标识<br/> {t=a[i];a[i]=a[i+1];a[i+1]=t;dn=i;}<br/> j=up+1;up=0;<br/> for(i=dn;i>=j;i—)<br/> //从后(dn)往前(up)扫描,将较大值往后排<br/> if(a[i-1]>a[i])<br/> //最后一次交换后的i值作为上界(up)标识<br/> {t=a[i-1];a[i-1]=a[i];a[i]=t;up=i;} <br/> }<br/> ss="冒泡算法";<br/> return;<br/>}<br/>// B、快速排序算法(递归) <br/>static int split(double p[],int n) //用于快速排序法的表的分割函数<br/>{<br/> int i,j,k,l;<br/> double t;<br/> i=0;j=n-1; <br/> k=(i+j)/2; <br/> if ((p[i]>=p[k])&&(p[k]>=p[j])) l=k; //在表的前、中、后三个元素中选取中项作为分割点<br/> else if((p[i]>=p[j])&&(p[j]>=p[k])) l=j;<br/> else l=i;<br/> t=p[l]; //...<br/> p[l]=p[i]; //将此时的第一个元素p[i]的值移到中项p[l]<br/> while(i!=j)<br/> { <br/> while((i<j )&&(p[j]>=t)) //从后向前检查,发现不大于t的p[j]就把它丢到前面的p[i]<br/> j--;<br/> if(i</j><j )<br/> {<br/> p[i]=p[j];i++; //...<br/> while((i<j)&&(p[i]>=t)) //从前向后检查,发现不小于t的p[i]就把它丢给后面的p[j]<br/> i++;<br/> if(i</j><j )<br/> {p[j]=p[i];j--;} //...<br/> }<br/> }<br/> p[i]=t; //在i,j指向同一个位置的时候,再把分割值赋给p[i]<br/> return(i); //返回分割线位置<br/>}<br/>//排序函数<br/>void quick(double b[],int n)<br/>{<br/> int m,i;<br/> double *s;<br/> if(n>10)<br/> {<br/> i=split(b,n); //对表进行分割,同时得到分割线位置<br/> quick(b,i); //对分割后的前部分子表进行快速排序<br/> s=b+(i+1); m=n-(i+1); <br/> quick(s,m); //对后面子表进行快速排序<br/> ss="快速排序算法";<br/> }<br/> else<br/> bubble(b,n); //子表长度若小于10,则调用冒泡法排序 <br/> return;<br/>} <br/>// C、简单插入排序算法<br/>void insort(double c[],int n)<br/>{<br/> int i,k; <br/> double t;<br/> for(i=1;i<n ;i++)<br/> {<br/> t=c[i]; <br/> for (k=i-1;k>=0;k--) //将t的值依次往前插入<br/> if (c[k]>t) <br/> c[k+1]=c[k];<br/> else break; //跳出,避免无畏的比较<br/> c[k+1]=t;<br/> }<br/> ss="简单插入排序算法";<br/> return;<br/>}<br/>// D、希尔排序算法<br/>void shell_sort(double d[],int n)<br/>{<br/> int i,h,k;<br/> double t;<br/> h=n/2; //在步进h恒等于1时,等价于插入排序法<br/> while (h>0)<br/> {<br/> for(i=h;i</n><n -1;i++)<br/> {<br/> t=d[i];<br/> k=i-h;<br/> while((k>=0)&&(d[k]>t))<br/> {d[k+h]=d[k];k-=h;}<br/> d[k+h]=t;<br/> }<br/> h=h/2; //减小步进,进行下一轮的排序<br/> }<br/> ss="希尔排序算法";<br/> return;<br/>}<br/>// E、堆排序算法(适用于大规模的线性表排序)<br/>static int sift(double p[],int i,int n) //用于堆排序法的表的筛选函数<br/>{<br/> int j;<br/> double t;<br/> t=p[i];j=2*(i+1)-1;<br/> while (j< =n)<br/> {<br/> if((j<n)&&(p[j]<p[j+1])) j++;<br/> if (t<p[j])<br/> {p[i]=p[j];i=j;j=2*(i+1)-1;}<br/> else j=n+1;<br/> }<br/> p[i]=t;<br/> return(0);<br/>}<br/>void heap(double e[],int n)<br/>{<br/> int i,mm;<br/> double t;<br/> mm=n/2;<br/> for(i=mm-1;i>=0;i--)<br/> sift(e,i,n-1);<br/> for(i=n-1;i>=1;i--)<br/> {<br/> t=e[0];e[0]=e[i];e[i]=t;<br/> sift(e,0,i-1);<br/> } <br/> ss="堆排序算法";<br/> return;<br/>}<br/>// 定义一个用于调用特定排序函数对数组进行排序的函数<br/>void sorting(int s,double p[],int n) <br/>{<br/> void (*f)(double[],int); //(*f)(double,int) ——指向函数的指针<br/> switch (s)<br/> {<br/> case 1: f=bubble; <br/> break;<br/> case 2: f=quick; <br/> break;<br/> case 3: f=insort; <br/> break;<br/> case 4: f=shell_sort; <br/> break;<br/> case 5: f=heap; <br/> break; <br/> }<br/> (*f)(p,n); // 调用对应函数排序<br/> return;<br/>}<br/>//***************** Main函数 **********************<br/>int main()<br/>{<br/> int n,f=0; <br/> cout< <"定义随机数组数据成员个数(1~1000间的随机数组)"<<endl;<br/> cin>>n;<br/> rnda r(n); //产生一个有n个成员的随机数组 <br/> cout< <"随机数组:"<<endl;<br/> r.print();<br/> while(f<1||f>5)<br/> {<br/> cout< <"选择排序算法:"<<endl;<br/> cout<<"... 1、双向扫描冒泡算法"<<endl;<br/> cout<<"... 2、快速排序算法"<<endl;<br/> cout<<"... 3、简单插入排序算法"<<endl;<br/> cout<<"... 4、希尔排序算法"<<endl;<br/> cout<<"... 5、堆排序算法"<<endl<<endl;<br/> cout<<"选择:";<br/> cin>>f; <br/> } <br/> sorting(f,r.p,r.num);<br/> cout< <endl<<"按["<<ss<<"]排列后数组:"<<endl;<br/> r.print(); <br/> return 0;<br/>} <br/></code></pre>iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0tag:blogger.com,1999:blog-7630626813003569974.post-89392745924468797802009-03-20T02:13:00.000+08:002017-08-27T16:32:07.171+08:00Windows 7 Build7057及其激活<img alt="Windows 7 Ultimate" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxaKkx0r4qrQycKQbXAPLaxCU7aDk-ULJ4Xo7cM3rwpxwwwS71G5_5uE8yrK7qBprUseI8ftuPyudKoZzY_C3jxEt5tQPi3Ux146Ya5eEWKHQDExbbtLu9e8ykY-GYKbgX3wQIpI71ZCw/s640/win7_ultimate.jpg" title="Windows 7 Ultimate" width="640" /><br />
<blockquote>Windows 7 Build 7057 x86 下载: <br />
大小:2.51GB <br />
版本:7057.0.090305-2000 <br />
文件名:7057.0.090305-2000_x86fre_client_en-us_Retail_Ultimate-GB1CULFRER_EN_DVD <br />
下载地址:<a href="http://www.mininova.org/tor/2368441">http://www.mininova.org/tor/2368441</a></blockquote><p style="text-indent: 2em;">继第一个beta版放出后,Windows 7依然保持快速地更新势头,截止到目前最新版本已经到了Build 7061,不过这个版本目前还没能找到好的下载资源;而最近放出的Build 7057 可以方便下载到,并且能够在线激活使用,对于Windows 7 的抢先体验已经不错了。</p><p style="text-indent: 2em;">Windows 7新增了对虚拟磁盘的支持,可以直接在Vdisk上安装完系统。Windows 7的桌面,7057新加入了很多漂亮的主题、壁纸:<br />
<img alt="Windows 7" border="0" data-original-height="900" data-original-width="1440" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUFIillTE5MIQtsPdrF8nqlfAflB_jV4N1pWB1DGZbwxf8I0ECb6oK5bgam6r9DNCEkgatI16H9BIIKKqcG1sdTeE4J4HoaLg63EDwwouRgaV-D7CK4-1cN61eN8UxfGlAZZii_WaWAXs/s640/desktop.jpg" title="Windows 7" width="640" /><br />
<br />
下面是现在用的Windows 2008 Server Ent 的桌面对比(同Vista)<br />
<img src="https://lh4.ggpht.com/_ThH-Wy43k8U/ScJEKsRD0pI/AAAAAAAAJGs/MbEspSZitbY/s512/win2008desktop.jpg" /><br />
Windows 7的资源管理器相对Vista做了一些改进<br />
<img alt="Windows 7" src="https://lh3.ggpht.com/_ThH-Wy43k8U/ScJD6HtcH0I/AAAAAAAAJGA/ETyWwmDhLVY/s512/explorer.jpg" title="Windows 7" /><br />
新的 Media Center 体验,支持图片、音乐、视频播放 。<br />
<img border="0" data-original-height="460" data-original-width="769" height="382" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgL-cpV09PhfiOrXvacIssW3b7LL6yt4nAEpDGeBJD2q5Ucf6eyC6OY2YZq4WBWQCxhBdzAmpmpyY2MtXl6r5eEiZFnKLjQ0r1QGp0_Hw7fqlcOUnHiCjPZp6p-Lsmt6YAlKHrGS7qMbsY/s640/windows_media_center.png" width="640" /><br />
<br />
Build 7057 里附带的IE8的版本<br />
<img border="0" data-original-height="332" data-original-width="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZPBkzGFkT6GB9em1tTprTqC_WOZsBcSku7KErLZAgTUwWPv4TyPJqwYPLhtOGak9ZN2iKWEugP3zZTkn46lcmebpRo8gVG5jdM3RLNrvj50gF2qUBlwaRNnEQTwhNXxGXeeUS_kdasNI/s1600/Internet+Explore+8+RC1.jpg" /><br />
</p>--------------------------------------------------------------------------------------<br />
以下是 Windows 7 build 7057 可用的序列号,能通过在线激活 !<br />
<table border="1" cellpadding="2" cellspacing="0" style="width: 512px;"><tbody>
<tr> <td valign="top" width="255">32-Bit</td> <td valign="top" width="255">64-Bit</td> </tr>
<tr> <td valign="top" width="255">6JKV2-QPB8H-RQ893-FW7TM-PBJ73 <br />
TQ32R-WFBDM-GFHD2-QGVMH-3P9GC <br />
GG4MQ-MGK72-HVXFW-KHCRF-KW6KY <br />
4HJRK-X6Q28-HWRFY-WDYHJ-K8HDH <br />
QXV7B-K78W2-QGPR6-9FWH9-KGMM7</td> <td valign="top" width="255">7XRCQ-RPY28-YY9P8-R6HD8-84GH3 <br />
RFFTV-J6K7W-MHBQJ-XYMMJ-Q8DCH <br />
482XP-6J9WR-4JXT3-VBPP6-FQF4M <br />
JYDV8-H8VXG-74RPT-6BJPB-X42V4 <br />
D9RHV-JG8XC-C77H2-3YF6D-RYRJ9</td> </tr>
</tbody></table>成功激活系统后,通过 slmgr.vbs -dlv 命令可以看到激活后可以使用到2010年6月2号 。<br />
<img border="0" data-original-height="409" data-original-width="468" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2pc1jNth33g-YAHxHVfoU-Imhq9bdrneYUGU6ZLnRBl-mA9-a0tRFnoe9LfnPqeqy-ocKZuc607fxqbiEra7RdEWSY3aK-k2BCV3vgGCQLwFhPuRILUXCUpM6AEOFv3eyy1NCSeGcKTI/s1600/slmgr.vbs+-dlv.jpg" /><br />
<br />
slmgr.vbs –xpr 命令显示系统已彻底激活:<br />
<img border="0" data-original-height="169" data-original-width="282" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPiUFmUtfM53AlubHlMblL0gK8um3KbVgthHtk9rG5ovT4mwXsjb8QBK0xQoKY6Y1Q3ly4F19bhUQbYYDSEo7ov4pF0uoZ6_m7ibkU3r61gEzFzgnrTKp5H3uhOAm-7QSXikrZSma3K30/s1600/slmgr.vbs+-xpr.jpg" /><br />
iX.http://www.blogger.com/profile/00061260980836260911noreply@blogger.com0