Linux Programming I/O Redirection、Pipeline

輸入、輸出、重新導向

標準輸入(STDIN):代碼表示0,程式執行時所需要的輸入資料。
標準輸出(STDOUT):代碼表示1,程式”正常“執行時所產生的輸出資料。
標準錯誤輸出(STDERR):代碼表示2,程式”錯誤“執行時輸出的訊息。
重新導向:改變資料的輸出入方向。


Image source:Piping and Redirection,output,Reading from STDIN in script(Linux)

標準輸出 STDOUT

一般的程式執行結果會顯示在螢幕上,也就是說預設的標準輸出是螢幕。我們可以使用 > 來將標準輸出導向至檔案。
語法:

1
N > FILE
1
Example:將 ```ls -al``` 輸出內容導向至ls.txt

ls -al > ls.txt

1
2
3
4
若當前資料夾中``` ls.txt ```檔案不存在,則為自動生成一個並寫入。
若此檔案存在,則會清除原始內容並寫入```ls -al```輸出資料。
若想要保留原始資料並將輸出內容接續檔案原內容,則可以使用```>>```替換原本的```>```。
Example:

ls -al >> ls.txt

1
2
3
4

## 標準錯誤輸出 STDERR
程式發生錯誤時,錯誤訊息預設也是輸出在螢幕上。
Example:

ls no_file.txt
ls: cannot access no_file.txt: No such file or directory

1
若想要將錯誤訊息輸出至```ls.txt```內,如果下這行command:

ls no_file.txt > ls.txt
ls: cannot access no_file.txt: No such file or directory

1
錯誤訊息依然顯示在螢幕,而```ls.txt```也不會寫入錯誤訊息。原因是```>```預設代碼為```1```,此時我們要指定代碼為```2```,也就是指定將輸出錯誤訊息寫入```ls.txt```內。

ls no_file.txt 2> ls.txt
cat ls.txt
ls: cannot access no_file.txt: No such file or directory

1
若想將標準輸出與標準錯誤輸出的資料一起寫入同個檔案可以這樣寫:

ls no_file.txt > ls.txt 2>&1

1
2
3
```2>&1```代表將**標準錯誤輸出**的資料導入**標準輸出**
不要忘記```>```預設為**正常輸出**。
驗證:

ls no_file.txt > ls.txt 2>&1
cat ls.txt
ls: cannot access no_file.txt: No such file or directory
touch no_file.txt
ls no_file.txt > ls.txt 2>&1
cat ls.txt
no_file.txt

1
2
3
4
原先無```no_file.txt```此檔案,執行```ls```並將錯誤訊息寫入```ls.txt```,新增```no_file.txt```並再一次執行```ls```,可以看到輸出訊息有所變化且正常輸出與錯誤輸出資料都寫入```ls.txt```內。

導向語法有許多組合,以下都可以達到將正常輸出與錯誤輸出寫入同一個檔案內。
Example:

ls no_file.txt 2> ls.txt 1>&2
ls no_file.txt >& ls.txt
ls no_file.txt &> ls.txt

1
2
```>&``` ```&>```為將所有輸出導向至檔案。
實用例子:儲存iptables規則至檔案

iptables-save > iptables.rule

1
2
3
4

## 標準輸入
需要輸入資料的Linux程式若執行時沒有給予資料的話,預設將會從鍵盤讀取資料
Example:

cat
1
1
2
2

1
2
使用```<```,將指定的檔案設定為程式的標準輸入
Example:

cat < ls.txt
no_file.txt

1
2
3
4
5
6
7

## 管線 pipe
若要把兩個或以上的程式輸入出串連起來,可以使用```|```(pipe)。
![](https://i.imgur.com/sN2gQjM.png)
[Image Soruce:鳥哥的 Linux 私房菜 第十章、認識與學習BASH](http://linux.vbird.org/linux_basic/0320bash.php)

Example:

ls | nl

1 file1
2 file2
3 directory3

```

nl為顯示行數的指令。
也就是 ls的輸出透過pipe成為了nl的輸入,達到了可顯示行數的ls功能。

/dev/null

-
一個類似垃圾桶的東西,若將輸出指向至此,則所有輸出將會被捨棄掉。

Reference

GTW-Linux I/O 輸入與輸出重新導向,基礎概念教學

用 ELK 分析與儲存 log 紀錄

Preface

Before, we installed and configured the rsyslog , MySQL and LogAnalyzer.
Now we use anothor tools to help us collect logs , and get better statistics.

Objective

Deploy the ELK and get statistics to analyze logs.

System environvent

Transport iptables log file to log server
Same environment in this note,and install the ELK in log server.

Install ELK

Install Java-OpenSDK

1
yum install java

Download ElasticSearch and extract it

1
2
3
4
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.3.0-linux-x86_64.tar.gz
tar -zxvf elasticsearch-7.3.0-linux-x86_64.tar.gz
cd elasticsearch
bin/elasticsearch #run to install

modify config file from config/elasticsearch.yml

1
2
3
4
cluster.name: log-elasticsearch
network.host: $SERVER_IP
http.port: 9200
discovery.seed_hosts:["127.0.0.1","[::1]","[$SERVER_IP]"]

Download Kibana and extract it

1
2
3
4
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.3.0-linux-x86_64.tar.gz
tar -zxvf kibana-7.3.0-linux-x86_64.tar.gz
cd kibana/config
vim kibana.yml

modify config file from config/kibana.yml

1
2
3
server.port: 5601
server.host: $SERVER_NAME
elasticsearch.hosts: ["http://$elasticsearch_SERVER_IP:9200"]

Download Logstash and extract it

1
2
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.3.0.tar.gz
tar -zxvf logstash-7.3.0.tar.gz

Transport method

Transport rsyslog log files through LogStash

Logstash config file

1
2
3
4
5
6
7
8
9
imput{
syslog{
port => "514"
}
}
output{
elasticsearch{hosts => ["$Elasticsearch_SERVER:9200"]}
stdout{}
}

start up logstash

1
bin/logstash -f config/syslog.conf

Transport log files through Filebeat

Download filebeat plugin

1
2
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.0-linux-x86_64.tar.gz
tar -zxvf filebeat-7.3.0-linux-x86_64.tar.gz

Modify filebeat config from ./filebeat.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#=====Filebeat inputs=====
filebeat.inputs:
-type: log
enabled: true
paths:
-/var/log/*.log

#=====Kibana=====
setup.kibana:
host: "192.168.0.250:5601"

#=====Outputs=====

#-----ElasticSearch output-----
output.elasticsearch:
hosts: ["$elasticsearch_SERVER_IP:9200"]

Browse kibana/discover then can show log in screen.

Some Errors

Insufficient space for shared memory file

clean the disk.

1
2
3
4
df -h
du -h -x --max-depth=1
ps aux
kill

Create Kibana index pattern forbidden

1
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Screenshot

log from rsyslog client through logstash

log from local through filebeat

Reference

ELK 常用架构及使用场景介绍
集中式日志系统 ELK 协议栈详解
ELK 錯誤訊息 max file descriptors [4096] for elasticsearch process is too low
elasticsearch 7 单机配置

Transport iptables log file to log server

Objective

Setting iptables log function in log client,and transport log file to log server.
Deploy LogAnalyzer on log server.

System environment

All System use CentOS 7.6

  1. R1(Network Adapter *2)
  2. log server (in LAN)

Build iptables rules and rsyslog in R1

1
2
3
4
5
iptables -A INPUT -i $LAN_INTERFACE -j LOG --log-prefix "INPUT ICMPv4:" --log-level notice
iptables -A INPUT -p tcp -j LOG --log-prefix "INPUT TCP:" --log-level notice
iptables -A INPUT -p udp -j LOG --log-prefix "INPUT UDP:" --log-level notice
iptables -t nat -A POSTROUTING -o ens33 -J LOG --log-level notice --log-prefix "NAT Packet:"
iptables -t nat -A POSTROUTING -o ens33 -J MASQUERADE

Save log file in /var/log/iptables.log
edit /etc/rsyslog.conf

1
*.* @192.168.0.1:514  //log server ip or FQDN and be a log client

enable ipv4.ip_forward in sysctl.conf

1
net.ipv4.ip_forward = 1

Build rsyslog in log server

edit /etc/rsyslog.conf

1
2
3
4
5
6
7
8
9
#uncommend these lines
$ModLoad imudp
$UDPServerRun 514

#add these line to allow log client
$AllowedSender UDP,192.168.0.0/24
kern.=notice /var/log/iptables.log #Save log in file

*.info;*.!notice;mail.none;authpriv.none;cron.none;

Deploy LogAnalyzer in log server

Install apache2 and php
(In this project,php version is 7.3,and defalut version in CentOS 7 is 5.4)

1
yum install httpd php

Recommand:Reboot the computer after finishing install or LogAnalyzer will not install successfully.

Download LogAnalyzer from official website and extract it

1
2
wget http://download.adiscon.com/loganalyzer/loganalyzer-4.1.7.tar.gz
tar -zxvf loganalyzer-4.1.7.tar.gz

To show report-images in LogAnalyzer,need to install GD module.

1
2
yum install gd gd-devel php-gd
systemctl restart httpd

Browse $YOUR_SERVER_IP/loganalyzer/src/install.php to install LogAnalyzer.

Use MariaDB(MySQL) to store log files , and use phpMyAdmin to manage.

Install MariaDB and configure

1
2
3
4
yum install epel-release
yum install mariadb-server php-mysql rsyslog-mysql
mysql_secure_installation
mysql -u root -p #Login

In this project , phpMyAdmin version is 4.9 , php version need 5.5 or later.
Download phpMyAdmin setup file from official website , and extract to /var/www , browse $SERVER_IP/phpmyadmin/setup to set configuration.

1
2
3
mysql -u root -p logdb </usr/share/doc/rsyslog/mysql-createDB.sql #create database
mv config.php config.php.backup #re-configure
touch config.php

Edit rsyslog.conf

1
2
3
4
#load ommysql module
$ModLoad ommysql
#:ommysql:資料庫IP,資料庫名,使用者名稱,密碼
kern.=notice :ommysql:localhost,logdb,$dbuser,$dbpassword

Screenshot



Reference

自行架設LogAnalyzer日誌管理伺服器
LogAnalyzer日誌分析工具安裝設定詳解
CentOS Linux 7 安裝 MySQL/MariaDB 資料庫教學
rsyslog+mysql+loganalyzer記錄系統log至資料庫並由web介面呈現

ssl-bump

peek

When a peek rule matches during step1, Squid proceeds to step2 where it parses the TLS Client Hello and extracts SNI (if any). When a peek rule matches during step 2, Squid proceeds to step3 where it parses the TLS Server Hello and extracts server certificate while preserving the possibility of splicing the client and server connections; peeking at the server certificate usually precludes future bumping (see Limitations).

splice

Become a TCP tunnel without decoding the connection. The client and the server exchange data as if there is no proxy in between.

bump

Establish a TLS connection with the server (using client SNI, if any) and establish a TLS connection with the client (using a mimicked server certificate). However, this is not what actually happens right now if a bump rule matches during step1.

terminate

Close client and server connections.

關閉client與server的連線

stare

When a stare rule matches during step1, Squid proceeds to step2 where it parses the TLS Client Hello and extracts SNI (if any). When a stare rule matches during step2, Squid proceeds to step3 where it parses the TLS Server Hello and extracts server certificate while preserving the possibility of bumping the client and server connections; staring at the server certificate usually precludes future splicing (see Limitations).

情境劇stare and bump

stare在step3時將所有連線設定為splice,也就是一般的TCP Tunnel連線,若現在要與某一網站進行bump,bump設定將至於stare與splice之間,如此即可達到指定網站bump的功能。

Reference

Squid, SquidGuard, and Lightsquid on pfSense 2.4
SslBump Peek and Splice
Squid configuration directive ssl_bump

squid on CentOS 7

需求

Client不用設定 Proxy 伺服器 IP,即可讓client之流量皆經由squid發送,並對一特定網站限制連線,本機系統使用 CentOS 7.6。

系統建置

CentOS 7.6(need NetworkAdapter *2) CentOS 7 建置筆記

Enable ipv4_forward

1
vim /etc/sysctl.conf

加入

1
net.ipv4.ip_forward=1

apply

1
sysctl -p /etc/sysctl.conf

Squid Intercept 原理

用戶端不需設定proxy相關設定,所有http與https流量由NAT導向至proxy Server,proxy Server將成為LAN與WAN之中間人(squid-in-the-middle),此一功能除可避免手動設定proxy外,也可提升安全性需求。

squid 設定

squid.conf(add in config part )

1
2
3
4
5
6
7
8
9
10
11
12
13
http_port 3128 intercept #for http 

https_prot 3129 intercept ssl-bump /
generate-host-certificates=on dynamic_cert_mem_cache_size=16MB /
cert=/etc/squid/ssl_cert/gss.com.cert /
key=/etc/squid/ssl_cert/myCA.key #for http

#Configure SSL Bump
always_direct allow all #forward all request
acl bump_sites dstdomain "/etc/squid/ssl_cert/bump_sites.txt" #forbid domain
acl step1 at_step SslBump1 #connect client first
ssl_bump peek step1
ssl_bump bump bump_sites

make certificate(openSSL)

1
2
3
openssl genrsa -out myCA.key 2048
openssl req -new -key myCA.key -out gss.com.csr
openssl x509 -req -days 365 -in gss.com.csr -signkey myCA.key -out gss.com.cert

產生出cert後,client端需匯入此憑證。

iptables

將LAN端port 80(http)與443(https)流量導向至squid

1
2
3
iptables -t nat -A PREROUTING -i ens37 -p tcp --dport 80 -j REDIRECT --to-port 3128
iptables -t nat -A PREROUTING -i ens37 -p tcp --dport 443 -j REDIRECT --to-port 3129
iptables -t nat -A POSTROUTING -o ens33 -j MASQUERADE

Screenshot

備忘錄

  1. CentOS 7 系統有default iptables rules,建議先全部清除。
  2. 當前模式為intercept,若client端指定proxy ip,則流量將被squid forbid,所以當squid為transparent(intercept)mode時,就不應該手動設定proxy ip 。
  3. 要清楚是squid本身擋掉封包,還是iptables。(善用wireshark與access.log)
  4. SELinux要關閉。

參考文獻

wiki squid(NAT REDIRECT)
wiki squid(DNAT)
build squid proxy with ssl bumping
squid正向代理
acl 設定
squid SSL相關特性
Squid configuration directive ssl_bump

CentOS 7 基本環境建置

字型與字型大小

1
vi .bash_profile
1
2
3
if
setfont sun12x22 //man setfont取得更多說明
fi

網路介面設定

1
2
3
nmcli d //檢視網路介面啟動狀態
nmtui //設定程式
ip a //檢視網路卡當前ip address以及netmask

網路卡設定檔路徑

1
/etc/sysconfig/network-scripts/ifcfg-**AdapterName**

安裝vim

1
yum install vim