HeZephyr贺志飞的博客
HeZephyr贺志飞的博客

Shell 23道例题实战

HeZephyr HeZephyr 收录于 系统架构
  约 11100 字 预计阅读 23 分钟  - 次阅读  

1 统计文件的行数

编写一个shell脚本以输出一个文本文件nowcoder.txt中的行数

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#!/bin/bash

# 方法1：使用 wc -l 和 awk
# 统计行数并使用 awk 提取第一个字段，即行数
lines=$(wc -l nowcoder.txt | awk '{print $1}')
echo "使用 wc -l 和 awk：$lines 行"

# 方法2：通过输入流传递文件内容给 wc -l
# 使用 < 操作符
lines=$(wc -l < nowcoder.txt)
echo "通过输入流：$lines 行"

# 方法3：使用 cat 和管道传递给 wc -l
# 使用 cat 命令和管道
lines=$(cat nowcoder.txt | wc -l)
echo "通过管道：$lines 行"

# 方法4：使用 sed 统计行数
# 使用 sed 的 -n '$=' 选项
lines=$(sed -n '$=' nowcoder.txt)
echo "使用 sed：$lines 行"

2 打印文件的最后5行

查看日志的时候，经常会从文件的末尾往前查看，请你写一个bash shell脚本以输出一个文本文件nowcoder.txt中的最后5行。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

#!/bin/bash

# 查看文件的前5行
echo "前5行："
head -5 nowcoder.txt
echo ""

# 查看文件的后5行
echo "后5行："
tail -5 nowcoder.txt
echo ""

# 查看文件的第5行到第20行
echo "第5行到第20行："
sed -n '5,20p' nowcoder.txt

3 输出 0 到 500 中 7 的倍数

写一个 bash脚本以输出数字 $0$ 到 $500$ 中 $7$ 的倍数$(0 7 14 21…)$的命令

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#!/bin/bash

# 方法1：使用 Bash 的扩展语法的 for 循环
echo "方法1：使用 Bash 的扩展语法的 for 循环"
for item in {0..500..7}
do 
    echo $item
done

echo "" # 分隔行

# 方法2：使用 seq 命令 
# seq [选项]... 首部 增量 尾部
echo "方法2：使用 seq 命令"
seq 0 7 500

echo "" # 分隔行

# 方法3：使用 while 循环
echo "方法3：使用 while 循环"
# 初始化变量
i=0
# 使用 while 循环
while [ $i -le 500 ]
do
    # 输出当前的 7 的倍数
    echo $i
    
    # 增加 7
    i=$((i + 7))
done

4 输出第5行的内容

编写一个bash脚本以输出一个文本文件nowcoder.txt中第$5$行的内容。

1
2
3
4
5
6
7
8

#!/bin/bash

# head 命令拿到前五行，再通过通道，通过tail取出来最后一行，即第五行
head -n 5 nowcoder.txt | tail -n 1
#!/bin/bash

# 使用sed 命令中的 p选项，打印第五行
sed -n 5p nowcoder.txt

5 打印空行的行号

编写一个shell脚本以输出一个文本文件nowcoder.txt中空行的行号（空行可能连续，从1开始输出）

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/bin/bash

# 使用 grep 命令匹配所有空行，并且输出匹配的行号。-n 选项表示输出匹配行的行号，'^$' 匹配空行。使用 cut 命令以 : 作为分隔符，提取每行的第一个字段，即行号。
grep -n '^$' nowcoder.txt | cut -d':' -f1

# 使用 awk 命令，NF 表示当前行的字段数，NR 表示当前行号。当字段数为0时，即当前行为空行，{ print NR } 输出当前行的行号。
awk 'NF == 0 { print NR }' nowcoder.txt

# 使用 sed 命令匹配所有空行，并输出匹配行的行号。-n 选项表示只输出指定的行，/^$/ 匹配空行，=表示输出匹配行的行号。
sed -n '/^$/=' nowcoder.txt

6 去掉空行

写一个 bash脚本以去掉一个文本文件nowcoder.txt中的空行

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/bin/bash

# 使用 grep 命令匹配所有非空行。-v 选项表示反转匹配，'^$' 匹配空行。
grep -v '^$' nowcoder.txt

# 使用 sed 命令删除匹配空行的行。/^$/ 匹配空行，d 命令删除匹配的行
sed '/^$/d' nowcoder.txt > nowcoder_no_empty_lines.txt

# 使用 awk 命令，NF 表示字段数，NF 为真时表示非空行。
awk 'NF' nowcoder.txt

7 打印字母数小于8的单词

写一个bash脚本以统计一个文本文件nowcoder.txt中字母数小于8的单词。

1
2
3
4
5
6
7

#!/bin/bash

# 使用 awk 命令遍历每个单词，NF 表示当前行的单词数，length($i) 表示当前单词的字母数，如果字母数小于8，则打印当前单词。
awk '{ for (i=1; i<=NF; i++) if (length($i)<8) print $i }' nowcoder.txt

# 使用 grep 命令匹配字母数小于8的单词。-o 选项表示只输出匹配的内容，\b 表示单词边界，\w\{1,7\} 匹配字母数在1到7之间的单词。
grep -o '\b\w\{1,7\}\b' nowcoder.txt

8 统计所有进程占用内存百分比的和

假设 nowcoder.txt 内容如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.4  77744  8332 ?        Ss    2021   1:15 /sbin/init noibrs splash
root         2  0.0  0.0      0     0 ?        S     2021   0:00 [kthreadd]
root         4  0.0  0.0      0     0 ?        I<    2021   0:00 [kworker/0:0H]
daemon     486  0.0  0.1  28340  2372 ?        Ss    2021   0:00 /usr/sbin/atd -f
root       586  0.0  0.3  72308  6244 ?        Ss    2021   0:01 /usr/sbin/sshd -D
root     12847  0.0  0.0   4528    68 ?        S<   Jan03   0:13 /usr/sbin/atopacctd
root     16306  1.7  1.2 151964 26132 ?        S<sl Apr15 512:03 /usr/local/aegis/aegis_client/aegis_11_25/AliYunDun
root     24143  0.0  0.4  25608  8652 ?        S<Ls 00:00   0:03 /usr/bin/atop -R -w /var/log/atop/atop_20220505 600
root     24901  0.0  0.3 107792  7008 ?        Ss   15:37   0:00 sshd: root@pts/0
root     24903  0.0  0.3  76532  7580 ?        Ss   15:37   0:00 /lib/systemd/systemd --user
root     24904  0.0  0.1 111520  2392 ?        S    15:37   0:00 (sd-pam)

以上内容是通过ps aux命令输出到nowcoder.txt文件中的，请你写一个脚本计算一下所有进程占用内存大小的和。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

#!/bin/bash

# 使用awk命令过滤到第一行并累加$4
awk 'BEGIN { sum=0 } NR > 1 { sum+=$4 } END { print sum }' nowcoder.txt

# 使用while循环读取，并用if跳过第一行，使用bc进行浮点数加法运算
sum=0
cnt=1
while read -r line;
do
    if [ $cnt -gt 1 ]; then
        mem=$(echo $line | awk '{ print $4 }')
        sum=$(echo "$sum+$mem" | bc)
    fi
    cnt=$((cnt+1))
done
echo $sum
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/bin/bash

sum=0
tail -n +2 nowcoder.txt | while read -r line;
do
    mem=$(echo $line | awk '{ print $4 }')
    sum=$(echo "$sum+$mem" | bc)
    echo $sum
done
echo $sum

在 Bash 中，管道中的命令会在子 shell 中执行，因此变量修改不会影响主 shell 中的变量。这就是为什么看到 sum 在循环内部被正确更新，但在循环外部仍然是初始值 0。

9 统计每个单词出现的个数

写一个bash脚本以统计一个文本文件nowcoder.txt 中每个单词出现的个数。

为了简单起见，你可以假设： nowcoder.txt只包括小写字母和空格，每个单词只由小写字母组成，单词间由一个或多个空格字符分隔。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/bin/bash

# 将空格转换为换行符，以便每个单词占一行
tr -s ' ' '\n' <nowcoder.txt |
	# 对单词进行排序
	sort |
	# 统计每个单词的出现次数
	uniq -c |
	# 调整输出格式为"单词 词频"
	awk '{ print $2, $1 }' |
	# 按词频升序排序，-k2,2 意味着只使用第二列进行排序，表示按数值进行排序（默认情况按字典序排序）
	sort -k2,2n 
	
# 使用 awk 统计每个单词的出现次数
# NF 表示当前行的字段数，即单词数
# 使用一个关联数组 cnt 存储每个单词出现的次数

awk '{
    for (i=1; i<=NF; i++)  # 遍历当前行的每个单词
        cnt[$i] += 1        # 将单词加入关联数组 cnt，统计出现次数
}
END {
    for (x in cnt)           # 遍历关联数组 cnt
        print x, cnt[x]       # 输出单词和对应的出现次数
}' nowcoder.txt | sort -k2,2n

10 第二列是否有重复

给定一个nowcoder.txt文件，其中有3列信息，如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

20201001 python 99
20201002 go 80
20201002 c++ 88
20201003 php 77
20201001 go 88
20201005 shell 89
20201006 java 70
20201008 c 100
20201007 java 88
20201006 go 97

编写一个shell脚本来检查文件第二列是否有重复，且有几个重复，并提取出重复的行的第二列信息（先按次数排序，如果次数相同，按照单词字母顺序排序），输入如下：

1
2

2 java
3 go
1
2
3
4
5
6
7
8

#!/bin/bash

cat nowcoder.txt | 
awk '{ print $2 }' | 
sort | uniq -c | 
awk '{ print $1, $2 }' | # 重新格式化输出
sort -k1,1n -k2,2 | # 按照出现次数和字母顺序排序
grep -v '1'	# 过滤出现次数不为 1 的行

11 转置文件的内容

写一个bash脚本来转置文本文件nowcoder.txt中的文件内容。 文件中每行列数相同，并且每个字段由空格分隔

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/bin/bash

# 读取文件并使用 awk 转置文件内容
awk '
{
    # 遍历当前行的每一个字段
    for (i = 1; i <= NF; i++) {
        a[NR, i] = $i  # 将每个字段存储在一个二维数组中，a[行号, 列号] = 值
    }
}
NF > p { p = NF }  # 如果当前行的字段数大于 p，则更新 p 为当前行的字段数

END {
    # 遍历每一列（由最大字段数 p 确定）
    for (i = 1; i <= p; i++) {
        # 遍历每一行（由总行数 NR 确定）
        for (j = 1; j <= NR; j++) {
            printf("%s%s", a[j,i], (j==NR ? "" : " "))  # 输出数组中对应的字段值，并在每个字段后添加空格，除非是最后一个字段
        }
        printf("\n")  # 每一列输出完之后换行
    }
}' nowcoder.txt

12 打印每一行出现的数字个数

写一个bash脚本，统计一个文本文件nowcoder.txt中每一行出现的1~5数字的个数，并且计算一下整个文档中一共出现了几个1~5数字的总数。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

#!/bin/bash

# 使用 awk 读取文件并统计每行中包含的特定数字（1, 2, 3, 4, 5）的数量
awk -F "[1,2,3,4,5]" '
BEGIN { 
    sum = 0  # 初始化 sum 变量，用于存储总和
} {
    # 打印当前行号 NR 以及当前行中包含的特定数字的数量 (NF - 1)
    print("line" NR " number: " (NF - 1))
    
    # 将当前行中包含的特定数字的数量累加到 sum
    sum += (NF - 1)
} END {
    # 打印总和
    print("sum is " sum)
}' nowcoder.txt

13 去掉所有包含this的句子

编写一个shell脚本以实现如下功能：去掉输入中含有this的语句，把不含this的语句输出

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/bin/bash

# -v 反转匹配
grep -v "this" nowcoder.txt

# sed 命令 -> d 删除 -> // 包含要搜索的字符串
sed '/this/d' nowcoder.txt

# awk 命令，$0为当前行的所有内容，!~ 是 awk 的模式匹配运算符，表示模式不匹配
awk '$0!~/this/ {print $0}' nowcoder.txt

14 求平均值

写一个bash脚本以实现一个需求，求输入的一个数组的平均值

第1行为输入的数组长度N

第2~N行为数组的元素，如以下为: 数组长度为4，数组元素为1 2 9 8

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

#!/bin/bash

awk 'BEGIN {
    sum = 0
}{
    if (NR == 1) {
        N = $1  # 将第一行的数字数量保存到变量 N 中
    } else {
        sum += $1  # 对随后的数字进行累加求和
    }
} END {
    printf("%.3f", sum / N)  # 输出平均值，保留三位小数
}'

15 去掉不需要的单词

写一个bash脚本以实现一个需求，去掉输入中含有B和b的单词。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

#!/bin/bash

# 使用 sed -n 命令打印不包含 'B' 和 'b' 的行
# /^[^bB]*$/ 表示匹配不包含 'B' 和 'b' 的行，^ 表示行开头，[^bB] 表示不包含 'B' 和 'b' 的任何字符，* 表示零次或多次重复，$ 表示行结尾
sed -n '/^[^bB]*$/p' nowcoder.txt

# 使用 grep -E -v 命令排除包含 'B' 和 'b' 的行
# -E 选项启用扩展的正则表达式，-v 选项表示反转匹配
grep -E -v "[bB]" nowcoder.txt

# 使用 awk 命令，遍历每个单词，如果不包含 'B' 和 'b'，则输出该单词
awk '{
    for (i = 1;i <= NF; i++) {
        if ($i !~ /b|B/) {  # 使用正则表达式匹配单词中不包含 'B' 和 'b' 的部分
            printf("%s ", $i)  # 输出不包含 'B' 和 'b' 的单词
        }
    }
}' nowcoder.txt

16 判断输入的是否为IP地址

写一个脚本统计文件nowcoder.txt中的每一行是否是正确的IP地址。

如果是正确的IP地址输出：yes

如果是错误的IP地址，且是四段号码的话输出：no，否则的话输出：error

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

#!/bin/bash

awk -F "." '{
    flag = "error"
    if (NF == 4) {
        flag = "yes"
        for (i = 1; i <= NF; i++) {
            if ($i > 255) {
                flag = "no";
                break;
            }
        }
    }
    printf(flag"\n")
}' nowcoder.txt

17 将字段逆序输出文件的每行

编写一个shell脚本，将文件nowcoder.txt中每一行的字段逆序输出，其中字段之间使用英文冒号:相分隔。

假设nowcoder.txt内容如下：

1
2

nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
root:*:0:0:System Administrator:/var/root:/bin/sh

你的脚本应当输出

1
2

/usr/bin/false:/var/empty:Unprivileged User:-2:-2:*:nobody
/bin/sh:/var/root:System Administrator:0:0:*:root
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/bin/bash

awk -F ":" '{
    for (i = 1; i <= NF; i++) {
        temp[i] = $i
    }
    for (i = NF; i >= 1; i--) {
        printf("%s%s", temp[i], (i == 1 ? "\n" : ":"))
    }
}' nowcoder.txt

18 域名进行计数排序处理

假设有一些域名，存储在nowcoder.txt里，现在需要写一个shell脚本，将域名取出并根据域名进行计数排序处理（降序）。

假设nowcoder.txt内容如下：

1
2
3

http://www.nowcoder.com/index.html
http://www.nowcoder.com/1.html
http://m.nowcoder.com/index.html

你的脚本应该输出：

1
2

2 www.nowcoder.com
1 m.nowcoder.com
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#!/bin/bash

awk -F '/' '{
    print($3)
}' nowcoder.txt |
sort |
uniq -c |
sort -r |
awk '{
    print($1" "$2)
}'

19 打印等腰三角形

编写一个shell脚本，输入正整数n，打印边长为n的等腰三角形。

示例：

输入：5

输出：

1
2
3
4
5

    *
   * *
  * * *
 * * * *
* * * * *
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

#!/bin/bash

read n
for ((i = 1; i <= n; i++))
do
    for ((j = 1; j <= n - i; j++))
    do
        printf " "
    done
    for ((j = 1; j <= i; j++))
    do
        if [[ $j -eq $i ]]; then
            printf "*"
        else
            printf "* "
        fi
    done
    printf "\n"
done

20 打印只有一个数字的行

假设有一个nowcoder.txt，编写脚本，打印只有一个数字的行。

1
2
3
4
5
6
7

#!/bin/bash

awk -F "[0-9]" '{
    if (NF == 2) {
        print($0)
    }
}'

21 格式化输出

有一个文件nowcoder.txt，里面的每一行都是一个数字串，编写一个shell脚本对文件中每一行的数字串进行格式化：每$3$个数字加入一个逗号（,）。

例如：数字串为“123456789”，那么需要格式化为123,456,789。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#!/bin/bash

# 使用-F分割数字串
awk -F "" '{
    for (i = 1; i <= NF; i++) {
        printf($i)
        if ((NF - i) % 3 == 0 && i != NF) {
            printf(",")
        }
    }
    printf("\n")
}'

22 处理文本

有一个文本文件nowcoder.txt，假设内容格式如下：

1
2
3
4
5

111:13443
222:13211
111:13643
333:12341
222:12123

现在需要编写一个shell脚本，按照以下的格式输出：

1
2
3
4
5
6
7
8

[111]
13443
13643
[222]
13211
12123
[333]
12341
1
2
3
4
5
6
7
8
9

#!/bin/bash

awk -F ":" '{
    cnt[$1] = cnt[$1] $2 "\n"
} END {
    for (i in cnt) {
        printf("[%s]\n%s", i, cnt[i])
    }
}' nowcoder.txt

23 Nginx日志分析1-IP访问次数统计

假设 Nginx 的日志存储在 nowcoder.txt 里，内容如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"

现在需要编写 Shell 脚本统计出 2020 年 4 月 23 号访问 IP 的对应次数，并且按照次数降序排序。你的脚本应该输出：

1
2
3
4
5

5 192.168.1.22
4 192.168.1.21
3 192.168.1.20
2 192.168.1.25
1 192.168.1.24
1
2
3
4
5
6
7
8
9

#!/bin/bash

# 通过grep过滤，再统计排序
grep "23/Apr/2020" nowcoder.txt |
awk '{ print $1 }' |
sort |
uniq -c |
sort -r |
awk '{ print $1, $2 }'

24 Nginx日志分析2-统计某个时间段的IP访问量

假设 Nginx 的日志存储在 nowcoder.txt 里，内容如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"

现在需要编写 Shell 脚本统计 2020年04月23日20点至23点去重后的 IP 访问量，你的脚本应该输出：

1

5

输出说明：2020年04月23日20点至23点，共有 192.168.1.24、192.168.1.25、192.168.1.20、192.168.1.21、192.168.1.22 共 5 个 IP 访问了。

1
2
3
4
5
6
7

#!/bin/bash

grep "23/Apr/2020:2[0-3]" nowcoder.txt |
awk '{ print $1 }' |
sort |
uniq |
wc -l

25 nginx日志分析3-统计访问3次以上的IP

假设nginx的日志我们存储在nowcoder.txt里，格式如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"

现在需要编写shell脚本统计访问3次以上的IP，你的脚本应该输出：

1
2
3

6 192.168.1.22
5 192.168.1.21
4 192.168.1.20
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#!/bin/bash

awk '{ print $1 }' nowcoder.txt |
sort |
uniq -c |
sort -r |
awk '{
    if ($1 > 3) {
        print $1, $2
    }
}'

26 Nginx日志分析4-查询某个IP的详细访问情况

假设Nginx的日志存储在nowcoder.txt里，内容如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

192.168.1.20 - - [21/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:16:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"

现在需要编写shell脚本查询192.168.1.22的详细访问次数情况，按访问频率降序排序。你的脚本应该输出：

1
2

4 /1/index.php
2 /3/index.php
1
2
3
4
5
6
7
8

#!/bin/bash

grep "192.168.1.22" |
awk '{ print $7 }' |
sort |
uniq -c |
sort -r |
awk '{ print $1, $2 }'

27 nginx日志分析5-统计爬虫抓取404的次数

假设nginx的日志存储在nowcoder.txt里，内容如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

192.168.1.20 - - [21/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 301 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:21:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:05:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 200 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 300 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 500 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:59:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"

现在需要编写shell脚本统计百度爬虫抓取404的次数，你的脚本应该输出

1

2
1
2
3

#!/bin/bash

grep "404" | grep "www.baidu.com" | wc -l

28 Nginx日志分析6-统计每分钟的请求数

假设Nginx的日志存储在nowcoder.txt里，内容如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

192.168.1.20 - - [21/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [21/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [21/Apr/2020:21:21:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.23 - - [21/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [22/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [22/Apr/2020:15:26:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:08:05:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Baiduspider"
192.168.1.21 - - [23/Apr/2020:09:20:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:10:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:14:12:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:15:00:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:15:00:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Baiduspider"
192.168.1.25 - - [23/Apr/2020:16:15:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.24 - - [23/Apr/2020:20:27:49 +0800] "GET /2/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.25 - - [23/Apr/2020:20:27:49 +0800] "GET /3/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.20 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:20:27:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.22 - - [23/Apr/2020:22:10:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"
192.168.1.21 - - [23/Apr/2020:23:59:49 +0800] "GET /1/index.php HTTP/1.1" 404 490 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0"

现在需要编写Shell脚本统计每分钟的请求数，并且按照请求数降序排序。你的脚本应该输出：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

5 20:27
4 15:00
2 22:10
2 14:12
2 10:27
1 23:59
1 21:21
1 16:15
1 15:26
1 09:20
1 08:05
1
2
3
4
5
6
7

#!/bin/bash

awk -F ":" '{ print $2":"$3 }' |
sort |
uniq -c |
sort -r |
awk '{ print $1, $2 }'

29 netstat练习1-查看各个状态的连接数

假设netstat命令运行的结果我们存储在nowcoder.txt里，格式如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6160            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 172.16.56.200:41856     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49822     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49674     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:42316     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:44076     172.16.240.74:6379      ESTABLISHED
tcp        0      0 172.16.56.200:49656     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:58248     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:50108     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41944     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:35548     100.100.32.118:80       TIME_WAIT
tcp        0      0 172.16.56.200:39024     100.100.45.106:443      TIME_WAIT
tcp        0      0 172.16.56.200:41788     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58260     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:41812     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:41854     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58252     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:49586     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41754     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:50466     120.55.222.235:80       TIME_WAIT
tcp        0      0 172.16.56.200:38514     100.100.142.5:80        TIME_WAIT
tcp        0      0 172.16.56.200:49832     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:52162     100.100.30.25:80        ESTABLISHED
tcp        0      0 172.16.56.200:50372     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:50306     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49600     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41908     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:60292     100.100.142.1:80        TIME_WAIT
tcp        0      0 172.16.56.200:37650     100.100.54.133:80       TIME_WAIT
tcp        0      0 172.16.56.200:41938     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49736     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41890     172.16.34.144:3306      ESTABLISHED
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 0.0.0.0:45881           0.0.0.0:*
udp        0      0 127.0.0.53:53           0.0.0.0:*
udp        0      0 172.16.56.200:68        0.0.0.0:*
udp6       0      0 ::1:323                 :::*
raw6       0      0 :::58                   :::*                    7

现在需要编写shell脚本查看系统tcp连接中各个状态的连接数，并且按照连接数降序输出。你的脚本应该输出如下：

1
2
3

ESTABLISHED 22
TIME_WAIT 9
LISTEN 3
1
2
3
4
5
6
7
8

#!/bin/bash

grep "tcp" |
awk '{ print $6 }' |
sort |
uniq -c |
sort -nr |
awk '{ print $2, $1 }'

30 netstat练习2-查看和3306端口建立的连接

假设netstat命令运行的结果我们存储在nowcoder.txt里，格式如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6160            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 172.16.56.200:41856     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49822     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49674     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:42316     172.16.34.143:3306      ESTABLISHED
tcp        0      0 172.16.56.200:44076     172.16.240.74:6379      ESTABLISHED
tcp        0      0 172.16.56.200:49656     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:58248     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:50108     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41944     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:35548     100.100.32.118:80       TIME_WAIT
tcp        0      0 172.16.56.200:39024     100.100.45.106:443      TIME_WAIT
tcp        0      0 172.16.56.200:41788     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58260     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:41812     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:41854     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58252     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:49586     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41754     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:50466     120.55.222.235:80       TIME_WAIT
tcp        0      0 172.16.56.200:38514     100.100.142.5:80        TIME_WAIT
tcp        0      0 172.16.56.200:49832     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:52162     100.100.30.25:80        ESTABLISHED
tcp        0      0 172.16.56.200:50372     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:50306     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49600     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41908     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:60292     100.100.142.1:80        TIME_WAIT
tcp        0      0 172.16.56.200:37650     100.100.54.133:80       TIME_WAIT
tcp        0      0 172.16.56.200:41938     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49736     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41890     172.16.34.144:3306      ESTABLISHED
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 0.0.0.0:45881           0.0.0.0:*
udp        0      0 127.0.0.53:53           0.0.0.0:*
udp        0      0 172.16.56.200:68        0.0.0.0:*
udp6       0      0 ::1:323                 :::*
raw6       0      0 :::58                   :::*                    7

现在需要你查看和本机3306端口建立连接并且状态是established的所有IP，按照连接数降序排序。你的脚本应该输出

1
2
3

10 172.16.0.24
9 172.16.34.144
1 172.16.34.143
1
2
3
4
5
6
7
8
9

#!/bin/bash

grep "tcp.*ESTABLISHED" |
awk '{ print $5 }' |
awk -F ":" '$2 == 3306 { print $1 }' |
sort |
uniq -c |
sort -nr |
awk '{ print $1, $2 }'

31 netstat练习3-输出每个IP的连接数

假设netstat命令运行的结果我们存储在nowcoder.txt里，格式如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6160            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 172.16.56.200:41856     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49822     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49674     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:42316     172.16.34.143:3306      ESTABLISHED
tcp        0      0 172.16.56.200:44076     172.16.240.74:6379      ESTABLISHED
tcp        0      0 172.16.56.200:49656     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:58248     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:50108     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41944     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:35548     100.100.32.118:80       TIME_WAIT
tcp        0      0 172.16.56.200:39024     100.100.45.106:443      TIME_WAIT
tcp        0      0 172.16.56.200:41788     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58260     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:41812     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:41854     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58252     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:49586     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41754     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:50466     120.55.222.235:80       TIME_WAIT
tcp        0      0 172.16.56.200:38514     100.100.142.5:80        TIME_WAIT
tcp        0      0 172.16.56.200:49832     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:52162     100.100.30.25:80        ESTABLISHED
tcp        0      0 172.16.56.200:50372     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:50306     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49600     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41908     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:60292     100.100.142.1:80        TIME_WAIT
tcp        0      0 172.16.56.200:37650     100.100.54.133:80       TIME_WAIT
tcp        0      0 172.16.56.200:41938     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49736     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41890     172.16.34.144:3306      ESTABLISHED
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 0.0.0.0:45881           0.0.0.0:*
udp        0      0 127.0.0.53:53           0.0.0.0:*
udp        0      0 172.16.56.200:68        0.0.0.0:*
udp6       0      0 ::1:323                 :::*
raw6       0      0 :::58                   :::*                    7

现在需要你输出每个IP的连接数，按照连接数降序排序。你的脚本应该输出

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

172.16.0.24 10
172.16.34.144 9
100.100.142.4 3
0.0.0.0 3
172.16.34.143 1
172.16.240.74 1
120.55.222.235 1
100.100.54.133 1
100.100.45.106 1
100.100.32.118 1
100.100.30.25 1
100.100.142.5 1
100.100.142.1 1
1
2
3
4
5
6
7
8
9

#!/bin/bash

grep "tcp" |
awk '{ print $5 }' |
awk -F ":" '{ print $1 }' |
sort |
uniq -c |
sort -nr |
awk '{ print $2, $1 }'

32 netstat练习4-输出和3306端口建立连接总的各个状态的数目

假设netstat命令运行的结果我们存储在nowcoder.txt里，格式如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:6160            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 172.16.56.200:41856     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49822     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49674     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:42316     172.16.34.143:3306      ESTABLISHED
tcp        0      0 172.16.56.200:44076     172.16.240.74:6379      ESTABLISHED
tcp        0      0 172.16.56.200:49656     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:58248     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:50108     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41944     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:35548     100.100.32.118:80       TIME_WAIT
tcp        0      0 172.16.56.200:39024     100.100.45.106:443      TIME_WAIT
tcp        0      0 172.16.56.200:41788     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58260     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:41812     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:41854     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:58252     100.100.142.4:80        TIME_WAIT
tcp        0      0 172.16.56.200:49586     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41754     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:50466     120.55.222.235:80       TIME_WAIT
tcp        0      0 172.16.56.200:38514     100.100.142.5:80        TIME_WAIT
tcp        0      0 172.16.56.200:49832     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:52162     100.100.30.25:80        ESTABLISHED
tcp        0      0 172.16.56.200:50372     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:50306     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:49600     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41908     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:60292     100.100.142.1:80        TIME_WAIT
tcp        0      0 172.16.56.200:37650     100.100.54.133:80       TIME_WAIT
tcp        0      0 172.16.56.200:41938     172.16.34.144:3306      ESTABLISHED
tcp        0      0 172.16.56.200:49736     172.16.0.24:3306        ESTABLISHED
tcp        0      0 172.16.56.200:41890     172.16.34.144:3306      ESTABLISHED
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 0.0.0.0:45881           0.0.0.0:*
udp        0      0 127.0.0.53:53           0.0.0.0:*
udp        0      0 172.16.56.200:68        0.0.0.0:*
udp6       0      0 ::1:323                 :::*
raw6       0      0 :::58                   :::*                    7

现在需要你输出和本机3306端口建立连接的各个状态的数目，按照以下格式输出 TOTAL_IP表示建立连接的ip数目

TOTAL_LINK表示建立连接的总数目

1
2
3

TOTAL_IP 3
ESTABLISHED 20
TOTAL_LINK 20
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#!/bin/bash

# 统计包含3306且协议为tcp的总IP数量
TOTAL_IP=$(grep "tcp" nowcoder.txt |
	awk '{ print $5 }' |
	awk -F: '$2 == 3306 {print $1, $2 }' |
	sort |
	uniq |
	wc -l)

echo "TOTAL_IP $TOTAL_IP"

# 统计包含3306且状态为ESTABLISHED且协议为tcp的数量
ESTABLISHED=$(awk '/3306/ { if ($6 == "ESTABLISHED" && $1 == "tcp") print $5 }' nowcoder.txt |
	wc -l)

echo "ESTABLISHED $ESTABLISHED"

# 统计包含3306且协议为tcp的连接数量
TOTAL_LINK=$(awk '/3306/ { if ($1 == "tcp") print $5 }' nowcoder.txt |
	wc -l)

echo "TOTAL_LINK $TOTAL_LINK"

33 业务分析-提取值

假设我们的日志nowcoder.txt里，内容如下

1
2
3
4
5

12-May-2017 10:02:22.789 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:Apache Tomcat/8.5.15
12-May-2017 10:02:22.813 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:May 5 2017 11:03:04 UTC
12-May-2017 10:02:22.813 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:8.5.15.0
12-May-2017 10:02:22.814 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:Windows, OS Version:10
12-May-2017 10:02:22.814 信息 [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:x86_64

现在需要你提取出对应的值，输出内容如下

1
2
3
4

serverVersion:Apache Tomcat/8.5.15
serverName:8.5.15.0
osName:Windows
osVersion:10
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/bin/bash

grep -o "Server version:.*" nowcoder.txt |
	awk -F ":" '{print "serverVersion:" $2}'
grep -o "Server number:.*" nowcoder.txt |
	awk -F ":" '{print "serverName:" $2}'
grep -o "OS Name:.*" nowcoder.txt |
	awk -F "[:,]" '{print "osName:" $2}'
grep -o "OS Version:.*" nowcoder.txt |
	awk -F ":" '{print "osVersion:" $2}'

34 ps分析-统计VSZ,RSS各自总和

假设命令运行的结果我们存储在nowcoder.txt里，格式如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  37344  4604 ?        Ss    2020   2:13 /sbin/init
root       231  0.0  1.5 166576 62740 ?        Ss    2020  15:15 /lib/systemd/systemd-journald
root       237  0.0  0.0      0     0 ?        S<    2020   2:06 [kworker/0:1H]
root       259  0.0  0.0  45004  3416 ?        Ss    2020   0:25 /lib/systemd/systemd-udevd
root       476  0.0  0.0      0     0 ?        S<    2020   0:00 [edac-poller]
root       588  0.0  0.0 276244  2072 ?        Ssl   2020   9:49 /usr/lib/accountsservice/accounts-daemon
message+   592  0.0  0.0  42904  3032 ?        Ss    2020   0:01 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       636  0.0  0.0  65532  3200 ?        Ss    2020   1:51 /usr/sbin/sshd -D
daemon     637  0.0  0.0  26044  2076 ?        Ss    2020   0:00 /usr/sbin/atd -f
root       639  0.0  0.0  29476  2696 ?        Ss    2020   3:29 /usr/sbin/cron -f
root       643  0.0  0.0  20748  1992 ?        Ss    2020   0:26 /lib/systemd/systemd-logind
syslog     645  0.0  0.0 260636  3024 ?        Ssl   2020   3:17 /usr/sbin/rsyslogd -n
root       686  0.0  0.0 773124  2836 ?        Ssl   2020  26:45 /usr/sbin/nscd
root       690  0.0  0.0  19472   252 ?        Ss    2020  14:39 /usr/sbin/irqbalance --pid=/var/run/irqbalance.pid
ntp        692  0.0  0.0  98204   776 ?        Ss    2020  25:18 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 108:114
uuidd      767  0.0  0.0  28624   192 ?        Ss    2020   0:00 /usr/sbin/uuidd --socket-activation
root       793  0.0  0.0 128812  3148 ?        Ss    2020   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data   794  0.0  0.2 133376  9120 ?        S     2020 630:57 nginx: worker process
www-data   795  0.0  0.2 133208  8968 ?        S     2020 633:02 nginx: worker process
www-data   796  0.0  0.2 133216  9120 ?        S     2020 634:24 nginx: worker process
www-data   797  0.0  0.2 133228  9148 ?        S     2020 632:56 nginx: worker process
web        955  0.0  0.0  36856  2112 ?        Ss    2020   0:00 /lib/systemd/systemd --user
web        956  0.0  0.0  67456  1684 ?        S     2020   0:00 (sd-pam)
root      1354  0.0  0.0   8172   440 tty1     Ss+   2020   0:00 /sbin/agetty --noclear tty1 linux
root      1355  0.0  0.0   7988   344 ttyS0    Ss+   2020   0:00 /sbin/agetty --keep-baud 115200 38400 9600 ttyS0 vt220
root      2513  0.0  0.0      0     0 ?        S    13:07   0:00 [kworker/u4:1]
root      2587  0.0  0.0      0     0 ?        S    13:13   0:00 [kworker/u4:2]
root      2642  0.0  0.0      0     0 ?        S    13:17   0:00 [kworker/1:0]
root      2679  0.0  0.0      0     0 ?        S    13:19   0:00 [kworker/u4:0]
root      2735  0.0  0.1 102256  7252 ?        Ss   13:24   0:00 sshd: web [priv]
web       2752  0.0  0.0 102256  3452 ?        R    13:24   0:00 sshd: web@pts/0
web       2753  0.5  0.1  14716  4708 pts/0    Ss   13:24   0:00 -bash
web       2767  0.0  0.0  29596  1456 pts/0    R+   13:24   0:00 ps aux
root     10634  0.0  0.0      0     0 ?        S    Nov16   0:00 [kworker/0:0]
root     16585  0.0  0.0      0     0 ?        S<    2020   0:00 [bioset]
root     19526  0.0  0.0      0     0 ?        S    Nov16   0:00 [kworker/1:1]
root     28460  0.0  0.0      0     0 ?        S    Nov15   0:03 [kworker/0:2]
root     30685  0.0  0.0  36644  2760 ?        Ss    2020   0:00 /lib/systemd/systemd --user
root     30692  0.0  0.0  67224  1664 ?        S     2020   0:00 (sd-pam)
root     32689  0.0  0.0  47740  2100 ?        Ss    2020   0:00 /usr/local/ilogtail/ilogtail
root     32691  0.2  0.5 256144 23708 ?        Sl    2020 1151:31 /usr/local/ilogtail/ilogtail

现在需要你统计VSZ，RSS各自的总和（以M兆为统计），输出格式如下

1
2

MEM TOTAL
VSZ_SUM:3250.8M,RSS_SUM:179.777M
1
2
3
4
5
6
7

#!/bin/bash

awk '{
    sum_vsz = sum_vsz + $5
    sum_rss = sum_rss + $6
}END{
    print("MEM TOTAL \n" "VSZ_SUM:" sum_vsz/1024 "M," "RSS_SUM:" sum_rss/1024 "M")}'

相关内容

Buy me a coffee~
HeZephyr 支付宝支付宝
HeZephyr 微信微信
更新于 2024-11-04 
CC BY-NC-SA 4.0
阅读原始文档查看源码编辑此页报告问题
Shell
返回 | 主页
0%