Skip to content

Commit 7e31a3c

Browse files
digoaldigoal
committed
10.0
1 parent d56b90e commit 7e31a3c

24 files changed

+2737
-0
lines changed

‎201703/20170312_01.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
## PostgreSQL 10.0 主动防御 - 可配置是否允许执行不带where条件的update\delete
2+
3+
### 作者
4+
digoal
5+
6+
### 日期
7+
2017-03-12
8+
9+
### 标签
10+
PostgreSQL , 10.0 , 主动防御 , 是否允许执行不带where条件的update\delete
11+
12+
----
13+
14+
## 背景
15+
你是否曾经被不带where 条件的SQL误伤过呢？
16+
17+
比如
18+
19+
update tbl set amount=amount-100 where id=?;
20+
21+
缺少where条件，就变成了
22+
23+
update tbl set amount=amount-100;
24+
25+
正常情况下，这样的SQL不应该在业务逻辑中出现。通常出现在SQL注入，又或者误操作中。
26+
27+
如果你真的不小心执行了，那么全表的数据都会被删除或者更新，最快的恢复手段是flash back query，PostgreSQL中，可以使用xlog，生成UNDO，比如将xlog_level设置为logical，同时表的match必须设置为记录FULL OLD VALUE。
28+
29+
那么就有方法从xlog中生成UNDO，flash back该表。
30+
31+
flashback query属于被动防御的话，数据库有没有主动防御措施呢？
32+
33+
## 主动防御
34+
PostgreSQL提供了一个机制，允许你设置参数
35+
36+
+bool allow_empty_deletes = true;
37+
+bool allow_empty_updates = true;
38+
39+
从而允许是否能执行不带where 条件的update或delete.
40+
41+
这个参数可以设置为全局、会话级、用户级、库级、或者事务级别。
42+
43+
设置后，你就能控制是否允许执行不带条件的update,delete了。
44+
45+
## 扩展
46+
其实不带where条件的update, delete还不够全面。比如where 1=1或者where true，都需要防范。
47+
48+
还有我们甚至可以设置百分比（比如百分之多少的记录被UPDATE,DELETE时，或者超过多少记录被DML后，回退整个事务）
49+
50+
通过PostgreSQL提供的钩子可以完成以上功能。
51+
52+
## 参考
53+
https://commitfest.postgresql.org/12/948/
54+
55+
https://www.postgresql.org/message-id/flat/[email protected]#[email protected]
56+
57+
https://www.postgresql.org/message-id/attachment/45216/training_wheels_001.patch
58+

‎201703/20170312_02.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
## PostgreSQL 10.0 新增数十个IO等待事件监控
2+
3+
### 作者
4+
digoal
5+
6+
### 日期
7+
2017-03-12
8+
9+
### 标签
10+
PostgreSQL , 10.0 , 等待事件 , IO
11+
12+
----
13+
14+
## 背景
15+
PostgreSQL 10.0新增了数十个IO等待事件，描述系统调用的等待。
16+
17+
```
18+
Hi All,
19+
20+
Attached is the patch, which extend the existing wait event infrastructure
21+
to
22+
implement the wait events for the disk I/O. Basically pg_stat_activity's
23+
wait
24+
event information to show data about disk I/O as well as IPC primitives.
25+
26+
Implementation details:
27+
28+
- Added PG_WAIT_IO to pgstat.h and a new enum WaitEventIO
29+
- Added a wait_event_info argument to FileRead, FileWrite, FilePrefetch,
30+
FileWriteback, FileSync, and FileTruncate. Set this wait event just before
31+
performing the file system operation and clear it just after.
32+
- Pass down an appropriate wait event from caller of any of those
33+
functions.
34+
- Also set and clear a wait event around standalone calls to read(),
35+
write(), fsync() in other parts of the system.
36+
- Added documentation for all newly added wait event.
37+
38+
Open issue:
39+
- Might missed few standalone calls to read(), write(), etc which need
40+
to pass the wait_event_info.
41+
42+
Thanks to my colleague Robert Haas for his help in design.
43+
44+
Please let me know your thought, and thanks for reading.
45+
46+
Thanks,
47+
Rushabh Lathia
48+
www.EnterpriseDB.com
49+
```
50+
51+
IO等待事件如下
52+
53+
```
54+
/* ----------
55+
+ * Wait Events - IO
56+
+ *
57+
+ * Use this category when a process is waiting for a IO.
58+
+ * ----------
59+
+ */
60+
+typedef enum
61+
+{
62+
+ WAIT_EVENT_READ_DATA_BLOCK,
63+
+ WAIT_EVENT_WRITE_DATA_BLOCK,
64+
+ WAIT_EVENT_SYNC_DATA_BLOCK,
65+
+ WAIT_EVENT_EXTEND_DATA_BLOCK,
66+
+ WAIT_EVENT_FLUSH_DATA_BLOCK,
67+
+ WAIT_EVENT_PREFETCH_DATA_BLOCK,
68+
+ WAIT_EVENT_WRITE_REWRITE_DATA_BLOCK,
69+
+ WAIT_EVENT_SYNC_REWRITE_DATA_BLOCK,
70+
+ WAIT_EVENT_TRUNCATE_RELATION_DATA_BLOCKS,
71+
+ WAIT_EVENT_SYNC_RELATION,
72+
+ WAIT_EVENT_SYNC_IMMED_RELATION,
73+
+ WAIT_EVENT_READ_BUFFILE,
74+
+ WAIT_EVENT_WRITE_BUFFILE,
75+
+ /* Wait event for XLOG */
76+
+ WAIT_EVENT_READ_XLOG,
77+
+ WAIT_EVENT_READ_COPY_XLOG,
78+
+ WAIT_EVENT_WRITE_XLOG,
79+
+ WAIT_EVENT_WRITE_INIT_XLOG_FILE,
80+
+ WAIT_EVENT_WRITE_COPY_XLOG_FILE,
81+
+ WAIT_EVENT_WRITE_BOOTSTRAP_XLOG,
82+
+ WAIT_EVENT_SYNC_INIT_XLOG_FILE,
83+
+ WAIT_EVENT_SYNC_COPY_XLOG_FILE,
84+
+ WAIT_EVENT_SYNC_BOOTSTRAP_XLOG,
85+
+ WAIT_EVENT_SYNC_ASSIGN_XLOG_SYNC_METHOD,
86+
+ /* Wait event for CONTROL_FILE */
87+
+ WAIT_EVENT_WRITE_CONTROL_FILE,
88+
+ WAIT_EVENT_WRITE_UPDATE_CONTROL_FILE,
89+
+ WAIT_EVENT_SYNC_WRITE_CONTROL_FILE,
90+
+ WAIT_EVENT_SYNC_UPDATE_CONTROL_FILE,
91+
+ WAIT_EVENT_READ_CONTROL_FILE,
92+
+ /* Wait event for REORDER BUFFER */
93+
+ WAIT_EVENT_READ_REORDER_BUFFER,
94+
+ WAIT_EVENT_WRITE_REORDER_BUFFER,
95+
+ /* Wait event for LOGICAL MAPPING */
96+
+ WAIT_EVENT_READ_APPLY_LOGICAL_MAPPING,
97+
+ WAIT_EVENT_WRITE_LOGICAL_MAPPING_REWRITE,
98+
+ WAIT_EVENT_SYNC_LOGICAL_MAPPING_REWRITE,
99+
+ WAIT_EVENT_SYNC_LOGICAL_MAPPING_REWRITE_HEAP,
100+
+ WAIT_EVENT_TRUNCATE_LOGICAL_MAPPING_REWRITE,
101+
+ /* Wait event for SNAPBUILD */
102+
+ WAIT_EVENT_WRITE_SNAPBUILD_SERIALIZE,
103+
+ WAIT_EVENT_READ_SNAPBUILD_RESTORE,
104+
+ WAIT_EVENT_SYNC_SNAPBUILD_SERIALIZE,
105+
+ /* Wait event for SNRU */
106+
+ WAIT_EVENT_READ_SLRU_PAGE,
107+
+ WAIT_EVENT_WRITE_SLRU_PAGE,
108+
+ WAIT_EVENT_SYNC_SLRU_FLUSH,
109+
+ WAIT_EVENT_SYNC_SLRU_WRITE_PAGE,
110+
+ /* Wait event for TIMELINE HISTORY */
111+
+ WAIT_EVENT_READ_TIMELINE_HISTORY_WALSENDER,
112+
+ WAIT_EVENT_READ_TIMELINE_HISTORY_WRITE,
113+
+ WAIT_EVENT_WRITE_TIMELINE_HISTORY,
114+
+ WAIT_EVENT_WRITE_TIMELINE_HISTORY_FILE,
115+
+ WAIT_EVENT_SYNC_TIMELINE_HISTORY_WRITE,
116+
+ WAIT_EVENT_SYNC_TIMELINE_HISTORY_FILE,
117+
+ /* Wait event for TWOPHASE FILE */
118+
+ WAIT_EVENT_READ_TWOPHASE_FILE,
119+
+ WAIT_EVENT_WRITE_RECREATE_TWOPHASE_FILE,
120+
+ WAIT_EVENT_SYNC_RECREATE_TWOPHASE_FILE,
121+
+ /* Wait event for SYSLOGGER */
122+
+ WAIT_EVENT_READ_SYSLOGGER_FILE,
123+
+ WAIT_EVENT_WRITE_SYSLOGGER_FILE,
124+
+ /* Wait event for REPLSLOT */
125+
+ WAIT_EVENT_READ_RESTORE_REPLSLOT,
126+
+ WAIT_EVENT_WRITE_REPLSLOT,
127+
+ WAIT_EVENT_SYNC_RESTORE_REPLSLOT,
128+
+ WAIT_EVENT_SYNC_SAVE_REPLSLOT,
129+
+ /* Wait event for copydir */
130+
+ WAIT_EVENT_READ_COPY_FILE,
131+
+ WAIT_EVENT_WRITE_COPY_FILE,
132+
+ /* Wait event RELMAP FILE */
133+
+ WAIT_EVENT_READ_LOAD_RELMAP_FILE,
134+
+ WAIT_EVENT_WRITE_RELMAP_FILE,
135+
+ WAIT_EVENT_SYNC_WRITE_RELMAP_FILE,
136+
+ /* Wait event for LOCK FILE */
137+
+ WAIT_EVENT_READ_CREATE_LOCK_FILE,
138+
+ WAIT_EVENT_READ_ADDTODATEDIR_LOCK_FILE,
139+
+ WAIT_EVENT_READ_RECHECKDATADIR_LOCK_FILE,
140+
+ WAIT_EVENT_WRITE_CREATE_LOCK_FILE,
141+
+ WAIT_EVENT_WRITE_ADDTODATEDIR_LOCK_FILE,
142+
+ WAIT_EVENT_SYNC_ADDTODATEDIR_LOCK_FILE,
143+
+ WAIT_EVENT_SYNC_CREATE_LOCK_FILE
144+
+} WaitEventIO;
145+
```
146+
147+
对于判断系统瓶颈，又多了一个有力信息。
148+
149+
## 参考
150+
https://commitfest.postgresql.org/13/975/
151+
152+
https://www.postgresql.org/message-id/flat/CAGPqQf0LsYHXREPAZqYGVkDqHSyjf=KsD=k0GTVPAuzyThh-VQ@mail.gmail.com#CAGPqQf0LsYHXREPAZqYGVkDqHSyjf=KsD=k0GTVPAuzyThh-VQ@mail.gmail.com

‎201703/20170312_03.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
## PostgreSQL 10.0 内置角色 - 监控、管理用户会话
2+
3+
### 作者
4+
digoal
5+
6+
### 日期
7+
2017-03-12
8+
9+
### 标签
10+
PostgreSQL , 10.0 , 内置角色 , 监控角色 , 管理用户会话角色 , pg_backend_pid , pg_monitor
11+
12+
----
13+
14+
## 背景
15+
PostgreSQL 10.0 开始植入了一些内置的角色，例如pg_backend_pid角色可用于cancel, terminate任何PID，除此之外没有其他超级用户的权限，这种用户可以给DBA或者给一些业务OWNER使用，便于它们在紧急情况下行使权力，杀掉一些会话。
16+
17+
同时为了便于DBA或者业务OWNER查看到数据库的健康状况，PostgreSQL也增加了一个监控角色pg_monitor，拥有监控角色的权限后，可以查看一些统计信息。
18+
19+
同时添加了一个角色pg_read_all_gucs，这个角色可以查看所有的GUC配置。
20+
21+
```
22+
Per the discussion at
23+
https://www.postgresql.org/message-id/CA%2BOCxoyYxO%2BJmzv2Micj4uAaQdAi6nq0w25BPQgLLxsrvTmREw%40mail.gmail.com,
24+
attached is a patch that implements the following:
25+
26+
- Adds a default role called pg_monitor
27+
- Gives members of the pg_monitor role full access to:
28+
pg_ls_logdir() and pg_ls_waldir()
29+
pg_stat_* views and functions
30+
pg_tablespace_size() and pg_database_size()
31+
Contrib modules:
32+
pg_buffercache,
33+
pg_freespacemap,
34+
pgrowlocks,
35+
pg_stat_statements,
36+
pgstattuple and
37+
pg_visibility (but NOT pg_truncate_visibility_map() )
38+
- Adds a default role called pg_read_all_gucs
39+
- Allows members of pg_read_all_gucs to, well, read all GUCs
40+
- Grants pg_read_all_gucs to pg_monitor
41+
42+
Note that updates to contrib modules followed the strategy recently
43+
used in changes to pgstattuple following discussion here, in which the
44+
installation SQL script is left at the prior version, and an update
45+
script is added and default version number bumped to match that of the
46+
upgrade script.
47+
48+
Patch includes doc updates, and is dependent on my pg_ls_logdir() and
49+
pg_ls_waldir() patch
50+
(https://www.postgresql.org/message-id/CA+OCxow-X=D2fWdKy+HP+vQ1LtrgbsYQ=CshzZBqyFT5jOYrFw@mail.gmail.com).
51+
52+
--
53+
Dave Page
54+
Blog: http://pgsnake.blogspot.com
55+
Twitter: @pgsnake
56+
57+
EnterpriseDB UK: http://www.enterprisedb.com
58+
The Enterprise PostgreSQL Company
59+
```
60+
61+
未来PostgreSQL还会对植入更多的内置角色，让数据库的权限分组管理更加便捷。逐步形成像Oracle这样内部有许多角色可选的状况。
62+
63+
## 参考
64+
https://www.postgresql.org/message-id/attachment/49958/pg_monitor.diff
65+
66+
https://www.postgresql.org/message-id/flat/CA+OCxoyRdsc1xyLfF9s698gUGyPXBs4CvJ+0Gwo8U65NmYJ7pw@mail.gmail.com#CA+OCxoyRdsc1xyLfF9s698gUGyPXBs4CvJ+0Gwo8U65NmYJ7pw@mail.gmail.com
67+
68+
https://commitfest.postgresql.org/13/1031/
69+

‎201703/20170312_04.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
## PostgreSQL 10.0 QUERY进度显示
2+
3+
### 作者
4+
digoal
5+
6+
### 日期
7+
2017-03-12
8+
9+
### 标签
10+
PostgreSQL , 10.0 , query进度
11+
12+
----
13+
14+
## 背景
15+
query进度的显示，有一个插件pg_query_state支持。
16+
17+
PostgreSQL 10.0将在内核层面增加一个这样的支持。
18+
19+
首先会在analye命令上尝试，但是它会设计为通用的视图，支持其他命令，诸如CREATE INDEX, VAUUM, CLUSTER等。
20+
21+
```
22+
Hello Hackers,
23+
24+
Following is a proposal for reporting the progress of ANALYZE command:
25+
26+
It seems that the following could be the phases of ANALYZE processing:
27+
1. Collecting sample rows
28+
2. Collecting inherited sample rows
29+
3. Computing heap stats
30+
4. Computing index stats
31+
5. Cleaning up indexes
32+
33+
The first phase is easy if there is no inheritance but in case of
34+
inheritance we need to sample the blocks from multiple heaps.
35+
Here the progress is counted against total number of blocks processed.
36+
37+
The view provides the information of analyze command progress details as
38+
follows
39+
postgres=# \d pg_stat_progress_analyze
40+
View "pg_catalog.pg_stat_progress_analyze"
41+
Column | Type | Collation | Nullable | Default
42+
-------------------+---------+-----------+----------+---------
43+
pid | integer | | |
44+
datid | oid | | |
45+
datname | name | | |
46+
relid | oid | | |
47+
phase | text | | |
48+
heap_blks_total | bigint | | |
49+
heap_blks_scanned | bigint | | |
50+
total_sample_rows | bigint | | |
51+
52+
I feel this view information may be useful in checking the progress of
53+
long running ANALYZE command.
54+
55+
56+
The attached patch reports the different phases of analyze command.
57+
Added this patch to CF 2017-03.
58+
59+
Opinions?
60+
61+
Note: Collecting inherited sample rows phase is not reported yet in the
62+
patch.
63+
64+
Regards,
65+
Vinayak Pokale
66+
NTT Open Source Software Center
67+
```
68+
69+
## 参考
70+
[《官人要杯咖啡吗? - PostgreSQL实时监测PLAN tree的执行进度 - pg_query_state》](../201612/20161208_01.md)
71+
72+
https://commitfest.postgresql.org/13/1053/
73+
74+
https://www.postgresql.org/message-id/flat/[email protected]#[email protected]
75+

0 commit comments

Comments
 (0)