Git Product home page Git Product logo

pgreport's Introduction

Introduction
============
pgreport is a tool for generating statistic analysis of postgresql.conf parameters and kernel parameters for efficient use of your environment. It examines the environment and produces the report showing recommended values for each parameter. Current it works only for standalone machine.It generates output as PDF File. 


Download
--------
You can download latest version of pgreport at
https://github.com/Rushikesh005/pgreport/archive/master.zip .


Installation
------------
You need to have python and postgresql-server installed on your system.
Extract the downloaded zip to a convenient location.
Its a python script.There is no need to build/compile pgreport.


Usage
-----
  python pgreport.py [$PGDATA/postgresql.conf]
  

  for example:
  python pgreport.py /etc/postgresql/9.1/main/postgresql.conf
  
  Output:
  The execution of the program will take some time
  The output i.e; Report.pdf will be generated in the same directory of your program.


You can make pgreport.py executable,

$ chmod +x pgreport.py
$ ./pgreport $PGDATA/postgresql.conf

Todo
====

Testing.
Work on additional parameters.

Bugs
====

Program is under testing phase.If you find any bugs please mail the below concerned person.

Contact
=======
 * Initial commits are done in the git repostiory at http://github.com/Rushikesh005/pgreport

If you have any hints, changes or improvements, please contact:

 * Rushikesh Patil [email protected]
 * Prashant Pandey [email protected]

pgreport's People

Contributors

rushikesh005 avatar

Watchers

James Cloos avatar  avatar

pgreport's Issues

Why we need to display memory information which represents the current state of memory

  1. Why we are displaying the current state of memory to user ?
  2. How that is useful for our goal ?
  3. Is there any value of postgresql.conf parameter that we are fixing on the basis of this ?

as per report:

Memory Info All values in KB

Free RAM : 860132
Total OS Cached : 4923024
Cached Mem : 767088
Total RAM+SWAP : 4988892
Free SWAP : 4128760
Total SWAP : 4128760
Total RAM : 2055040

Buffers : 27176

Several problems in output

I tested the module and found many problems in it also listed improvement points . Following is the snapshot for the same

You may feel that those are unnecessary changes, but it can confuse user who is not very well aware of PostgreSQL
For reference you can look at some standard module that will help a lot
reference:
https://github.com/vmware/pg_rewind
https://github.com/michaelpq/pg_arman

$ python pgreport.py /home/samrat/pg_rewind/data-patched-sync/postgresql.conf
PostgresConf file location :/home/samrat/pg_rewind/data-patched-sync/postgresql.conf
Disk /dev/mapper/vg_puppetserver-lv_root doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_swap doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_root doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_swap doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_root doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_swap doesn't contain a valid partition table
cat: /sys/block/mapper/queue/scheduler: No such file or directory
cat: /sys/block/mapper/queue/scheduler: No such file or directory

Finding os configuration.....

writing to os.conf....

Exctrating Memory Information information....

Exctracting kernel information....

Exctrating OS information.....

Finding RAID information....

finding disk and partitions information....

Finding file system information.....

finding disk read speed....

find scheduler information .....

Exctracting OS instruction length

Exctracting current value os shared_buffer from conf file

exctracting Total RAM Value

Calculating Recommended value of Shared buffer

['shb:Upper limit 8GB', 'shb:No need to change SHMMAX value']

Exctracting recommeded shared_buffer value from shared_memory

Calculating Recommended value for work memory

writting to recommend.txt

getting current value os work memory

getting default value for FSYNC

getting default value for synchronousCommit

getting defalut value for wal_buffers

getting total RAM

getting default value for full_pageWrites

getting shared buffer

getting wal_buffer

getting total_RAM

getting shared buffer

getting total_RAM

Traceback (most recent call last):
File "file_loc.py", line 249, in
do_cal()
File "file_loc.py", line 84, in do_cal
free_disk_part = find_free_space()
File "file_loc.py", line 43, in find_free_space
disk_free[lm[0]] = lm[4]
IndexError: list index out of range

Using font Courier size = 10
Input file => report
Writing pdf file report.pdf ...
Wrote file report.pdf


Improvement points:

PostgresConf file location :/home/samrat/pg_rewind/data-patched-sync/postgresql.conf
Its "postgresql.conf" not "PostgresConf" sorry for nitpicking comments but it matters a lot

Disk /dev/mapper/vg_puppetserver-lv_root doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_swap doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_root doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_swap doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_root doesn't contain a valid partition table
Disk /dev/mapper/vg_puppetserver-lv_swap doesn't contain a valid partition table
Why these errors are coming ?

cat: /sys/block/mapper/queue/scheduler: No such file or directory
cat: /sys/block/mapper/queue/scheduler: No such file or directory
Are there no scheduler related files on system ?

Finding os configuration.....

No need to append # before each message just keep it simple (as per standards)

Finding file system information.....

finding disk read speed....

find scheduler information .....

all the above messages are in "ing" form and this one differs form other
please maintain consistency in the system

Exctracting OS instruction length

Exctracting current value os shared_buffer from conf file

You mean "Exctracting current value of shared_buffer from conf file"

exctracting Total RAM Value

"Extracting total RAM value"

Calculating Recommended value of Shared buffer

No need to use short forms if you write shb then no one in PostgreSQL world will understand the meaning of it. just keep it as simple

['shb:Upper limit 8GB', 'shb:No need to change SHMMAX value']
why you are giving o/p recommendation here ? just keep it in final report

Exctracting recommeded shared_buffer value from shared_memory

Spell check

getting total RAM

what do you mean by total RAM ?

Traceback (most recent call last):
File "file_loc.py", line 249, in
Why you are giving traceback here ?
If you want to do it then you should use another file for that

IndexError: list index out of range
?

Using font Courier size = 10
Is there a need to add font information in o/p meassages?

Input file => report
No need of this

Wrote file report.pdf
No need of this

feel free to ask me if you have any problem :)

Improve contents in README

Please improve the contents in readme as this is the first impression anyone gets when he see your project this should be clean and neat with only required contents

Take a look at reference README : https://github.com/vmware/pg_rewind/blob/master/README

Improvement points:

  1. No need to use headers and big fonts keep it simple
  2. write brief description for only necessary things
    for eaxample:

The content of introduction which you have written can be written in following manner:

pgreport is a tool for generating statistic analysis of postgresql.conf parameters and kernel parameters for efficient use of your environment. It examines the environment and produces the report showing recommended values for each parameter. Current it works only for standalone machine
this is for just example purpose and lot more needs to be added into this

  1. Always write the correct names , they have a long history behind them, that will reflect you quality of work and respect towards them have a look at link :

http://www.computerworld.com/s/article/9006258/Ever_wonder_how_Red_Hat_got_its_name_?source=rss_news10

for example:
pgreport not Pgreport
PostgreSQL not Postgresql
postgresql.conf not Postgresql.cong
Red Hat not Redhat

  1. Add download information, if anyone gets your repository though git protocol then he should find download link in readme file, this will also make you git repository popular :p
  2. Installation
    Add a line that you need to have python installed o your system
    can we make it as executable ?
    for usage can be example:
    $ chmod +x pgreport.py
    $ ./pgreport $PGDATA/postgresql.conf
  3. Add a brief information about how it operates

etc...

Please follow standard guideline while writting the code

I observed that you have written code like class assignments no meaningful names to variables, functions , no sufficient comments in the code

have a look at the small piece of code in file_loc.py file:

import subprocess
import re
import operator
import sys
ljust_val = 40 -------- > what is ljust_val ?
kd = 3 -------- > what does kd mean, same apples for all the below code ?
PostgresPath = sys.argv[1]
def get_disk_list():
d = {}
f = open("os.conf","r")
file = f.readlines()
f.close()
flag = 0
disk_list = []
for line in file:
if "Disks and partitions" in line:
flag = 1
elif flag == 1:
flag =2
elif flag == 2 and "----" not in line and line != "\n":
if "File System information" in line:
break
if "Disk" in line:
disk_list.append(line.split(' ')[0])
return disk_list

Improvement points:
Python is very powerful dynamic programming language when you write a code in it, it should have meaning

while naming functions and variables use following rule:
lowercase with words separated by underscores as necessary to improve readability.

Have a look at this link from pythons official site:
http://legacy.python.org/dev/peps/pep-0008/

Junk text at start of the pdf report

The generated report seems to have some junk text before report starts which i have copied and pasted below
%PDF-1.4
1 0 obj
<<
/Creator (PyText2Pdf Version 1.1.1 By Anand B Pillai )
/CreationDate (D:20140325133901)
/Producer (PyText2Pdf Version 1.1.1(\251 Free Software Foundation, 2004))
/Title (report)

endobj
2 0 obj
<<
/Type /Catalog
/Pages 3 0 R

endobj
4 0 obj
<<
/BaseFont /Courier /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font >>

endobj
5 0 obj
<<
/Font << /F1 4 0 R >>
/ProcSet [ /PDF /Text ]

endobj
6 0 obj
<<
/Type /Page
/Parent 3 0 R
/Resources 5 0 R
/Contents 7 0 R

endobj
7 0 obj
<<
/Length 8 0 R

stream
BT
/F1 10 Tf
1 0 0 1 50 752 Tm
12 TL

Unnecessary characters in pdf report

1.>>fsync
remove >> across the report
2.
Comments = ["add 'noatime' options under \etc
fstab for disk :/dev/sda9", "add 'noatime' options under \etc\fstab for disk
:/dev/sda3", "add 'noatime' options under \etc\fstab for disk :/dev/sda1", "
add 'noatime' options under \etc\fstab for disk :/dev/sda2", "add 'noatime' o
ptions under \etc\fstab for disk :/dev/sda6", "add 'noatime' options under
etc\fstab for disk :/dev/sda8", "add 'noatime' options under \etc\fstab for
disk :/dev/sda5"]
This does not look right at all

Comments = ['$syssctl vm.overcommit_memory=2']
Again it seems some substitution has not taken place. Similar issue across the report

['9762.00 MB/sec'] Please remove '. Similar issue across the report

  1. Current_value Recommended_Value. Consistency is important. V should be small in Recommended_value

Problems in content

  1. RAID levels supported by kernel : []
    This looks incorrect
  2. In Disks and partitiions please put headers to the table
    Partition Size in MB? OS ?
    /dev/sda7 : 32768000 : Linux : NONE

Remove junk and unnecessary files from module

I can see a lot of unnessary files in the module please remove them
for example:

  1. test.txt ? Do we need this ?
  2. p1.py~ ? lokks like you have accidentally added swap file
  3. report ? why we need this if this is an intermediate file which gets generated when module runs then you should delete this at the end .i.e when module finishes
  4. report.pdf ? you github commit should not contain this file
  5. recommend.txt ? again intermediate file
    6 postgresql.conf ? why this files is in github commit ?
  6. postgre_info.pyc ? what is this file ? and why do we need this in commit ?
  7. os.conf.pdf ? again intermediate file
  8. os.conf ? again intermediate file
  9. report.pdf ? we dont need this in github commit ? do we ?

The point I am trying to make is keep your code repository as clean as possible I am sure you can find more files like this if you look carefully at your code
please clean it

Give meaningful names to source code files

You should give meaningful name to the source code files so that if anyone wants to look at some specific code he should be able to guess it just reading file name

for example:

  1. p1.py
  2. p3.py

what these file do ?
Please think carefully and rename your files in module

Preferred output format (report.pdf)

The details in the report looks very unstructured and confusing
According to me the preferred way to do this in tabular format where reader can get all the things related to particular parameter in one place and single row is as follows:

-------------------------------------- OS Configurations --------------------------------------
Parameter Value (KB)
Free RAM 860132
.
.
-------------------------------------- kernel resources --------------------------------------
Parameter Value
SEMMNS 32000
.
.

-------------------------------------- postgresql.conf parameters --------------------------------------
Parameter Current value Upper limit Recommended value

  1. Shared memory
    shared_buffer 128 MB 8GB 802.75 MB
    .
  2. WRITE AHEAD LOG
    fsync ON - ON
    .
    .

Problems in report file

Postgesql.conf file path =
spelling mistake also don't make P capital
should be postgresql.conf file path =

operating system name : Linux
Its not operating system name your are displaying its a kernel name
to get OS name run following command
$ uname -o
GNU/Linux

if you already used uname command then cross check it once again:
$ uname --help

-s, --kernel-name print the kernel name
-n, --nodename print the network node hostname
-r, --kernel-release print the kernel release
-v, --kernel-version print the kernel version
-m, --machine print the machine hardware name
-p, --processor print the processor type or "unknown"
-i, --hardware-platform print the hardware platform or "unknown"
-o, --operating-system print the operating system

Memory Info
Use full forms Memory Information:
And which system command you are using to get this ?

SEMMNS
Give full form also
for reference look : http://www.postgresql.org/docs/9.3/static/kernel-resources.html

CPU info
also add number of cores

in Disks and partitions info
Partition : not : on : NONE
What are the column names ?

Scheduler information
On my environment scheduler information is blank why is that ?

PostgreSQL Parameters
Do not use short forms please use names of parameters as it is otherwise no one will understand

temp_file_limit
temp:Only Supported by PostgreSQL > 9.2
This field is blank even I am using PostgreSQL-9.4

I will work on this and let you know further comments

SyntaxError: invalid syntax

Downloaded the zip and unzipped code. Executed
python /var/lib/pgsql/9.2/data/postgresql.conf

File "/var/lib/pgsql/9.2/data/postgresql.conf", line 113
shared_buffers = 32MB # min 128kB
SyntaxError: invalid syntax

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.