Flat File Foes - Relational Database Management Systems

Flat Files are Dead - Long live RDMS!

This has been the battle cry of SQL, Oracle, SAP and other database admins - and almost every consultant - for many years. Since the first publication of the idea of using a Relational Database came about in the late 70s, people have touted the benefits of them: less storage space, more efficient (less data moved internally) and centralized look at the data. However, they never ever talk about the downsides of RDMS - to get ANY data in or out you MUST write an application. All the data is never in a single spot - so you cannot just copy 1 table (file) and have everything - you must copy EVERYTHING in order to backup ANYTHING (you just tripled your storage needs if lucky, you never have just 1 backup!) 

Unlike flat files where if a physical file corruption occurs the whole file (data) is gone (reasons to have backups!), in RDMS if a table is corrupted only the items in that table are gone - but then realizing that it IS corrupted is a lot harder and a person has to look at EVERY table in them to actually find out if one is bad - and then figure out how to roll back EVERY table to the same point in time to restore to.

For those who do not know how a RDBMS works here is a base example with some history.

A flat file is like the old Hollerith cards - all the data fits on 80 columns where holes can be punched. If you use only the first 20 columns the other 60 take up space and is of no use - but you still must try and read all 80 columns each time. When this got moved onto 2400 foot reels of tape the same was true - even if you did not use a field it still had to be read.

Now imagine if you had every address of every user in the USA - all 310 million people - in the above flat file you would have 310 million data fields with the name of a city in it. For Portland Oregon that means 537081 duplicate entries of Portland in the database - lots of space used up. A RDB saves space by only have 1 entry in a table with the word Portland, and in the records of each person a pointer - which takes up way less space - points to the entry in that table where the word Portland is stored. So when a record is read the pointer is read which tells the system which table where  the actual data is at and then read the value and shows it for a report, screen - whatever. Course since this is just a word - the same entry could be used for those people living in Portland Maine - but their pointer for the state would point to their state name in a table when retrieved.

A person looking at the RDBM has a hard time to reading all this - because often a table of pointers points to other tables of pointers which is within a program used to decipher the cipher in the fields to get to the real value - thus you need a program to get into and out of it on even how it is set up. There is no way to ever see a single record without a full fledged application written to even DO that. Compare this to a single Excel Spreadsheet where 1 line contains everything about that one record.

To me the flat file is a lot more functional - you can always export it to a CSV file, all the records are there logically and readable by a human, and it is self contained when the number of distinct records is below 100,000. A file cabinet of data. With a RDBM there is a whole lot more to it than what I described above even with the benefit - the knowledge - and number of people required to run it - goes up a WHOLE lot once you jump into a true RDBMS.

Teaching the Basics

Watching the flooding within Oregon and Washington - and watching the people going through high water and then having to be rescued (a few did not get rescued) - shows that what is taught in schools is NOT what is needed to survive - but what is politically correct to be taught.

Why? Because the Government (all levels) has been gradually been teaching everyone that you NEVER do anything yourself - you ask for help and let the appointed Government people to come solve your problems for you - they are basically saying you are too stupid to know how to do anything so leave it to "us" to help you.

Now government is created by PEOPLE so that chores that they WANT to do is more efficiently tasked to someone else who can do it full time. You do not see a farm raising or planting EVERY type of food - they pick 1 or two and become good at it - and other farmers plant, or raise other food since it is just more efficient doing it that way. Government was create for the same purpose. Everyone CAN be a soldier - but doing it full time means they cannot do anything else - so the creation of standing armies  since the 1200s - special purpose people doing what people want done but they cannot do full time themselves - is more efficient than having everyone in the Army and no one planting food, raising meat, fishing etc.

Practical survival skills is NOT taught even in the "wild west" of Oregon and Washington. Driving a car through high water is fine - IF you know how high "high" is (to paraphrase President Clinton). If the water would come up more than 4 inches over the tailpipe of the car - your car will stall. This is due to the water pressure and the design of the modern engines REQUIRING only so much back pressure in the exhaust system (due to environmental regs on emissions caused them to recycle fumes, use backpressure to reuse past expelled fuel etc) - above that level then the gasses cannot leave the cylinder and thus it will become more and more rich and then eventually fail to ignite.

Also, once the water level gets above the tailpipe in most every design that means the water is now pushing against the sides of a vehicle. Below that threshold there really is no problem, above that then the car will also start to FLOAT! Now a floating car with side pressure (like when crossing a stream) means that the car will be pushed off the roadway into DEEP water - and now you are often under water. Most roadbeds are 2 to 15 feet above the rest of the land.

Is any of this practical stuff taught in schools - no. The Government just states 'Do not drive in high water' and NEVER explains why - they know best after all - and then people still drive through - because like a lot of things that Government does since they can -- they never teach the reasons WHY a rule was created out of an unrelated law that people never had a direct say in due to the way most every government in the world works.

 

 

Risk Management =/ Productivity

I was trying to find the Australian / New Zealand standard document ANZ 4360 - Risk Management Standards - and I found out everyone talks about it - but no one actually publishes the standards at all - can't be found.

 

I did find http://www.pc.gov.au/ - which is the Australian Government Productivity Commission - and found out that Risk Management is also no where to be found.

 

Which means if you do Risk Management you will not be productive.