Querying a Disorganized Database
Hi there, I have a disgustingly organized database to extract data from. I have some experience in SQL, but not quite what I would require in order to get data in the exact form that I require for an automated extraction to work. I have built a barebones query, but it functions under an assumption that is not always true. I think I need help building a loop, but since I have never seen anything quite this bad, I think I need another opinion before I go crazy trying to build something. Any help would be appreciated.
The database is built up in a series of tables where each table represents an entity in an oil field (a well). Each of these tables has a timestamp (point type char(30) for reasons beyond my comprehension), and three variables to pull regarding the entity (one shown in the example). The trick, and the most frustrating part of this, is that each of the three variables is set up across 40 columns in the table, each with different data through time. That is, there is a variable for pressure, and it has 40 separate data points in a single row. The output I require has a timestamp for each data point, and has each data point in a single column, as such, the data has to be reorganized and 39 of the 40 data points require a synthesized timestamp (see attached).
My current query functions based on a 15 second space between all data points and therefore does not display all data properly. It also lives on the wrong side of a VPN so I had to screenshot it, I apologize. It sets up to retrieve an external table name (for the well name) and a timestamp for the where statement, and then builds a table out of a bunch of union statements. I imagine that this is sub-optimal, but as mentioned, I am inexperienced with SQL.
I guess what I really need help with is, is this a ridiculous way of pulling data/is there a better way, and if not really, how do I apply the interpolation of data? I have set up the math for interpolation but have no idea how to make the logic occur in a structured manner.
Advice and instruction is appreciated.
Example.xlsx
Interoplation.PNG
Query.PNG
The database is built up in a series of tables where each table represents an entity in an oil field (a well). Each of these tables has a timestamp (point type char(30) for reasons beyond my comprehension), and three variables to pull regarding the entity (one shown in the example). The trick, and the most frustrating part of this, is that each of the three variables is set up across 40 columns in the table, each with different data through time. That is, there is a variable for pressure, and it has 40 separate data points in a single row. The output I require has a timestamp for each data point, and has each data point in a single column, as such, the data has to be reorganized and 39 of the 40 data points require a synthesized timestamp (see attached).
My current query functions based on a 15 second space between all data points and therefore does not display all data properly. It also lives on the wrong side of a VPN so I had to screenshot it, I apologize. It sets up to retrieve an external table name (for the well name) and a timestamp for the where statement, and then builds a table out of a bunch of union statements. I imagine that this is sub-optimal, but as mentioned, I am inexperienced with SQL.
I guess what I really need help with is, is this a ridiculous way of pulling data/is there a better way, and if not really, how do I apply the interpolation of data? I have set up the math for interpolation but have no idea how to make the logic occur in a structured manner.
Advice and instruction is appreciated.
Example.xlsx
Interoplation.PNG
Query.PNG
ASKER
Hi dqmq, thank you for the response. I guess what I really need is a way to interpolate dates and times for all of the parameters that I union based on interpolation between the row's timestamp and the next row's timestamp.
Thanks!
Thanks!
what control over the database do you have?
ie could you introduce triggers on the individual tables to maintain 1 common table with data for all the wells, and possible sort out your problem with the 40 columns as well?
or could you reorganise the database to just have the one well table and implement the existing solution as a set of views for each individual well , possibly with an instead of trigger to handle the updating?
ie could you introduce triggers on the individual tables to maintain 1 common table with data for all the wells, and possible sort out your problem with the 40 columns as well?
or could you reorganise the database to just have the one well table and implement the existing solution as a set of views for each individual well , possibly with an instead of trigger to handle the updating?
ASKER
While a fantastic idea, Lowfatspread, I do not have sufficient control of the database to do something like that, the database is in a control network. The most crucial piece of this is synthesizing a timestamp to be associated with each data point in the row, and being able to view the timestamps and data points together. Thanks for the suggestion!
So you are saying, the time interval between row 1 and row 2 should be divided evenly into 120 time intervals, one for each column?
ASKER
The time interval should be divided by 40, as there are 40 columns per parameter, each division of 40 will be applied to the three variables - does that make sense?
Thank you again.
Thank you again.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Done. After organizing the data, the database had even worse quality information than I had originally suspected. Regardless, the query is set up using a similar form to what's above. Thank you so much for taking the time to assist me with that. If you find yourself in Saskatchewan, Canada, I owe you drink.
To minimize dynamic SQL, I would consider creating a view for each table that does all the unions and returns the desired structure.
As for the timeinterval, can you describe what you need?