Solved

Querying a Disorganized Database

Posted on 2011-09-19
8
456 Views
Last Modified: 2012-05-12
Hi there, I have a disgustingly organized database to extract data from.  I have some experience in SQL, but not quite what I would require in order to get data in the exact form that I require for an automated extraction to work.  I have built a barebones query, but it functions under an assumption that is not always true.  I think I need help building a loop, but since I have never seen anything quite this bad, I think I need another opinion before I go crazy trying to build something.  Any help would be appreciated.

The database is built up in a series of tables where each table represents an entity in an oil field (a well).  Each of these tables has a timestamp (point type char(30) for reasons beyond my comprehension), and three variables to pull regarding the entity (one shown in the example).  The trick, and the most frustrating part of this, is that each of the three variables is set up across 40 columns in the table, each with different data through time.  That is, there is a variable for pressure, and it has 40 separate data points in a single row.  The output I require has a timestamp for each data point, and has each data point in a single column, as such, the data has to be reorganized and 39 of the 40 data points require a synthesized timestamp (see attached).

My current query functions based on a 15 second space between all data points and therefore does not display all data properly.  It also lives on the wrong side of a VPN so I had to screenshot it, I apologize.  It sets up to retrieve an external table name (for the well name) and a timestamp for the where statement, and then builds a table out of a bunch of union statements.  I imagine that this is sub-optimal, but as mentioned, I am inexperienced with SQL.

I guess what I really need help with is, is this a ridiculous way of pulling data/is there a better way, and if not really, how do I apply the interpolation of data?  I have set up the math for interpolation but have no idea how to make the logic occur in a structured manner.

Advice and instruction is appreciated.

Example.xlsx
Interoplation.PNG
Query.PNG
0
Comment
Question by:MeraGroup
  • 4
  • 3
8 Comments
 
LVL 42

Expert Comment

by:dqmq
ID: 36562128
The union structure looks OK to me.

To minimize dynamic SQL, I would consider creating a view for each table that does all the unions and returns the desired structure.

As for the timeinterval, can you describe what you need?
0
 

Author Comment

by:MeraGroup
ID: 36562651
Hi dqmq, thank you for the response.  I guess what I really need is a way to interpolate dates and times for all of the parameters that I union based on interpolation between the row's timestamp and the next row's timestamp.

Thanks!
0
 
LVL 50

Expert Comment

by:Lowfatspread
ID: 36564617
what control over the database do you have?

ie could you introduce triggers on the individual tables to maintain 1 common table with data for all the wells, and possible sort out your problem with the 40 columns as well?

or could you reorganise the database to just have the one well table and implement the existing solution as a set of views for each individual well , possibly with an instead of trigger to handle the updating?
0
Forrester Webinar: xMatters Delivers 261% ROI

Guest speaker Dean Davison, Forrester Principal Consultant, explains how a Fortune 500 communication company using xMatters found these results: Achieved a 261% ROI, Experienced $753,280 in net present value benefits over 3 years and Reduced MTTR by 91% for tier 1 incidents.

 

Author Comment

by:MeraGroup
ID: 36567248
While a fantastic idea, Lowfatspread, I do not have sufficient control of the database to do something like that, the database is in a control network.  The most crucial piece of this is synthesizing a timestamp to be associated with each data point in the row, and being able to view the timestamps and data points together.  Thanks for the suggestion!
0
 
LVL 42

Expert Comment

by:dqmq
ID: 36568344
So you are saying, the time interval between row 1 and row 2 should be divided evenly into 120 time intervals, one for each column?
0
 

Author Comment

by:MeraGroup
ID: 36568394
The time interval should be divided by 40, as there are 40 columns per parameter, each division of 40 will be applied to the three variables - does that make sense?

Thank you again.
0
 
LVL 42

Accepted Solution

by:
dqmq earned 500 total points
ID: 36568838
--this view returns the duration of all but the last row
Create View Well1View1
as
select t1.*, t2.timestamp - t1.timestamp as duration
from Well1Table T1, Well1Table T2
where
t2.timestamp = (select min(t3.timestamp) from Well1Table1 t3 where t3.timestamp > T1.timestamp)


--this view synthesizes the timestamp
Create View Well1View2
as
Select (timestamp + (duration/40*(1-1))) as timestamp, presA1, presB1, presC1 from Well1View1
union all
Select (timestamp + (duration/40*(2-1))) as timestamp, presA2, presB2, presC2 from Well1View1
...
Select (timestamp + (duration/40*(40-1))) as timestamp, presA40, presB40, presC40 from Well1View1
...


Declare @SQL1 varchar(4000)
SET @SQL1 = Select * from ' + @WellName + 'View2'
insert into Temptable
   exec (@SQL)

 
 
0
 

Author Closing Comment

by:MeraGroup
ID: 36571952
Done.  After organizing the data, the database had even worse quality information than I had originally suspected.  Regardless, the query is set up using a similar form to what's above.  Thank you so much for taking the time to assist me with that.  If you find yourself in Saskatchewan, Canada, I owe you drink.
0

Featured Post

Online Training Solution

Drastically shorten your training time with WalkMe's advanced online training solution that Guides your trainees to action. Forget about retraining and skyrocket knowledge retention rates.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
how to make geography query faster?  SQL 7 46
sql server cross db update 2 23
Problem Backingh Up Transaction Log on Old SQL000 Database 3 27
Sql server Error message 3 16
JSON is being used more and more, besides XML, and you surely wanted to parse the data out into SQL instead of doing it in some Javascript. The below function in SQL Server can do the job for you, returning a quick table with the parsed data.
For both online and offline retail, the cross-channel business is the most recent pattern in the B2C trade space.
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Viewers will learn how the fundamental information of how to create a table.

730 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question