Solved

Querying a Disorganized Database

Posted on 2011-09-19
8
395 Views
Last Modified: 2012-05-12
Hi there, I have a disgustingly organized database to extract data from.  I have some experience in SQL, but not quite what I would require in order to get data in the exact form that I require for an automated extraction to work.  I have built a barebones query, but it functions under an assumption that is not always true.  I think I need help building a loop, but since I have never seen anything quite this bad, I think I need another opinion before I go crazy trying to build something.  Any help would be appreciated.

The database is built up in a series of tables where each table represents an entity in an oil field (a well).  Each of these tables has a timestamp (point type char(30) for reasons beyond my comprehension), and three variables to pull regarding the entity (one shown in the example).  The trick, and the most frustrating part of this, is that each of the three variables is set up across 40 columns in the table, each with different data through time.  That is, there is a variable for pressure, and it has 40 separate data points in a single row.  The output I require has a timestamp for each data point, and has each data point in a single column, as such, the data has to be reorganized and 39 of the 40 data points require a synthesized timestamp (see attached).

My current query functions based on a 15 second space between all data points and therefore does not display all data properly.  It also lives on the wrong side of a VPN so I had to screenshot it, I apologize.  It sets up to retrieve an external table name (for the well name) and a timestamp for the where statement, and then builds a table out of a bunch of union statements.  I imagine that this is sub-optimal, but as mentioned, I am inexperienced with SQL.

I guess what I really need help with is, is this a ridiculous way of pulling data/is there a better way, and if not really, how do I apply the interpolation of data?  I have set up the math for interpolation but have no idea how to make the logic occur in a structured manner.

Advice and instruction is appreciated.

Example.xlsx
Interoplation.PNG
Query.PNG
0
Comment
Question by:MeraGroup
  • 4
  • 3
8 Comments
 
LVL 42

Expert Comment

by:dqmq
ID: 36562128
The union structure looks OK to me.

To minimize dynamic SQL, I would consider creating a view for each table that does all the unions and returns the desired structure.

As for the timeinterval, can you describe what you need?
0
 

Author Comment

by:MeraGroup
ID: 36562651
Hi dqmq, thank you for the response.  I guess what I really need is a way to interpolate dates and times for all of the parameters that I union based on interpolation between the row's timestamp and the next row's timestamp.

Thanks!
0
 
LVL 50

Expert Comment

by:Lowfatspread
ID: 36564617
what control over the database do you have?

ie could you introduce triggers on the individual tables to maintain 1 common table with data for all the wells, and possible sort out your problem with the 40 columns as well?

or could you reorganise the database to just have the one well table and implement the existing solution as a set of views for each individual well , possibly with an instead of trigger to handle the updating?
0
 

Author Comment

by:MeraGroup
ID: 36567248
While a fantastic idea, Lowfatspread, I do not have sufficient control of the database to do something like that, the database is in a control network.  The most crucial piece of this is synthesizing a timestamp to be associated with each data point in the row, and being able to view the timestamps and data points together.  Thanks for the suggestion!
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 42

Expert Comment

by:dqmq
ID: 36568344
So you are saying, the time interval between row 1 and row 2 should be divided evenly into 120 time intervals, one for each column?
0
 

Author Comment

by:MeraGroup
ID: 36568394
The time interval should be divided by 40, as there are 40 columns per parameter, each division of 40 will be applied to the three variables - does that make sense?

Thank you again.
0
 
LVL 42

Accepted Solution

by:
dqmq earned 500 total points
ID: 36568838
--this view returns the duration of all but the last row
Create View Well1View1
as
select t1.*, t2.timestamp - t1.timestamp as duration
from Well1Table T1, Well1Table T2
where
t2.timestamp = (select min(t3.timestamp) from Well1Table1 t3 where t3.timestamp > T1.timestamp)


--this view synthesizes the timestamp
Create View Well1View2
as
Select (timestamp + (duration/40*(1-1))) as timestamp, presA1, presB1, presC1 from Well1View1
union all
Select (timestamp + (duration/40*(2-1))) as timestamp, presA2, presB2, presC2 from Well1View1
...
Select (timestamp + (duration/40*(40-1))) as timestamp, presA40, presB40, presC40 from Well1View1
...


Declare @SQL1 varchar(4000)
SET @SQL1 = Select * from ' + @WellName + 'View2'
insert into Temptable
   exec (@SQL)

 
 
0
 

Author Closing Comment

by:MeraGroup
ID: 36571952
Done.  After organizing the data, the database had even worse quality information than I had originally suspected.  Regardless, the query is set up using a similar form to what's above.  Thank you so much for taking the time to assist me with that.  If you find yourself in Saskatchewan, Canada, I owe you drink.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

I wrote this interesting script that really help me find jobs or procedures when working in a huge environment. I could I have written it as a Procedure but then I would have to have it on each machine or have a link to a server-related search that …
Ever wondered why sometimes your SQL Server is slow or unresponsive with connections spiking up but by the time you go in, all is well? The following article will show you how to install and configure a SQL job that will send you email alerts includ…
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.
Using examples as well as descriptions, and references to Books Online, show the documentation available for datatypes, explain the available data types and show how data can be passed into and out of variables.

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

25 Experts available now in Live!

Get 1:1 Help Now