Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Querying a Disorganized Database

Posted on 2011-09-19
8
Medium Priority
?
493 Views
Last Modified: 2012-05-12
Hi there, I have a disgustingly organized database to extract data from.  I have some experience in SQL, but not quite what I would require in order to get data in the exact form that I require for an automated extraction to work.  I have built a barebones query, but it functions under an assumption that is not always true.  I think I need help building a loop, but since I have never seen anything quite this bad, I think I need another opinion before I go crazy trying to build something.  Any help would be appreciated.

The database is built up in a series of tables where each table represents an entity in an oil field (a well).  Each of these tables has a timestamp (point type char(30) for reasons beyond my comprehension), and three variables to pull regarding the entity (one shown in the example).  The trick, and the most frustrating part of this, is that each of the three variables is set up across 40 columns in the table, each with different data through time.  That is, there is a variable for pressure, and it has 40 separate data points in a single row.  The output I require has a timestamp for each data point, and has each data point in a single column, as such, the data has to be reorganized and 39 of the 40 data points require a synthesized timestamp (see attached).

My current query functions based on a 15 second space between all data points and therefore does not display all data properly.  It also lives on the wrong side of a VPN so I had to screenshot it, I apologize.  It sets up to retrieve an external table name (for the well name) and a timestamp for the where statement, and then builds a table out of a bunch of union statements.  I imagine that this is sub-optimal, but as mentioned, I am inexperienced with SQL.

I guess what I really need help with is, is this a ridiculous way of pulling data/is there a better way, and if not really, how do I apply the interpolation of data?  I have set up the math for interpolation but have no idea how to make the logic occur in a structured manner.

Advice and instruction is appreciated.

Example.xlsx
Interoplation.PNG
Query.PNG
0
Comment
Question by:MeraGroup
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
8 Comments
 
LVL 42

Expert Comment

by:dqmq
ID: 36562128
The union structure looks OK to me.

To minimize dynamic SQL, I would consider creating a view for each table that does all the unions and returns the desired structure.

As for the timeinterval, can you describe what you need?
0
 

Author Comment

by:MeraGroup
ID: 36562651
Hi dqmq, thank you for the response.  I guess what I really need is a way to interpolate dates and times for all of the parameters that I union based on interpolation between the row's timestamp and the next row's timestamp.

Thanks!
0
 
LVL 50

Expert Comment

by:Lowfatspread
ID: 36564617
what control over the database do you have?

ie could you introduce triggers on the individual tables to maintain 1 common table with data for all the wells, and possible sort out your problem with the 40 columns as well?

or could you reorganise the database to just have the one well table and implement the existing solution as a set of views for each individual well , possibly with an instead of trigger to handle the updating?
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 

Author Comment

by:MeraGroup
ID: 36567248
While a fantastic idea, Lowfatspread, I do not have sufficient control of the database to do something like that, the database is in a control network.  The most crucial piece of this is synthesizing a timestamp to be associated with each data point in the row, and being able to view the timestamps and data points together.  Thanks for the suggestion!
0
 
LVL 42

Expert Comment

by:dqmq
ID: 36568344
So you are saying, the time interval between row 1 and row 2 should be divided evenly into 120 time intervals, one for each column?
0
 

Author Comment

by:MeraGroup
ID: 36568394
The time interval should be divided by 40, as there are 40 columns per parameter, each division of 40 will be applied to the three variables - does that make sense?

Thank you again.
0
 
LVL 42

Accepted Solution

by:
dqmq earned 2000 total points
ID: 36568838
--this view returns the duration of all but the last row
Create View Well1View1
as
select t1.*, t2.timestamp - t1.timestamp as duration
from Well1Table T1, Well1Table T2
where
t2.timestamp = (select min(t3.timestamp) from Well1Table1 t3 where t3.timestamp > T1.timestamp)


--this view synthesizes the timestamp
Create View Well1View2
as
Select (timestamp + (duration/40*(1-1))) as timestamp, presA1, presB1, presC1 from Well1View1
union all
Select (timestamp + (duration/40*(2-1))) as timestamp, presA2, presB2, presC2 from Well1View1
...
Select (timestamp + (duration/40*(40-1))) as timestamp, presA40, presB40, presC40 from Well1View1
...


Declare @SQL1 varchar(4000)
SET @SQL1 = Select * from ' + @WellName + 'View2'
insert into Temptable
   exec (@SQL)

 
 
0
 

Author Closing Comment

by:MeraGroup
ID: 36571952
Done.  After organizing the data, the database had even worse quality information than I had originally suspected.  Regardless, the query is set up using a similar form to what's above.  Thank you so much for taking the time to assist me with that.  If you find yourself in Saskatchewan, Canada, I owe you drink.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Recently we ran in to an issue while running some SQL jobs where we were trying to process the cubes.  We got an error saying failure stating 'NT SERVICE\SQLSERVERAGENT does not have access to Analysis Services. So this is a way to automate that wit…
Using examples as well as descriptions, and references to Books Online, show the different Recovery Models available in SQL Server and explain, as well as show how full, differential and transaction log backups are performed
Via a live example, show how to backup a database, simulate a failure backup the tail of the database transaction log and perform the restore.

660 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question