We help IT Professionals succeed at work.
Research Question

Excel function to retrieve data from www.clinicaltrials.gov

64 Views
Last Modified: 2020-10-01
Hello Experts!

I'm looking for a way to connect a list of clinical trials in Excel to related trial data at ClinicalTrials.Gov .

The attached Excel file has a list of clinical trial IDs in the first column. I would like to retrieve selected data from www.clinicaltrials.gov using the available API.

API Documention available at https://clinicaltrials.gov/api/gui/ref/api_urls

I am using Office 365 ProPlus with  Excel for Office 365 MSO 32 bit.

Would Excel's Get & Transform tools work?  Or could we use VBA to write a custom function?

I don't anticipate having more than 500 trials and 20 fields.   

EE-Excel clinical trial API 20200923.xlsx 

Thank you! 

Comment
Watch Question

CERTIFIED EXPERT

Commented:
Would Excel's Get & Transform tools work? 
that will probably work.

for example, see the attached.

EE-Excel-clinical-trial-API-2020092_b.xlsx

Author

Commented:
Thanks Ryan. I tried the Get & Transform web view and can see how you imported the field list.

I'm now trying to write the URL that will fetch the 250+ trial records and selected fields.

Would this approach make sense: Write a vba script that creates a URL for each NCTId and selected fields, then automate the Get & Transform step to import each row (in this test file, approx 250+ trial records). Or any other approach? 

Study Fields(demo)ClinicalTrials.gov/api/query/study_fields?expr=heart+attack&fields=NCTId,Condition,BriefTitleReturns values from selected API fields for a large set of study records. Select the fields returned using the fields parameter (shown in the Query Parameters table). For a complete list of fields, use the Study Fields List info URL.Returns 20 study records by default.Returns up to 1,000 study records when minimum rank and maximum rank parameters are set.Returns in JSON or CSV format when the format parameter is set (fmt=JSON or fmt=CSV, respectively).

CERTIFIED EXPERT

Commented:
Would this approach make sense: Write a vba script that creates a URL for each NCTId and selected fields, then automate the Get & Transform step to import each row (in this test file, approx 250+ trial records). Or any other approach? 

I never done such complex manipulation before, but if it's within Excel, I guess it's possibly feasible...

Scott FellDeveloper
CERTIFIED EXPERT
Fellow
Most Valuable Expert 2013

Commented:
Are you familiar with other languages?  It would seem this may be easier to do outside of Excel where you can program logic and storing arrays etc.  In VBA you should be able to do this using xmlhttppost. I am not well versed in vba but have done this in vbs and at least I am pretty sure that is similar.  

Ryan has a similar example https://www.experts-exchange.com/questions/29190386/extract-field-in-xml-using-vba.html#a43134101..  With VBA or VBS you have to manually extract out XML and json can be troublesome.  Just about every other language has automation for this part. 
Bill PrewTest your restores, not your backups...
CERTIFIED EXPERT
Expert of the Year 2019
Top Expert 2016

Commented:
Not sure I will have time until next week, but I may have some VBA code that did something similar using JSON data stream.  I'll take a look when I can...


»bp
Bill PrewTest your restores, not your backups...
CERTIFIED EXPERT
Expert of the Year 2019
Top Expert 2016

Commented:
Okay, got some done on this today, and it's not fool proof it seemed to work for the data you had.

I used an open source JSON parsing module I had used previously, but did a little manipulation of the returned JSON stream before I parsed it to make things a little simpler.

Also no error handling built in at this point, or any fancy formatting of the sheet after it is populated.

I left the data in the sheet I got when I ran it, took a minute or two to populate.

You can run the macro named "GetTrialData" and it will erase the data on the first sheet and then pull it again.  Be patient while it works along, not a lot of visual indicators that it's working other than the cursor.

Hope this might be useful as a starting point, or maybe even good enough for you needs.

EE29195591.xlsm


»bp

Author

Commented:
Guys thanks so much for this. It's been over six years since I used experts-exchange, and its great to be back.  I've already got my subscription paid for with this solution.

Bill, can I ask you to help make a few tweaks to the solution?
1. Format dates as dates.
2. Populate the column containing the NCT_IDs by pointing to a column in another sheet (may or may not be in the same workbook), and deduplicating the NCT_IDs. This way I have option to integrate the solution with my master file.
3. If the script encounters an invalid NCT_ID not recognized by the clinicaltrial.gov API, then either highlight or other otherwise indicate which records were not retrieved. [not critical] 
Bill PrewTest your restores, not your backups...
CERTIFIED EXPERT
Expert of the Year 2019
Top Expert 2016

Commented:
Bill, can I ask you to help make a few tweaks to the solution?

I'll digest those and come back with potential changes, or questions if needed.


»bp
Test your restores, not your backups...
CERTIFIED EXPERT
Expert of the Year 2019
Top Expert 2016
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Gentlemen, many thanks for your interest in this question.  I struggled with it for so long until I remembered Experts-Exchange. I think the solution can help a lot of other people that also need to access clinical trial records. The API has over 300 fields, compared to only 20 fields that can be downloaded via the website. Again thank you! 
Bill PrewTest your restores, not your backups...
CERTIFIED EXPERT
Expert of the Year 2019
Top Expert 2016

Commented:
Thanks Bill, the solution works great and I really appreciate you adding in the formatting and error correction.  And you are absolutely right, it's better for me to generate the input list separately.

Thanks for the feedback, glad that was useful.

If you have any issues you can always message me on the site...


»bp

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.