Rho site logo

Rho Knows Clinical Research Services

Brook White

Recent Posts

12 Resume Tips That Can Help You Get a Clinical Research Job

Posted by Brook White on Wed, Sep 13, 2017 @ 11:36 AM

resume tips for clinical research jobsI’ve been working at Rho for 10 years and at CROs for more than 15 years, and in that time, I’ve reviewed a lot of resumes for job seekers in many different positions. Here are some resume and CV tips to help you stand out with the recruiters, hiring managers, and interview teams that make the difference between getting an interview or a rejection letter for the clinical research job you really want.

Note: The tips I’m sharing here are for job seekers in the US. International standards can differ.

Keywords Matter

In most cases, the first look at your resume won’t be a thorough one. During that first pass, a recruiter is probably looking for a handful of keywords that they associate with the position. What terms are you using to search for jobs? Those same terms should show up prominently on your resume.

Does that mean you can’t get a job if you haven’t held that job before? No. You just need to make sure that it is clear how your experience is applicable. For example, if you are seeking a job as a CRA but haven’t had CRA as your title, you can still make sure that words like monitoring and site visit show up in the accomplishments and descriptions of your roles. For competitive positions, most candidates are screened out during this step. Make sure you’re not one of them.

Proofread, Proofread, Proofread

I’m always surprised by the number of resumes I see that have basic spelling, grammar and formatting errors. Generally, I wouldn’t consider hiring someone with these sorts of errors. This may sound picky, but clinical research requires close attention to detail for nearly every position at every level.

You should carefully proofread your resume. Then, ask someone with strong writing and editing skills to do the same. Don’t have access to someone that’s up to the job? Check out Grammarly. It is a free online tool that will eliminate most of these errors. Make sure you list the correct company and job title for which you are applyi

best candidate for a clinical research job

ng. Listing either one incorrectly shows a lack of attention to detail and tells the recruiter you aren’t committed to their company.

Tailor Your Resume to the Position

Start by carefully reading the posted job description.  What specific skills and experience does the job require?  Make sure you highlight these skills on your resume and that it is obvious how your experience aligns with the required experience for the job.  What are the primary job duties and responsibilities?  Call-out how you have accomplished similar tasks in your previous work.  Finally, review the company’s website to see what values they highlight.  Quality? Teamwork? Fast-paced environment?  Think about how you can demonstrate the attributes they are looking for in the materials you are submitting.  This may seem daunting, but submitting 10 tailored resumes will produce better results than submitting 100 generic ones.  List your applicable skills at the beginning of your resume, not the end.  You want to capture the attention of the resume reviewer quickly.  

Demonstrate Knowledge of the Industry

Whether you are a clinical research veteran with 20 years of experience or are seeking an entry level position, your resume should reflect awareness of what is happening in the industry. If you are new to the industry or returning to the industry, there are a number of great free news sources that can help you with this. A few of my favorites are FierceCRO and FierceBiotech, Clinical Leader, and Applied Clinical Trials.

Consider how to work in key trends. Searching for a clinical operations role? Highlight your experience with risk-based monitoring. Looking for a job in clinical project management? Mention your experience working with patient advocacy groups to improve patient recruitment. Obviously, the depth of knowledge and awareness expected will differ based on your role and experience level, but these are the kinds of things that can be differentiators in a competitive field. One thing to note – make sure you can speak to every item on your resume if asked about your experience – no fabrications. If you can’t articulate that particular skill or ability during an interview, don’t list it on your resume or CV.


This is not a creative or design-focused industry.  Your resume does not need to be a work of art, but basic proper formatting is expected.  Use an easy to read font and font size with a light background color and a dark font color.  Have reasonable margins.  Use bullets.  Limit your resume to 2-3 pages maximum.  Focus on making it easy to read and easy to find desired information.  There are plenty of good free templates out there, so consider using one of those if you aren’t sure what good formatting looks like.

Cover Letter—Yes or No?

happy_business_colleauges.jpgWhether or not to include a cover letter depends both on your resume and the position for which you are applying. In some cases, it will be obvious how your skills and experience are transferable to the posted position. For example, you currently are a clinical data manager with experience in EDC system x and skills y and z applying for a job as a clinical data manager with experience in EDC system x and skills y and z. In this case, a cover letter isn’t necessary, although it would provide an opportunity for you to explain why you are interested in that company or that position.

In some cases, there isn’t a straight line between your work experience and the position you’re applying for. Or, maybe there is something on your resume that you would like to explain like a gap in your work history. Or, maybe the position is in another location and you want to voice your willingness to relocate. A cover letter can help you with any of these situations. If you do include a cover letter, make sure it is concise, well-written, and offers something more than what would be obvious from reading your resume. And don’t forget to ask a friend to proofread your cover letter! A great resume will be overlooked by a poorly written and grammatically incorrect cover letter.

Accomplishments Not Duties

For each position, you should include a brief summary of the responsibilities followed by a couple of core accomplishments.  It should not be a bulleted list of the twenty duties in the job description.  Finally, accomplishments should be specific and should include metrics where possible.  For example, a clinical project manager might list an accomplishment like “For a global phase 3 study, completed enrollment 6 weeks early and delivered topline results 3 days after database lock.”  This resume will get a lot further than one with a list that says managed global phase 3 studies, oversaw data management and statistical deliverables, and managed timelines.

Technical Skills

Recruiters and hiring managers are often looking for specific technology skills and even experience with specific software systems.  Ideally, these would be listed in the job posting, but that isn’t always the case.  Include both industry system types like CTMS, EDC, IRT, and statistical programming languages as well as specific system names like Medidata Rave and SAS.  Also include industry agnostic technologies that may be applicable to your role like MS Project, HTML, or Java.  If you have experience in clinical research, you should also list the therapeutic areas where you have experience. Also make sure to list any certifications like the Regulatory Affairs Certification (RAC), Project Management Professional (PMP), or Certified Clinical Research Associate (CCRA).

How Long Should My Resume Be?

The answer to how long your resume should be depends on how much work experience you have and the type of job you are seeking.  For recent graduates and early career candidates, about a page is a good rule of thumb.  For experienced candidates in most roles, 2-3 pages is an appropriate length.  For scientific roles where publications are expected to be included, CV length is highly variable and should be driven by career length, number of professional positions, and number of publications; however, you still want your biggest selling points on the first couple of pages.

Things You Don’t Need to Include on Your Resume

There are also a number of things you shouldn’t include on your resume:

  • Personal or demographic information like age, race, gender, religious preference, social security number, or marital status.
  • Photos.
  • Salary history or salary requirements.
  • “References available on request”—this is assumed, takes up space, and can make your resume seem dated.
  • Objective—I’ve never seen an objective that has made a difference in my decision for the positive.  However, an objective that isn’t well written or doesn’t align with what I’m looking for has caused me to rule candidates out.

Advice for Recent Graduates

Many recent college graduates struggle with what to include on their resumes besides their education if they don’t have professional work experience.  That is totally fine—when we are hiring entry level candidates we don’t expect them to have professional work experience.  Some beneficial things you can include are:

  • Volunteer experience
  • Non-professional work experience from restaurant jobs to dog walking to mowing lawns in the summer
  • Study abroad programs
  • Internships
  • Extracurricular activities, especially leadership positions
  • Class projects that are relevant to the position

Each of these provides valuable information about you.

Interested in working at Rho? Learn more about working at Rho!

View All Job Openings

10-Step Commercial Clinical Protocol Authoring Guide

Posted by Brook White on Thu, Aug 31, 2017 @ 01:57 PM

Lauren Neighbours, PhD, RACLauren Neighbours, PhD, RAC, is a Research Scientist at Rho. She leads cross-functional project teams for clinical operations and regulatory submission programs. Lauren partners with early-, mid-, and late-stage companies to develop and refine strategic development plans, design and execute clinical studies, lead regulatory submissions, and provide support for regulatory authority meetings and other consulting needs. She has over ten years of scientific writing and editing experience and has served as a lead author on clinical and regulatory documents for product development programs across a range of therapeutic areas.

Devin Rosenthal, PhDDevin Rosenthal, PhD, RAC, works with companies at all stages of development to help them shape their product development programs. He has experience across the full drug development spectrum through his roles in small biotech, big pharma, and at Rho, with particular focus on oncology, CNS, gastrointestinal, and respiratory indications. In addition to pharmaceutical development, Devin is also involved in strategic alliance and business development activities at Rho.

Genna Kingon, PhDGenna Kingon, PhD, RAC, is a Research Scientist at Rho involved in regulatory strategy and submission management from pre-IND to post-approval.  She also serves as a lead regulatory author on multiple programs for submissions to FDA and to various international regulatory authorities.  In particular, Genna focuses on rare disease programs and expedited approval pathways. 

begin with the end in mindA protocol is the most important document in a clinical study as it is the foundation for subsequent operational, regulatory, and marketing objectives for the development program. 

 Developing a protocol is an extensive undertaking that requires a cross-functional team and consideration of the position and role of the study in the full product development program.  Before the protocol authoring process even begins, a variety of activities and decisions are necessary to establish a strategy for success.  The following steps provide concepts and considerations that are essential in formulating the details that will become the protocol synopsis and ultimately the clinical study protocol. 


1.    Begin with the end in mind

our program team should first prepare an Integrated Product Development Plan (IPDP). This plan, which is largely based upon the desired final Target Product Profile (TPP) and product labeling, maps out all activities through marketing application submission and clearly outlines the purpose, position, and necessity of each study in the product development program. Without these documents, you run the risk of completing a study that fails to advance your product’s development or is markedly less valuable to development than it otherwise could be.

Among other things, the IPDP should contain the clinically meaningful endpoint(s) for your studies that will be acceptable to regulators and support the desired marketing claims for the product. Additionally, the IPDP should include an assessment of the actual and potential competitive products likely to be on the market at or near the time of product launch. This information will be essential for optimal study design and conduct, and will therefore improve the chances of ultimate product success. Cross-functional input and buy-in from all key internal and external stakeholders for each study, as well as on the full development plan, is a necessity.

2.    Design the study

clinical study designBefore you start thinking about the protocol study procedures and visit schedule, you need to understand your overall goals for the study, and how the data that are collected will not only support your product development strategy but ultimately move your program forward. For studies in the early phase of development, consider first outlining the study objectives, as well as the endpoints that specifically address those objectives in a measurable and meaningful way. The design of the study should then flow from those objectives and endpoints, making sure the technical and logistical aspects of the protocol maintain a focus on the end goals.

For all studies, consider developing the statistical analysis plan (SAP) before drafting the protocol. During SAP development, the study objectives and endpoints are comprehensively considered and designed, along with the specific analytical methods needed to optimally interpret the data. Choose a sample size that has sufficient statistical power to reliably detect outcomes and differences of interest and that meaningfully contributes to accumulation of an adequate safety database for your product, but is also as practical as possible to enable successful study completion. Then, explore study design options with the protocol objective(s), SAP, and the TPP in mind.

In designing your study, take the following into account:

  1. Map out how key study measures will be assessed, with what frequency, and in what kind of study population.  Properly defining the study population is essential, particularly to ensure that the inclusion and exclusion criteria appropriately select for the eventual target population, as well as for optimal assessment of safety and efficacy in that population. 
  2. Be sure that existing animal toxicology data are adequate to support any proposed duration of dosing, dose levels, and specific subject eligibility criteria.
  3. Be mindful of manufacturing capacity and schedules for study drug to ensure that your study is feasible given the cost of goods and timelines for manufacturing.  You may have to adjust the dosing duration, dosage, number of dose levels, or your study timeline to accommodate manufacturing limitations.  Even after your drug is manufactured, you may want or need to develop specialized packaging such as blister packaging or cold-chain logistics to help ensure study success.
  4. Remember that the more complex the study design (e.g., number of arms, number of objectives and endpoints, number or complexity of assessments), the greater the chances for errors, omissions, data quality issues, and unexpected complications during study execution; and, therefore, the greater the chance for study failure.  Study design should be laser focused on what is required to produce only the information necessary for product labeling and/or to progress the compound to the next stage of development.  For this reason, it is also important to avoid the common temptation of adding “nice-to-have” but inessential study components during the course of protocol development. 

3.  Define technical details

Establish or obtain an International Conference on Harmonisation (ICH)-compliant protocol template and develop and maintain a style guide and/or list of writing conventions to ensure consistency and clarity within and between study documents.  Establish the appropriate reviewing processes, and identify cross-functional reviewers (editorial, regulatory, clinical, statistical, data management, medical, product safety, senior management, etc.).  Record all key decisions and their rationale throughout the development and writing process.  Failure to do so may result in frequently having to revisit issues, causing unnecessary delays and changes in the protocol or development plan.

4.  Draft the synopsis

Generate the study schedule of events, and draft the synopsis.  The synopsis should be no more than 10 pages total.  Obtain feedback from cross functional subject matter experts, senior leadership from the sponsor/contract research organization (CRO), and potential clinical investigators and study site staff.  Revise and finalize the synopsis:  this is the foundation for the clinical study protocol.  


5.  Define operational details

Consider essential operational logistics such as laboratory test results required to enroll and/or randomize subjects (e.g., will this require local labs as opposed to a central lab?), total blood volume drawn, equipment and space necessary for subject evaluation, availability of specialist(s) for nonstandard assessments, storage and shipping requirements for clinical specimens and investigational product, and scheduling limitations/conflicts for study visits.  Consult both sponsor and CRO operations staff and study sites as necessary to determine the feasibility of the proposed operational plan.  

6.  Minimize the potential for amendments

simplify the protocol where possibleConsider what qualifies for inclusion in the protocol; detailed information that is not directly relevant to study conduct is usually better suited for operations manuals, which can be more easily updated throughout the study.  Avoid redundancy within the protocol; state everything once.  Use the synopsis as a tool to establish the foundation of the protocol.  At the completion of protocol development, the synopsis should be reviewed to ensure it accurately reflects the content of the final protocol (if it is intended to be appended to the protocol or used separately as an internal reference tool).  Continuously revising the synopsis while the protocol is being written is unnecessary and discouraged as this invariably leads to errors in one document or the other, as well as in the resulting study.  Whether or not a synopsis is included in the final protocol itself is often a matter of sponsor preference. 

7.  Draft the protocol

Prepare the protocol draft by expanding on the detail in the synopsis regarding the investigational plan, study schedule, analysis plan, safety monitoring, and the other outlined provisions.  Much of the protocol should be derived from template language, which generally does not change from protocol to protocol, but rather, only changes periodically following revised regulatory requirements or other administrative preferences.  Obtain additional review from cross-functional subject matter experts (which may include patient advocacy groups, as applicable), the sponsor and/or CRO personnel, and select study investigators.

Download: Protocol Template

Concurrent and/or Post-Protocol

8.  Draft the informed consent form (ICF)

Using an established and compliant informed consent form (ICF) template, draft the ICF with finalized protocol information at the appropriate reading level for the intended study subjects, which is rarely greater than about an eighth-grade level.  Obtain cross-functional subject matter expert and sponsor/CRO/site feedback.  Revise and finalize the form, which may require site- and institutional review board (IRB)-specific information or even site/IRB specific template language.  While the consent must include all required regulatory elements, strive to make the consent form as short as possible and without repetition.  A consent form that is overly complicated or too long to be easily read and understood fails in its purpose.

9.  Design case report forms (CRFs)

case report forms (CRFs)Capture data efficiently (fewer queries) with appropriate and reasonable CRF pages.  Be considerate of open-ended text boxes versus check boxes:  while an open-ended text box is preferable for describing unexpected, non-categorical events, check boxes are better for categorical items (e.g., ethnicity) to reduce the need for queries and to facilitate downstream data analysis.  The CRF should undergo interdisciplinary review by representatives from key functional areas (i.e., data management, biostatistics, programming, clinical operations, regulatory, safety, medical affairs) prior to finalization. 

10.  Design and compile operations manuals

The clinical sites will reference operations manuals for additional study information that is not specified in detail in the protocol (e.g., pharmacokinetic sampling procedures, shipping information, tissue collection procedures, investigational product preparation/dispensation, study contact information, etc.).  Use the manuals as an easily accessible reference for site study staff and a repository for information that has the potential to change during the study (e.g., shipping addresses if personnel/vendors are likely to change).

Download: Protocol Template


Could Your Drug Development Program Benefit from an NDA/BLA/PMA Gap Analysis?

Posted by Brook White on Wed, Aug 23, 2017 @ 09:37 AM

David Shoemaker, PhD--Senior Vice President R&DDavid Shoemaker, PhD, Senior Vice President R&D, has extensive experience in the preparation and filing of all types of regulatory submissions including primary responsibility for four BLAs and three NDAs.  He has managed or contributed to more than two dozen NDAs, BLAs, and MAAs and has moderated dozens of regulatory authority meetings.  

Jack Modell, MD--Vice President and Senior Medical OfficerJack Modell, MD, Vice President and Senior Medical Officer, is a board-certified psychiatrist with 30 years of experience in clinical research, teaching, and patient care including 10 years of experience in clinical drug development (phases 2 through 4) and successful NDA filings. Dr. Modell is a key opinion leader nationally known for leading the first successful development of preventative pharmacotherapy for the depressive episodes of seasonal affective disorder.

scott-burian.jpgScott Burian, PhD, Senior Research Scientist, has contributed to the development of a diverse range of small molecule, biologic, and nanoparticle-based products.  He has participated in numerous FDA interactions, including pre-IND meetings, Type A meetings, and Advisory Committee meetings. He is fully-versed in eCTD format and has authored a variety of CMC submissions, including numerous pre-IND meeting packages, INDs, NDAs, and IMPDs.

bridging the gap between clinical data and NDA submissionHere at Rho, we’ve helped many companies with their marketing application submissions. In fact, in the past six years, we’ve been a key service provider on 14 submissions, provided biostatistics support for 30 submissions, and prepared over 20 Integrated Summary of Safety (ISS) and Integrated Summary of Efficacy (ISE) SAPs. Over the course of working on these submissions, one common hurdle we see is that Sponsor companies often enter this stage without a strong understanding of what data they have and how that maps to a viable approval pathway.

Whether you plan to file a new drug application (NDA), a biologics license application (BLA), or a premarket approval application (PMA) with the FDA or a marketing authorization application (MAA) with the European Medicines Agency, you’ll need an in depth understanding of how the data you have from your clinical studies, nonclinical studies, and Chemistry, Manufacturing and Controls (CMC) / Quality development map to the requirements of the application. These requirements can be specific to the therapeutic area or regulatory authority, and are continually changing as science advances.

Discovering you don’t have all the data you need as you begin preparing your marketing application can lead to costly time delays. What can be done? We recommend undertaking a gap analysis following proof-of-concept in Phase II. This timing allows you to design your adequate and well-controlled studies to attain all necessary clinical data. Performing the gap analysis at this stage of development will also provide enough time to conduct additional nonclinical studies or CMC development that may be needed to support the application.
You need a cross-functional team of medical, regulatory, clinical, statistical, CMC, and toxicology experts with experience getting a product to market, ideally in the therapeutic area of interest. Many small to mid-size companies don’t have all of this expertise in-house, so the team will need to bring in outside support in the form of consultants or a contract research organization (CRO) that has this expertise.

A gap analysis starts with a detailed look at the existing data and regulatory communications. What is the format of the data? Anything you plan to submit will need to be in CDISC format, so if you need data from legacy studies, the data must be converted to CDISC format if the study was initiated after December 2016. Next, look at the label claims you plan to make. Do you have (or have a plan to collect) all the data needed to support those claims? This can be difficult to determine.

mapping clinical dataOnce you’ve determined the data you have and the data you’ll need, create a map that clearly identifies the deficiencies in your database. You may find that there are very few gaps and the data you’ve collected and will collect in your pivotal studies will adequately support your marketing application. You may also realize that you don’t need all of the data from your legacy studies, which can save you some time and money in CDISC conversion costs. Conversely, you may identify significant gaps in your database that require additional studies. That is still a good outcome because by performing the gap analysis you have clearly identified what needs to be completed and you will have sufficient time to gather the additional data. This could mean just completing your Phase 3 studies, or performing additional clinical (e.g. food effect studies) or nonclinical studies, or CMC development work, thus ensuring that upon completion of the Phase III studies, you will have a clear path to your marketing application submission.

So, is the additional time and expense of conducting a gap analysis worth it? Rho believes that the answer is most definitely, yes. However, we typically recommend waiting until proof-of concept has been demonstrated to conduct this analysis. At that point, you should have convinced yourself that you have a viable product and have a general idea of its characteristics and potential value to patients. An experienced team of medical, nonclinical, CMC, regulatory, and statistical experts can conduct a gap analysis relatively quickly and for a relatively limited cost. When compared to a significant delay between the end of Phase 3 and submission or an unsuccessful marketing application submission, it is almost certainly worth it.

Download: Marketing Application Planning Tool

How much does a clinical trial cost? Understanding the 8 major factors driving CRO bids

Posted by Brook White on Tue, Aug 15, 2017 @ 09:32 AM

how much a clinical trial costsThis is a question we frequently get in some form from Sponsors in early phases of development that are trying to figure out what it will cost to hire a CRO to conduct their clinical trials and associated regulatory activities. While the high degree in variability between trials would make it difficult to provide a single answer, we can explain what factors can drive costs up or down.

If you already have well defined information about your clinical trial, use this RFP specifications tool to request a more accurate and detailed estimate from Rho or another CRO.

Clinical Trial Cost Drivers

While there are many variables that can impact a clinical trial budget, these are some of the most common ones:

  1. Therapeutic Area: What therapeutic area or indication is the focus of the program?  Some will be more expensive than others.  For example, oncology studies are frequently highly complex and, in general, will be more costly than say an ophthalmology study with the same number of sites and subjects.
  2. Study Duration: How long will a study last?  Short studies like a seasonal allergy study where enrollment happens quickly and the treatment period is relatively short would typically be a less expensive study than one that may require many months of treatment and follow-up.
  3. Number of Patients: How many patients are needed?  The more patients needed, the higher the costs will be.  Later phase studies where an accurate assessment of efficacy is needed tend to be more expensive than earlier phase studies where establishing safety is the primary objective.
  4. Number and Location of Sites: How many sites do you need and where will they be located?  The more sites you need, the higher the cost will be.  Geography is also important.  In general, costs will increase for every additional country you add.  Also keep in mind that conducting studies in some countries will be more expensive than in others.
  5. Number of Labs and Procedures: The number and complexity of labs and procedures will impact the costs.  This includes direct costs from the CRO to monitor and manage the data as well as increased pass through costs from sites and central labs.
  6. Patient Population: Are you studying healthy or sick patient populations?  What is the prevalence of the indication?  This will impact your costs both in terms of safety considerations and speed of enrollment.  How much competition is there?  If you are working with a small patient population and multiple competing studies, this will likely drive up costs for patient recruitment as well as potentially increasing the enrollment duration.
  7. Clinical Monitoring Plan: How frequently do you anticipate needing monitoring visits?  The total number of monitoring visits is a key cost driver and will be determined by the frequency of visits desired as well as the duration of enrollment and treatment.
  8. Safety Profile: What are the safety concerns with the treatment and patient population?  The number and seriousness of adverse effects (AEs/SAEs) will impact costs, as will the need for a data safety monitoring board (DSMB) and any interim analyses needed to support it.

Looking for an answer tailored to your clinical study? Please contact us for cost estimates tailored to your program.  We're also happy to set up an expert consultation if you need assistance figuring out how these cost drivers apply to your program.

Kara Roberts, Director Proposals and Contracts, contributed to this post.

5 Tips for Creating an RFP

Heat Maps for Database Lock

Posted by Brook White on Tue, Aug 08, 2017 @ 11:50 AM

Kristen Mason, Senior BiostatisticianKristen Mason, MS, is a Senior Biostatistician at Rho. She has over 4 years of experience providing statistical support for studies conducted under the Immune Tolerance Network (ITN) and Clinical Trials in Organ Transplantation (CTOT). She has a particular interest in data visualization, especially creating visualizations within SAS using the graph template language (GTL). 

Heather Kopetskie, Senior BiostatisticianHeather Kopetskie, MS, is a Senior Biostatistician at Rho. She has over 10 years of experience in statistical planning, analysis, and reporting for Phase 1, 2 and 3 clinical trials and observational studies. Her research experience includes over 8 years focusing on solid organ and cell transplantation through work on the Immune Tolerance Network (ITN)and Clinical Trials in Organ Transplantation (CTOT) project.  In addition, Heather serves as Rho’s biostatistics operational service leader, an internal expert sharing biostatistical industry trends, best practices, processes and training.

Preparing a database for lock can be a burdensome process. It requires coordinated effort from an entire clinical study team, including, but not limited to, the clinical data manager, study monitor, biostatistician, clinical project manager, principal investigator, and medical monitor. The team must work together to ensure the accuracy and reliability of the data, but with so many sites, subjects, visits, case report forms (CRFs), and data points it can be difficult to stay on top of the entire process. 

Using existing metadata (see Mining Metadata for Clinical Research Activities for more information on metadata) graphics can be created to visually represent the overall status of each requirement for database lock. This is possible using a graphic called a ‘heat map’ that displays the CRF metadata. The resulting graphic is shown below. 

heat map showing CRF metadata for database lock

The graphic has one row per subject and one column for each CRF collected at each visit. This results in one ‘box’ per subject per visit per CRF. Each box is colored and/or annotated to indicate the current status of each CRF. 

Broadly speaking, a quick glance at this graphic can show the clinical study team exactly how many CRFs have yet to be completed, where queries have not yet been closed, which CRFs have been source data verified, and whether or not an individual CRF has been locked.  Not to mention, all of this information can be identified for a specific subject at a specific visit for a specific CRF. 

Focusing on the details of our particular example, it is easy to see that no subject has yet initiated data entry for both Visit 4 and Visit 5. Additionally, three subjects have not started data entry for the Treatment Visit, ten for Visit 1, fifteen for Visit 2, and twenty-four for Visit 3. An open query remains for several subjects on the TRT form at the Treatment Visit, and for just subject 88528 on the PE form at the Screening Visit. A handful of forms have been source verified and no CRFs have been locked. Additionally, the graphic provides detail on the total number of subjects, visits, and CRFs for the study. This helps reveal specifics such as which visits are more burdensome with multiple CRFs and exactly how far along the subjects are in the study. 

Historically, this information has been conveyed through pages and pages of multiple listings, which can take minutes if not hours to decipher. Having all of the information in a single snapshot can help determine what steps need to be taken to get to database lock quickly and accurately. 

Further instruction on how to implement this graphic within SAS will be available soon. 

Post-Lock Data Flow: From CRF to FDA

Key Take-aways from the 2nd Annual Strategies in Patient Centered Clinical Research Conference

Posted by Brook White on Tue, Aug 01, 2017 @ 10:16 AM

In July, delegates from Rho, Shann Williams, David Cass, and Ryan Bailey, attended the 2nd Annual Strategies in Patient Centered Clinical Research conference in Boston.  They share their thoughts on the conference below.

Patient centricity is garnering broad attention in our industry, but finding time to thoroughly explore the principles and practices of the movement can be difficult.  Needless to say, we were excited to spend two full days immersed in discussion and collaboration with some of the industry pioneers in patient-centered clinical research.  Condensing all of the great content into a few paragraphs is impossible, but we wanted to share our take on the main themes and trends that stood out to us from the event.  

A Rapidly Evolving Movement

changes aheadWe are excited to see the patient centricity movement evolving and growing before our eyes.  We were fortunate to attend the inaugural conference in 2016, and the change from last year to this year was noteworthy.  Last year, much of the focus was on establishing the rationale for patient centricity, and convincing stakeholders that patient engagement was worthwhile.  This year, the presentations were much more focused on action and implementation.  The “why” and “what if” questions continued to permeate our conversations, but it’s encouraging to see that we’re not letting some uncertainty hinder progress.

Integrated Patient Engagement Specialists

integrationMany companies are prioritizing patient centricity by creating new positions and departments solely dedicated to patient engagement.  More importantly, these new patient engagement leaders are not merely figureheads, but integral corporate leaders with the authority to enact change at the business unit and study level.  Several presenters shared how their patient engagement teams are integrated into their R&D, study design, and protocol development efforts throughout the product lifecycle.  One presenter noted that, “every project coming to our protocol review committee will be asked if they considered patient insights when designing the trial.”

New Tools and Processes

With companies appointing patient-focused leaders and new departments, it is not surprising that many companies are now developing processes and tools to support better patient integration. The evolution of tools and procedural documentation provide a much needed structure and framework for moving patient centricity from concept to practice.
A sample of the tools mentioned:

  • Protocol Quality Assessment Tool with a patient engagement component
  • Patient Engagement Toolkit for Study Development
  • Unique Patient Plans for each therapeutic area
  • Redesigned Informed Consent Forms using professional designers and user interface experts
  • Simulation Labs to test drive the protocol with staff and patients before a trial launches   

Technology is a Tool, Not a Panacea


Several presenters discussed using emerging technologies to improve trials – e.g., using Uber and Lyft for transporting patients to visits, employing social media listening to determine patient needs, creating virtual, on-demand sites to conduct visits – but a prevailing theme was that technology alone will not achieve patient centricity.  One presenter lamented that in the past 35 years, pharmaceutical development has made no gains in time-to-approval or rate of approval, but cost has gone up an average of $3.0 billion.  Despite all the technological advances in the past three decades, none of the key performance indicators of our industry have improved.  We should leverage technology in our continuous improvement efforts, but not at the expense of investing in other strategies – like patient engagement – to reduce cost and time, and improve success rates. 

Metrics and Measurement

measurements and metricsYes, the dreaded “M” word.  From the outset, patient centricity has been hampered by a lack of data quantifying its benefit to trials.  Common sense arguments in favor of patient engagement – e.g., improved recruitment and retention, improved adherence, fewer protocol amendments – have propelled the movement, but the desire for metrics has not dwindled.  Capturing metrics takes time, and it can be difficult to determine the value of a set of broad and somewhat nebulous patient engagement strategies, but some groups are making headway.  Several presenters shared data reinforcing the value of patient centricity.


  • Internal review at a top-10 (annual revenue) Pharma company found that 26% of their protocol amendments were attributable to participant burden.  Another top-10 Pharma company reported seeing a year-over-year decrease in the number of protocol amendments after implementing patient-centered strategies.
  • Research between DIA and the Tufts Center for the Study of Drug Development found that the single strategy of engaging Patient Advocacy Groups in trial design, added 2-4 months of additional up front planning time, but ultimately saved millions of dollars in prevented protocol amendments and led to faster FDA approval.
  • Research out of the Clinical Trials Transformation Initiative (CTTI) found that improving the patient experience (e.g., user friendly informed consent, simpler eligibility criteria, reduced patient burden) could save 10s of millions of dollars in Phase II and Phase III trials. 


For an industry that is often criticized for moving too slowly and resisting change, it is inspiring to see the evolution in just one year’s time.  As companies move beyond concept and into implementation, we are eager to see how these changes will impact patients and improve our research.  

One of the quotes that stood out to us from the conference is “the biggest tragedy in clinical research is not a failed trial, it is a successful trial that fails in the real world because we didn’t design with the patient in mind.”  This serves as a poignant reminder that our work must be fundamentally focused on healing patients, not merely getting approval or getting to market, if we are to make a positive impact.

Webinar: Putting Patient Centered Principles Into Practice

Mining Metadata for Clinical Research Activities

Posted by Brook White on Wed, Jul 26, 2017 @ 09:48 AM

Derek Lawrence, Senior Clinical Data ManagerDerek Lawrence, Senior Clinical Data Manager, has 9 years of data management and analysis experience in the health care/pharmaceutical industry.  Derek serves as Rho's Operational Service Leader in Clinical Data Management, an internal expert responsible for disseminating the application of new technology, best practices, and processes.

Metadata: An Underutilized Resource

mine clinical database metadataAs anyone involved in clinical database creation knows, considerable resources are devoted to the development and validation of electronic data capture (EDC) systems. Once these databases are live and clinical data begin coming in, various processes for setting up data cleaning programming, database quality review, and reporting are put into play. Unfortunately, most of the processes are manual and require the data managers, programmers, and biostatisticians to have a series of specific conversations concerning the database’s setup, structure, and dynamic behavior that would in turn affect how programming tasks were approached and how biostatisticsshould best approach the data.

The solution for not only decreasing the amount of time spent setting up these activities, but also increasing the accuracy of said setup presents itself in the effective usage of the project’s metadata. This metadata, or “data about data”, spans all elements of the clinical database, including:

  • CRF metadata
    • Labels, formats, response options, entry requirements, field-level checks, etc.
  • Form metadata
    • Source data verification (SDV), signature participation, orientation (standard vs. log), etc.
  • Event metadata
    • Visit windows, associated CRFs, repeatability, access requirements, etc.
  • Query metadata
    • Current status, dates, resolutions, marking groups, etc.

Establishing Usable Datasets

data-mining.jpgThe first step in mining the metadata is to create machine-readable datasets from the source in question. In the case of most commercially- available EDC systems, the CRF and Event metadata contents of a project can be exported in a variety of formats (XML, Excel, etc.). During the nightly process by which clinical data are exported from our EDC studies and saved to the Rho network, we added a post-processing step where a macro reads in the exported study metadata files and produces working datasets. From here, these elements of the clinical database are machine-readable and available for use. Other standard EDC reports provide additional sources for Forms and Query metadata. These data can be extracted from the system either directly using an API (application programming interface) or by creating reports using EDC system-specific tools, which can be scheduled and saved to the network automatically. The contents of these reports can also be converted to datasets for ease of use.

A Wide Variety of Applications

From this point, we can automate a number of tasks that traditionally required manual review, specifications, and the application of subject matter expertise in order to successfully complete. From driving the database validation process to the creation of system performance metrics to the programming and configuration of statistical datachecks, the now-accessible metadata allows us to more rapidly and accurately initiate a multitude of tasks with much of the manual component removed. We will cover the use of some of the specific data monitoring and cleaning uses using study metadata in a series of future blog posts.

Post-Lock Data Flow: From CRF to FDA

Beating the Odds: 5 Strategies to Improve Clinical Trial Enrollment

Posted by Brook White on Tue, Jul 18, 2017 @ 10:03 AM

Brandy Lind, Senior Director OperationsBrandy Lind, Senior Director of Operations, has been in project management at Rho since 2004.  During this time, she has led several large programs and individual projects in multiple therapeutic areas.  She has worked on observational trials, interventional trials, and Phase I-III clinical trials throughout her career.

According to a 2013 report by the Tufts Center for Drug Development, timelines are typically extended to nearly double their original duration in order to meet enrollment goals which is a significant contributor to increased costs.  Hitting the clinical trial enrollment goal is not only important to meeting overall study timelines and budgets, it is critical to the success of the clinical study.  If you don't have enough participants, you won't have enough data to support your objectives.  That's why it's important to plan for enrollment just as you would plan for collecting and analyzing the data. In this post, we’ll share 5 keys to successful enrollment.

1. Identify sites with the most potential

choose the sites with the most potential, choose the biggest marbleIt all starts by choosing the right sites.  Research sites with experience in your therapeutic area.  Are there existing site networks that can be leveraged?  Check clinicaltrials.gov  to see which sites are currently participating in similar studies or have in the past. One of the best ways to find qualified, high performing sites is to reuse sites that have demonstrated both an ability to enroll and produce high quality data.  Past performance is one of the best indicators of how sites will perform on future studies.

Put the time in up front to complete a full feasibility.  It can be tempting to jump right into site start-up, but time and money spent on feasibility upfront can save you time and money downstream by preventing protocol amendments and enrollment struggles.  Ask sites about how they identify subjects, how many patients are already in their database, and how many patients come to their facility each month.  Find out what they do when they have exhausted their own database.  Competing studies can impact their ability to enroll on your study, so find out if they are currently conducting other studies in the same patient population.  If they are, press for information about whether they have access to enough patients for both studies or if inclusion/exclusion criteria are different enough to allow them to support both.

Get feedback from sites on the eligibility criteria and feasibility of operationalizing the protocol.  They may have suggestions for changes that can increase the enrollment rate without impacting the scientific integrity of the protocol.  The sites also know the patients well and can help you assess patient interest.  Consider whether the indication is serious enough that patients will want to participate and also consider existing therapies that are available to patients that may impact enrollment.

Study logistics can also impact patients’ willingness to participate, so consider the burden the schedule of events will place on participants.  How long will visits be?  Will they have to stay overnight?  Does transportation to and from the site present an obstacle for patients?  If you have an elderly or pediatric population, consider the impact on caretakers and how they will need to be involved in the process.  Sites can help you identify opportunities to ease these burdens for study participants and their caretakers.

Finally, prioritize sites based on start-up timeframe, recruitment potential, and previous research experience.  Make sure you also identify some back-up sites that can replace underperforming sites or be added later to boost enrollment.  You can achieve some efficiency by getting these sites through start-up along with the main sites you have identified.

2. Set expectations with sites

Make sure you set clear expectations with sites from the beginning rather than waiting until you start to see issues.  CRAs should work closely with each site to develop an individualized enrollment plan.  Where will they get patients?  What recruitment materials will work best for their site?  Are there other healthcare providers they can network with to gain additional participants?  A one-size-fits-all approach to recruitment rarely works.

Once sites are activated, have routine calls, individually and as a group, to discuss clinical trial enrollment strategies and plans.  Have sites share what’s working and what’s not, share ideas, and build relationships.  Find out from sites why patients don’t want to enroll.  If you do have to adjust the protocol, the earlier the better. During group calls, have your high performing sites share lessons learned, tips, and successes. 

Develop a plan for how you will address sites who are not meeting enrollment expectations.  Give them all the support you can, but also be prepared to follow through with your mitigation plan if they are not able to turn things around.

3. Start with accurate projections

start with accurate projections of enrollment ratesIt is tempting to create timelines based on best case scenarios.  Excitement about getting a new product to market, helping patients, and meeting investor expectations can all be strong motivators to be overly optimistic about enrollment projections, but this will just create more delays and increase costs downstream.  Be realistic about how many patients each site can enroll each month based on feasibility -- you (and your investors) don't want surprises!

Consider study-specific issues that may affect enrollment like seasonal effects.  Create graphics to show expected enrollment over time versus actual enrollment to date. Having this information at your fingertips can help you make informed decisions about whether to add a back-up site, change recruitment methods, and project when actual enrollment is likely to end.  Use this information to continually reassess your upfront assumptions, so you can be proactive if enrollment isn’t moving forward as expected.

4. Build relationships with sites

Maintaining strong relationships with sites is critical.  They provide key insights into the therapeutic landscape.  They are close to patients and can help you understand patient concerns and perspectives, which can help improve recruitment and retention.  

Use the investigator meeting as an opportunity to build and strengthen these relationships.  Explain the importance of the study and the potential benefits for the patient population.  Engage sites and make them a partner in your research. Creating a situation where sites want to work with you again will be of great benefit to future studies!

5. Collaborate with your CRO

handshake.jpgLook for a CRO that acts as an extension of your team and sees your study as a collaborative endeavor.  During enrollment, it's important to have constant communication between the Sponsor and CRO regarding the status of site activations and enrollment numbers, risks to the clinical trial if we are not on track to hit the goals, and mitigation strategies to get enrollment back on track.

7 Tips to Use Social and Digital Media to Recruit and Engage with Clinical  Trial Patients

10 Interviewing Tips for Jobs in Clinical Research

Posted by Brook White on Tue, Jul 11, 2017 @ 11:03 AM

are you ready for an interview for a clinical research job?I’ve been working at Rho for nearly 10 years and in the industry for more than 15 years.  During that time, I’ve interviewed a lot of people for a lot of different jobs in clinical research.  Here are a few tips that can help you stand out in the interview process.

1. Get the basic stuff right.

Regardless of the industry or the job you are interviewing for, there are some basic things you need to get right.  Show up on time.  Dress appropriately.  In most cases, that means wear a suit.  Be polite and respectful to everyone you meet, regardless of whether you think they are involved in the hiring process.  Shake hands.  Make eye contact when you are talking with people.  Silence your phone during the interview.  Be prepared to take some notes.  Even if you don’t use it, having a notepad and pen sends the message that you are prepared.  Send thank you notes—emails are fine.  This all may seem very basic, but I continue to be surprised by the number of job candidates who fail at one or more of these things.

2. Do your research.

The amount of information available today is astonishing.  Take advantage of it.  Before you ever walk in the door for the interview, you should learn as much about both the company and the people you will be interviewing with as you can.  Start by reviewing the company’s website and social media channels.  Search for news about the company including interviews with leaders of the company.  This can be a great source of insight.  If you have the names of the individuals you will meet during your interview, look them up on LinkedIn.  Most people today working in clinical research will have at least a basic profile out there, and some will have a lot more.  If you don’t have specific names, look at people with similar titles to the one you are seeking to see what kinds of experience they have.  

3. Know or Learn the Industry.

Whether you are a clinical research veteran with 20 years of experience or are seeking an entry level position, make sure you are up to date with what is happening in the industry.  Read up on current events in clinical research, make sure you are aware of industry trends, and brush up on applicable regulatory knowledge.  There are a number of great free news sources that can help you with this.  A few of my favorites are FierceCRO and FierceBiotech, Clinical Leader, and Applied Clinical Trials.  Obviously, the depth of knowledge and awareness expected will differ based on your role and experience level.

4. Use Your Network.

It’s a small world.  Do you know anyone who works at the company?  Let them know you are applying and ask questions about what it’s like to work there.  Even if you don’t know someone at that company, you likely know someone who knows someone or you know someone who works at another CRO that could be good sources of information and advice.  LinkedIn is a great tool for identifying these connections.  It will also let you see shared professional groups that might provide you with a connection.

5. Use Storytelling to Demonstrate Your Ability to Do the Job.

use storytelling in the interviewWhen you are answering questions in the interview, be prepared to provide specific examples from your previous experience.  If you are a recent graduate, examples from school projects and classes are acceptable.  Some interviewers will expect it, but, even when they don’t, telling a story that demonstrates your ability to do the job is much more powerful than providing a hypothetical answer.  Think through your accomplishments and your best learning experiences.  While it might not seem obvious, think about projects and studies where things have gone wrong.  Showing that you understand why things went wrong and how you learned from it actually can have more impact than someone who only talks about positive experiences.  If you want to learn more about using storytelling to better deliver your message, check out this presentation from Jeff Polish(and if you are ever in Durham, check out his live story-telling group The Monti).

Whenever possible, use experiences that demonstrate specific knowledge of the job.  For CRAs, this might mean talking about experiences during site visits, working with investigators and site staff, and writing trip reports.  For quality assurance professionals, it might mean talking about experiences with sponsor audits or regulatory inspections.

6. View Each Question as an Opportunity.

By the time you get to the interview, hopefully you have a good understanding of what your potential employer is looking for in the position.  For job openings at Rho, information specific to the role can be found in the posted job description and general information about what we are looking for in employees can be found on our website, particularly in the Our Values section.  While you should provide straightforward answers to the questions asked during the interview, each question is also an opportunity to demonstrate how you meet one or more of the desired qualities or skills for the job.  For example, I may not ask a direct question about team work, but being a team player is an important quality for Rho employees.  An astute job seeker could answer another question, like a question about their greatest accomplishment, with a story that shows how they worked with a team to achieve that accomplishment.  It’s okay to take a minute and think about your answer before responding. 

7. Don’t Bad Mouth Former Colleagues, Bosses, or Employers.

It never helps you to talk badly about former colleagues, bosses, or employers.  It’s fine to share challenges, as long as you talk about how you overcame them. When you cross the line into complaining, it can create the impression that you were the problem or that you will be difficult to manage.  We already know you probably aren’t that happy or you wouldn’t be looking for other opportunities, so you don’t need to get into the nitty-gritty of why you are leaving.  And it’s a small world (see item 4), so there is always a chance that the person you are talking to knows some of these people or already has an opinion about your prior employers.

8. Know Why You Want the Job.

The reason may be obvious to you, but be prepared to articulate why you want the job.  Be able to talk both about why working for the company appeals to you as well as why the specific role you applied for appeals to you.

9. Ask (Good) Questions.

questionsMost interviewers will leave time at the end for you to ask questions.  It is fine if you come up with some during the interview, but come prepared with a couple of questions.  If you will be interviewed by multiple people, make sure you have questions for each.  Some of the questions should be role specific and some should be about the company.  Here are some good options:

  • What do you like best about working at <company>?
  • What qualities do you think it takes to be successful in <job role>?
  • How would my success be measured if I were to be offered this job?
  • What are the greatest challenges in this role?

Unless you are talking to a recruiter or HR representative, avoid questions about compensation, benefits, and company policies.  Also, be cautious about asking questions related to requirements you may have—wanting to work alternate hours, what type of office space  you would have,  or upcoming time off for a planned vacation.  Most of these can be discussed and negotiated when an offer is made.

10. Close the Deal.

close the dealAt the end of the interview, if you really want the job, say so.  Close out the conversation by saying in a relatively direct way that you want the job and briefly restating why they should hire you.  It can be something as simple as “I’ve really enjoyed hearing more about <role> at <company>.  I think my skills and experience would be a great fit and I would like the job.”  Remember, send a thank you note referencing something you discussed in your interview.  This shows the interviewer you were actively listening and engaged.

Want to work at Rho?  Check out our job openings.

Watch: Meet Our Experts

Why Depression Studies So Often Fail:  Don’t Blame “Placebo Response”

Posted by Brook White on Thu, Jun 29, 2017 @ 02:34 PM

Jack Modell, Vice President and Senior Medical OfficerJack Modell, Vice President and Senior Medical Officer, is a board-certified psychiatrist with over 35 years of experience in clinical research, including 20 years conducting trials, teaching, and providing patient care in academic medicine, and 15 additional years of experience in clinical drug development (proof of concept through market support), medical affairs, successful NDA filings, medical governance, drug safety, compliance, and management within the pharmaceutical and CRO industries. Jack has authored over 50 peer-reviewed publications across numerous medical specialties and has lead several successful development programs in the neurosciences. Jack is a key opinion leader in the neurosciences and is nationally known for leading the first successful development of preventative pharmacotherapy for the depressive episodes of seasonal affective disorder.

Prior to joining the pharmaceutical and contract research organization industries, I was in clinical practice for twenty years as a psychiatrist and medical researcher.  And something I noticed very early on among my patients with major mental illnesses, particularly those with severe depression and psychotic disorders, was that they did not generally get better – at least not for more than a day or two – by my simply being nice to them, treating them with ineffective medications (e.g., vitamins when no vitamin deficiency existed), seeing them weekly for office visits, or by providing other so-called supportive interventions that did not directly address the underlying illness.  To be clear, this is not to say that kindness and supportive therapy are not critical to the patient-physician relationship (“The secret of the care of the patient is in caring for the patient” [Frances Weld Peabody, 1927]), but rather that kindness and support alone rarely make a biologically based illness substantially improve or disappear. 

With this background, I vividly recall my surprise upon being asked shortly after I joined the pharmaceutical industry:  “Can you help us figure out how to decrease the nearly 50% placebo-response rate we see in antidepressant trials for major depressive disorder?”  “Fifty percent?” I replied, incredulously.  “There’s no way that 50% of patients in a true major depressive episode get better on placebos or just by seeing the doctor every couple of weeks!”  “Seriously?” was the reply, and they showed me voluminous data supporting their figure.

I spent the next few years trying to figure out this apparent paradox.  Not surprisingly, the answer turned out to be multifactorial.  After careful review of internal and external data, as well as published explanations for high “placebo response rates” in clinical depression trials (much of which also applies to clinical trials in general), the following three factors emerged as being of particular importance because they are easily mitigated by proper trial design, thorough research staff training, and meticulous oversight of study conduct.

(1)  Subjects being admitted into clinical trials often had depressive symptoms, but did not truly meet criteria for major depressive disorder.  Examples include subjects with personality disorders whose symptoms wax and wane considerably with external factors (e.g., family or job stress), subjects with depressive symptoms in response to a particular stressor (not of sufficient severity or duration to meet formal criteria for a major depressive episode and likely to abate with the passage of time), and subjects who – for various reasons – may feign or exaggerate symptoms for the purpose of seeking attention or gaining access to a clinical trial.  Unlike the patients I encountered in my clinical practice, subjects with these presentations often do improve with supportive interventions and placebo. 

Recruitment of truly depressed subjects is made even more difficult by the widespread availability of reasonably effective medication options. Patients in the throes of a major depressive disorder, who sometimes have difficulty even making it through the day, are rarely keen to commit to the additional efforts, uncertainties, and treatment delays involved with a clinical trial when an inexpensive prescription for an effective generic antidepressant can now be filled in a matter of minutes. Indeed, as more and more generally safe and effective medications have become approved and readily available for a variety of illnesses, the motivation for patients to join clinical trials in the hope of finding an effective treatment has correspondingly decreased.

(2) The second factor is somewhat difficult to discuss because it sometimes provokes an understandable defensive response in clinical investigators.  Consciously or unconsciously, many investigators and clinical raters inflate or deflate clinical ratings to enable the subject to gain entry into, or remain enrolled in, a clinical trial.  Most commonly, this is done by subtly – and sometimes not so subtly – coaching subjects on their answers, or when subject responses or findings seem to fall in between scale severity ratings, by rounding up or down to a rating that is more likely to qualify the subject for the trial. 

The effect of this practice is diagrammed in the following figures, specific examples of which can be seen in these references.1-3 In Figure 1, the white bell-shaped distribution is the expected distribution in severity rating scores of an unselected clinical population presenting for clinical trial participation, let’s say with a mean score at shown at X̄n. Not uncommonly, what we see in clinical trials in which a certain scale severity score is required for study entry (depicted by the vertical light blue line, with a score to the right of it required for entry) is not the expected right half of this bell-shaped distribution, but rather a distribution like that shown by the orange curve, which is essentially the right-half of the bell-shaped distribution with a large proportion of subjects whose ratings fell short of required severity for study entry (to the left of the blue line) “pushed” to the right, over the blue line, so that the subjects now qualify for study inclusion, with the mean of those thus selected shown at X̄s.

Figure 1


At the first follow-up visit, when raters (and subjects) now have little incentive to influence rating scores to meet a pre-specified criterion, the scores of the entire included population are free to relax towards their true values and assume the pre-selection and more normally distributed pattern.  Moreover, subjects and investigators, expecting that the onset of treatment should coincide with at least some clinical improvement, may bias rating scores during this period to reflect this expectation even though the signs and symptoms of the illness may have yet to show true change.  During this same time, any actual clinical improvement will also result in the rating score mean shifting leftward (white arrow, figure 2), but because the measured change – from the initial X̄s of the selected population to the new mean (X̄n1; orange arrow, figure 2) – is generally much greater than a true treatment effect during this period, any real changes are obscured and the ability to detect a true drug-placebo difference may be lost.  While this early “improvement” in rating scores for subjects in clinical trials may appear to be a “placebo effect” and is often confused with it, this apparent improvement is instead the result of artificially inflated scale scores regressing back to their original true distribution, in combination with whatever actual treatment and placebo effects may have occurred during this time.  Unfortunately, the introduction of non-qualified subjects to the study and rater bias will continue to hamper detection of actual drug-placebo differences throughout the course of the study.

Figure 2


(3) Finally, investigators and site staff often do not fully understand the true objective of the clinical trial:  it should never, for example, be “to show treatment efficacy” or to show that a product is “safe and well tolerated,” but rather, to test the null hypothesis of no treatment difference or to estimate likely treatment effect, as well as to faithfully and objectively record all adverse effects that may emerge during treatment.  Likewise, investigators and site staff often fail to understand the importance of complete objectivity and consistency in performing clinical ratings, the intention behind and importance of every inclusion and exclusion criterion (necessary for their proper interpretation and application), and the destructive effect on the outcome and scientific integrity of the trial that even well-intended efforts to include subjects who are not fully qualified can have.  

Each of these three factors can skew both drug and placebo trial populations and results, making it appear that subjects “improved” well beyond what would have resulted had there been strict adherence to protocol requirements and objective assessment of study entry and outcome measures.

What, then, can be done to prevent these problems from sabotaging the results of a clinical trial?  Foremost are thorough and meticulous investigator and rater education and training.  All too often, perfunctory explanations of the protocol and clinical assessment tools are provided at investigator meetings, and “rater training” takes the form of brief demonstrations of how the rating scales are used and scored, without actually testing raters to be certain that they fully understand how the scales are to be used and interpreted, including understanding scoring conventions, criteria, and necessary decision-making.4  Even seemingly sound training has marked limitations both immediately and as training effects deteriorate during conduct of the trial.4-7 

Training of the research staff must include not only a review of the protocol design and study requirements, but detailed explanations about why the trial is designed exactly as it is, the importance of strict adherence to study inclusion and exclusion criteria, and the necessity for complete honesty, objectivity, and consistency in conducting the clinical trial and in performing clinical assessments.  Detailed training on the disease under study is also important to ensure that all site staff have a complete understanding of the intended clinical population and disease being studied so that they can assess subjects accordingly.  And, as noted above, rater training must include not only education on the background, purpose, characteristics, and instructions for each scale or outcome measure used, but trainers, as well as investigators and raters, should be tested for adequate understanding and proficiency in use of each of these measures. 

Meticulous monitoring during the course of the study is also essential to ensure continued understanding of, and compliance with, protocol requirements, as well as accurate and complete documentation of study procedures and outcomes.  Study monitors and others involved with trial oversight should review data during the course of the trial for unexpected trends in both safety and efficacy data, and not simply for identification of missing data or isolated datum outliers.  Unexpected trends in safety data include adverse event reporting rates at particular sites that are much higher or lower than median reporting rates, and vital signs that are relatively invariant or favor certain values over time.  Unexpected trends in efficacy data include changes in closely related outcome measures that are incongruent – for example, objective and subjective ratings of a similar outcome differing considerably in magnitude or direction, that are much larger or smaller at particular sites than those observed at most sites, that occur in relatively fixed increments, and that show unusually similar patterns or values across subjects. 

Finally, and perhaps most importantly, is that no matter how well-informed or well-intentioned investigators and raters might be, humans simply cannot match computers in objectivity and consistency, including of the kind needed to make assessments based on subject responses to questions in clinical trials.  Unless being programmed to do so, a computer cannot, for example, coach a subject on how to respond, nor would it inflate or deflate ratings based on feelings, expectations, response interpretations, or desired outcomes.  A computer faithfully asks the same questions every time, following the same algorithm, and records responses exactly as provided by the subject.  Indeed, several studies have shown that computerized assessments of entry criteria and outcome measures in clinical trials – in particular interactive voice response systems (IVRS) and interactive web response systems (IWRS) – provide data of quality and signal-detection ability that meet and often exceed that obtained by human raters.1,3,7,8,9  For these reasons, strong consideration should also be given to using IVR and/or IWR systems for assessing study entry criteria and endpoints that allow such use.  

The author acknowledges John H. Greist, MD, for his outstanding research and input regarding these important findings and considerations.


  1. Greist JH, Mundt JC, Kobak K.  Factors contributing to failed trials of new agents:  can technology prevent some problems.  J Clin Psychiatry 2002;63[suppl 2]:8-13.
  2. Feltner DE, Kobak KA, Crockatt J, Haber H, Kavoussi R, Pande A, Greist JH.  Interactive Voice Response (IVR) for Patient Screening of Anxiety in a Clinical Drug Trial.  NIMH New Clinical Drug Evaluation Unit, 41st Annual Meeting, 2001, Phoenix, AZ.
  3. Mundt JC, Greist JH, Jefferson JW, Katzelnick DJ, DeBrota DJ, Chappell PB, Modell JG.  Is it easier to find what you are looking for if you think you know what it looks like?  J Clinical Psychopharmacol 2007;27:121-125.
  4. Kobak KA, Brown B, Sharp I, Levy-Mack H, Wells K, Okum F, Williams JBW.  Sources of unreliability in depression ratings.  J Clin Psychopharmacol 2009;29:82-85.
  5. Kobak KA, Lipsitz J, Billiams JBW, et. al.  Are the effects of rater training sustainable?  Results from a multicenter clinical trial.  J Clin Psychopharmacol 2007;27:534-535.
  6. Kobak KA, Kane JM, Thase ME, Nierenberg AA.  Why do clinical trials fail.  The problem of measurement error in clinical trials:  time to test new paradigms?  J Clin Psychopharmacol 2007;27:1-5.
  7. Greist J, Mundt J, Jefferson J, Katzelnick D.  Comments on “Why Do Clinical Trials Fail?”  The problem of measurement error in clinical trials:  time to test new paradigms?  J Clin Psychopharmacol 2007;27:535-536.
  8. Moore HK, Mundt JC, Modell JG, Rodrigues HE, DeBrota DJ, Jefferson JJ, Greist JH.  An Examination of 26,168 Hamilton Depression Rating Scale Scores Administered via Interactive Voice Response (IVR) Across 17 Randomized Clinical Trials.  J Clin Psychopharmacol 2006;26:321-324.
  9. http://www.healthtechsys.com/publications/ivrpubs2.html 

Webinar: Selecting Inclusion/Exclusion Criteria for Your Next Trial