Challenges

According to the scientists' letter, some coronavirus-related studies on the registry suffered from poor design, including insufficient sample sizes, inefficient control-group planning, poor randomization, unclear evaluation standards, incomplete data and other issues.

One notable example was a study on the benefits of soybean extract for severely infected patients that had just two people in each of its control and treatment groups.

Other observational studies have investigated the therapeutic effects of physical exercise, such as shadowboxing or taking six-minute walks every day, which experts said are hard to quantify and analyze scientifically.

Zhang Lan, director of the School of Pharmacology of Xuanwu Hospital at Capital Medical University in Beijing, said low sample sizes in some experiments are due to a gradual decline in the number of newly confirmed cases, as well as difficulties in finding patients who haven't received other treatments. All of these factors may affect the results.

"A big sample size can eliminate many biases and outliers that may otherwise interfere with the experiment," she said. "It is unscientific to say a drug is truly effective without proof from clinical trials. Without that, all the drugs used to treat COVID-19 (the coronavirus) are only considered potential candidates."

According to the open letter, an experiment would theoretically need around 1,000 subjects to prove a treatment's effectiveness on patients with mild symptoms. For severely infected patients, the sample size would need to be at least 800.

"It is impossible for the current number of patients to meet the sample demands of all the registered experiments," the scientists wrote. "But if the experiments are not well-designed, they are less likely to produce quality results that are effective and safe."

The letter rebuffed the view that clinical experiments can cut corners in design or ethical oversight during an ongoing epidemic in the hope of generating a larger number of insights more quickly.

"This notion is irresponsible for patients and can lead to unreliable results. Drugs and treatments that have been shown to be ineffectual can still have side effects; we cannot transfer the risks of ineffective drugs to patients," it said.

The signatories urged China's scientists to strictly uphold ethical and quality design for clinical trials, and called for stronger regulations and supervision from government bodies.

A biology professor from Beijing, who spoke on condition of anonymity, said research on new epidemics typically follows a "boom-bust "cycle in which there is high academic interest and support during an outbreak, but enthusiasm quickly wanes when it ends, leaving many projects unfinished.

"At times, it feels like the whole scientific community is grasping at straws and trying everything it can to curb the epidemic," he said.

"Despite the current emphasis on open science and generating findings, they are not excuses for poor experiment designs, and non-peer-reviewed results shouldn't be taken by the public as established fact."