Sunday, May 24, 2015

Rejection Letter For the Paper 'A Falsification of the Aristotelian Theory of the Free Fall and an Alternative Theory'

Rejection Letter For the Paper 'A Falsification of the Aristotelian Theory of the Free Fall and an Alternative Theory'

(a pdf of this text can be found here)

Cynicism is definitivly a problematic thing if being using in academic argumentations. Well, the text below is actually not meant to be cynical. It only tries to describe a non-exceptional situation in the current scientific selection process: the rejection of a paper that is based on measurements. The kind of argumentation in the reviews is not invented. It is an extraction of multiple reviews I got over the last few years. The text gets its cynical flavor because it refers to a work whose result and impact is known to us: the (hypothetical) experiment by Galilei on the free fall.
It should be clear that I do not and will not (not even try to) draw any parallels between my work or any other's work with the work by Galilei.
....have fun reading the text...
Stefan Hanenberg
stefan.hanenberg@gmail.com
version 0.1, Essen, 2015-05-23

Dear Mr. Galilei, 

thank you for submitting the paper 'A Falsification of the Aristotelian Theory of the Free Fall and an Alternative Theory' to the special issue 'Physics and Stuff' of our journal 'Software Technology Usage in Productive Industrial Development'.

Unfortunately, I have to tell you that the paper is rejected based on the common proposal of all reviewers.

The reviews are attached to this notification in order to help you improving your paper and improving your future work.

With kinds regards,The Editor

First Review


Overall merit: 1. Reject 
Reviewer expertise: 1. Expert


Summary


The paper gives a short introduction of the aristotelian understanding of the free fall and discusses potential problems with this theory. Then, the author runs a small controlled experiment where two cannonballs are dropped from some rather peculiar tower. From the experiment's measurements the author concludes that the aristotelean theory of the free fall must be wrong. In a second experiment, the author dropped even more cannonballs with different weights and proposes (based on the measurements) a different relatively trivial model that is from the author's perspective a better theory of the free fall.

Review

I am quite positive about empirical studies and results in general. Still, I cannot hide that I am not convinced at all about the here proposed experiment and the conclusions drawn by the author.

First, the description of the aristotilean theory and its relevance is hardly described. In fact, the author just says that it is a theory that exists since some centuries and plays a major role in our current understanding of the real world. However, it is unclear why this theory should be relevant at all, or why a change in the theory should be necessary (taking into account that this theory exists since centuries!). Hence, neither the background of the theory nor the possible impact of changes to that theory are clearly described which makes it hard for readers to understand why it should be even interesting to read this paper et al.

Second, the experiment is far from being convincing for a large number of reasons.

1) The author uses cannonballs of different weights and drops them from a tower. As the author mentions the tower is not an ordinary one but seems to have some peculiarities (why do such towers even exist in Italy?). As a consequence, it is quite plausible that the experiment results (whatever they are) are highly influenced by the choice of the tower itself and not by the theory being tested. Hence, the reviewer urgently asks the author to rerun a different experiment with a different tower. 

2) Next, the author choses cannonballs. As a consequence, he completely ignores that cannonballs serve a special purpose. Cannonballs are explicitely designed in a way that their behavior with respect to being shot or being dropped is very similar - no matter what their weight is. Such statements by cannonball designers are even completely ignored in the related work section. Definitively, well known papers such as "Why I think writing software for cannonball designers is a better option than having no job at all" by Fubar et al. '63 which is a fundamental and ground-breaking paper for the whole cannonball industry must be mentioned. Hence, it is clear that just because of the used subjects in the experiment it was not even possible to show anything else than just similar results - the experiment does not falsify any theory but just gives another indicator for the maturity of our cannonball industry.

3) The measurement process is not described. While the author mentions that the cannonballs had different weights (in the first experiment 1 kg vs. 10 kg) it is completely unclear how the measurements are performed - the precision of the measurements are completely unclear and the author does not discuss how the measurements were performed. Even worse, the author just describes the tower in terms of its height (without mentioning any other measurement) and - again the 56 meters are given without describing any other measurement. Taking into account that the tower seems to have some problems with its fundament, the height measurement is obviously not enough to describe the essential parts of the experiment. The time measurement is hardly described: the author just describes on page 3 that the time measurement started from the moment when a cannonball was dropped until the cannonball hit the floor and that the author used some clock that shows even miliseconds*10. We need to take into account that even the definition of a second changed over time (mean solar day second vs. period of the Earth's orbit around the Sun vs. atomic clocks). From that we can conclude that none of the time measurements is trustworthy -- as the author should have noticed, there are small deviations between the different measurements. This shows clearly that the measurements are not trustworthy at all. Hence, we must not conclude anything from the measurements.

4) The sample size is much to small for any serious study. Taking into account that for the first experiment - only 2 kinds of cannonballs have been used (again, all measurements were performed from the same tower) only 10 time measurements were collected it is completely impossible to generalize from the measurements to anything else. However, such kind of generalization is done in the second experiment. In the second experiment the author concludes from 10 different - yes, again cannonballs - and 20 different heights - again without giving a precise description of the measurements - that all measurements can be described with a formula t(h)=sqrt((2*h)/9.8). Again, such a formula cannot be derived from the measurements (for the reasons explained below).

5) The resulting formula appears completely unmotivated -- where does it come from and why should the free fall described in that way?

6) The analysis of the experiment is somehow obsure, especially when taking into account that a precise number is the result of the formula: not a single measurement matches the expected result from the formula! The author should have noticed that not a single measurement fulfills the formula. Instead of mentioning this obvious thing, the author tries to rescue his experiment with some statistical tests. The applied test (some so-called significant test) is not explained in detail and the rather mysterious results of this test are not explained. The author should have noticed that even an obvious test (the arithmetic mean of the results) differs from the formula's results.

7) The external validity of the experiment is in fact zero. Only cannonballs have been dropped in order to argue that the aristotelian theory does not work. I strongly advice to repeat this experiment with multiple other objects to be dropped from the tower. It seems obvious to drop additional things such as water, sand, or even complete ships in order to increase the experiment's external validity.

8) The related work section is far from being complete. Again, fundamental works about cannonball constructions are missing, no work is mentioned about the used tower, and not even different works on time measurements have been cited.

Hence, I conclude from the review above that the paper has to be rejected: the motivation is unclear, its relevance is unclear, the measurements are unclear and as a consequence, no conclusions must be drawn from these measurements. Additionally, the external validity of the experiment is not given. Although the proposed alternative model for the free fall is interesting the paper does not give any valid trust in the validity of the model.

Second Review


Overall merit: 1. Reject
Reviewer expertise: 1. Expert

Summary


The paper describes what happens when cannonballs are dropped from a tower. The author performs a number of measurements and states that these measurements conflict with an older theory of the free fall. Finally, the author describes his personal formula for the free fall which does not seem to be consistent with the older theory.

Review

While the general idea of the paper is interesting and the author's conclusions are quite innovative, the paper has a restricted perspective: the whole argumentation is based on quantitative measurements. Probably the most important information is missing in the paper: The design process of the experiment is completely unclear. Why was this extraordinary tower used for the experiment? Why were cannonballs used? Why was the measurement based on time? What were the reasons to come up with the final formula proposed in the paper? How can the formula be explained?

According to this, the general comment to the paper is that the paper lacks of any qualitative analysis that is relevant to the studied topic.

a) What additional observations were made in addition to time, height and weight? The pure use of quantitative data is a too restricted perspective on any aspect of daily life. The chosen cannonballs are only explained in terms of weight: it is clear that additional characteristics of cannonball are essential, too (What are they made off? Who was the producer? Has their functionality been tested before? How can the surface of the cannonballs be described? Were they comparable?). With respect to height, it is unclear how the tower can be described best. The author mentions (in addition to its height) only one special characteristic of the tower in one single sentence. All other aspects of the tower are completely ignored. With respect to the time measurement, it is unclear why the authors tried to measure time in such a complex way and not only asked people whether they saw differences in the free falls of the cannonballs. Additionally, the author not even tries to describe the different ways how the cannonballs fell down (although the measurements do show differences!). Hence, the most essential information -- the different ways how the cannonballs fell down -- are missing in the paper. Because of the resulting missing qualitative analysis it is not possible to find explanations for the differences in the way how cannonballs fall down from a tower.

b) The chosen experimental design reveals some obvious weaknesses. In the first experiment, the author uses two cannonballs in multiple measurements. Hence, the indiviual influences of a single cannonball is very high and it cannot be expected that the resulting measurements imply anything meaningful: it is well-known that AB experiments have the problem of unbalanced groups and this effect is even stronger in the here proposed experiment because of the use of the same cannonballs. In order to get rid of the problem, it is more desirable to have multiple different things to be dropped from the tower. Again, qualitative studies could help in order to find out what kind of different things could be dropped from the tower. An additional qualitative study could show what additional items could have been carried on top of the tower.

c) Because of the missing qualitative data, it is impossible to replicate the experiment. In case someone wants to replicate the experiment, it is necessary to understand what kind of tower could be used in the experiment and what kind of cannonballs could be dropped. Because of the special characteristics of the tower it seems even impossible to replicate the experiment, because it ia rather unlikely that such towers can be found somewhere else in the western hemisphere.

While I think that the provided quantitative data has some value it is still necessary to provide additional qualitative data and to run an additional qualitative analysis.

Minor comment:

The paper needs proofreading by a native english speaker.

Third Review

Overall merit: 1. Reject
Reviewer expertise: 1. Expert

Summary: 

The author describes an experiment that tries to falsify an existing theory: the aristotelian theory of the free fall. Based on two experiments, the author comes to the conclusion that the theory must be wrong and the author proposes an alternative theory.

Review:

The starting point of the paper is quite unusual. While most of the works that can be found today in physics address real world problems such as alternative sources for energy or the construction of large machineries somewhere in Europe (Switzerland), the paper addresses a more basic topic: the free fall. But it is unclear how this could be able to provide any usable or useful insights that could be applied today. Hence, the relevance of the work is unclear.

The line of reasoning in the paper is problematic. The authors try to falsify a theory (free fall) by some measurements in an AB experiment that actually do not differ: the author does not measure a difference between group A and group B. However, the literature explicitely says that measuring no differences is no indicator that there are no differences. It could only means that the experiment is problematic. Hence, the p-value of .99999 cannot be interpreted as no difference between A and B. The same is true for the second experiment where multiple measurements are compared (minor comment: the author forgot that a correction is needed because of the cummulated alpha error).

The author ignores completely that the theory of the free fall is no longer relevant - take into account that our industry is in the meantime able to construct things such as airplanes. Even more plausible comparisons are not being done. For example, birds do land with different speeds which directly contradicts almost everything that can be found in the paper. Hence, the proposed theory is not only a pure artificial one, it is even possible to find direct contradictions with it in reality (landing birds). Hence, the general idea of falsifying the aristotelian theory is not only irrelevant, it is wrong.

Minor comment: The author should have noticed that the proposed theory reveals results that are not measurable with the clock he used -- nor with any other clock that exists today (because all of them are not precise enough). As a consequence, it is clear that any arbitrary measurement inherently falsifies the proposed theory.


No comments:

Post a Comment