Let's say you see a technical paper that is clearly and fundamentally flawed, published in a well-respected journal.

Let's say you obtain the original data from the authors and put together a demonstration that the proposed method isn't working.

Let's say you write a Comment describing your results and send it to said journal.

And let's say the journal editors, while agreeing that you have a valid point, reject your Comment without further consideration because it is "not significant."

Now where else do you send your negative results for publication?

Grateful for any suggestions.

Would you consider blogging your analysis?

You bet. The analysis and then some - editor responses and critical analysis of these.

I would still like to have something resembling a paper to be more "citable".

And yes, Insight/Midas Journal are being considered.

PLOS One was another thought since they seem to be willing to publish negative results, but I think this isn't "primary" enough for them (they specifically do not want comments).
Originally posted by Matthew Brett:

Perhaps BMC Research Notes
  http://www.biomedcentral.com/bmcresnotes...
Unlike PLoS One they are happy to receive correspodence articles (I don't think they have to relate to an article published in BMC RN, though it's not totally clear from a quick glance...)
  http://www.biomedcentral.com/bmcresnotes...

Best wishes,
Ged


Originally posted by Torsten Rohlfing:

Nice find - might just work.

Thank you Ged!


Originally posted by Ged Ridgway:

How about an 'opinion article' on frontiers?

Here is one blog option

http://retractionwatch.wordpress.com/

I think it is always best to publish a response in the original journal in which the paper has appeared.
At least that's my opinion.
Best,

Arno

If the scientific publishers where doing their job, this
should be a simple matter of posting a comment
online to the original paper, in the Journal's web site...

but,

...as we all know, most of scientific publishing is a broken
industry whose goal is only to provide career evaluations
of academic employees to facilitate the job of academic
administrators. In short, papers are simple tokens to be
counted in academic resumes to facilitate the evaluation
of promotions and job applications. That's why it is ok
to continue using pre-Internet technology for scientific
papers... indeed... we still call them "papers".

Most publishers have no concern for disseminating
useful scientific information, much less for supporting
the reproducibility verification requirements of the
scientific method.

The societies that run most Journals are only interested
on the revenue that the journals provide and have lost
track of their mission of promoting the progress of
science an technology.

Academics, for the most part, play along in collusion with
these unscientific publishers, by volunteering their time
to review, edit, submit and cite papers, and thereby giving
credibility to the papers and the journals. Despite the fact
that such publications that have no scientific content, given
that they do not enable anyone to verify the replication of
results. They offer "peer-review" written opinons as the
deceptive substitute for what should have been an attempt
for replicating results. All forming a vicious circle in which
researcher's reputation is traded in exchange for journal's
reputation.

The worst part of this state of affairs is that senior researchers
in labs all over the world keep corrupting young researchers by
mentoring them with the distorted notion that scientific research
is about "Publish or Perish", a cheap marketing slogan that
only serves the profitable business of publishers and has no
regard for the economic investment that society at large make
in scientific research.

Mentors fail on their responsibility to train the next generation
of researchers on the actual honest practice of the scientific
method. Many grad students have no idea that the core
practice of the scientific method is the verification of
reproducibility, instead they are mislead to believe that
peer-review is a sufficient substitute, and that publising
papers is the final goal of scientific research.

We see the large majority of Journal and Conferences asking
for novelty, (which is indeed irrelevant to the scientifc method),
and we only see a handful of venues that care about reproducibility.
Journal are confused trying to play the role of the Patent Office,
and have further confused "scientists" with "inventors" despite
the fact that these are two very different professions.

In particular, their is an abysmal lack of venues where one
could publish attempts of replication of results, whether they
are positive or negative results.


It is time for each one of us to ask ourselves if we are in
this field, just because we needed a job, or whether we
honestly care about the practice of scientific research,
and if so, whether we have the character to take action
and fix the many things that are broken in scientific
publishing.

It is great to hear that you are considering publishing
your findings in the Insight Journal. It is certainly one of
the few places where reproducible articles are welcomed,
and where the materials that needed for replicating results
(source code, data, scripts) are actually a requirement.

                   http://www.insight-journal.org/

True to the real practice of science:

                           'Nullius in verba'

http://en.wikipedia.org/wiki/Nullius_in_...

                   "Take nobody's word for it"



------------------

More on the sad state of affairs of scientific publishing:

http://videolectures.net/cancerbioinform...
http://www.nature.com/nm/journal/v17/n1/...
http://www.nature.com/nature/journal/v48...
http://www.nature.com/news/replication-s...
http://online.wsj.com/article/SB10001424...
http://michaelnielsen.org/blog/reinventi...

Thanks to everyone for the suggestions!

(I was wondering what took Luis so long, but obviously there was a lot to type) ;)

Re retractionwatch - the original paper isn't factually false so much as to need retraction, just pointless.

Re original journal - well, that's what we tried, but there's a wall of arrogance and incompetence that prevents it. (Actually, quite a striking example of the Dunning-Kruger effect, https://en.wikipedia.org/wiki/Dunning%E2...)

Again, thanks to all, and I'll post a pointer here when we move ahead with whatever turns out to be the most appropriate option.

TR

Hi Torsten

For biomedical sciences, there is a special journal for negative results:

Journal of Negative Results in Biomedicine

http://www.jnrbm.com/

Maybe there is a similar journal for technical papers.

Cheers
J=FCrgen

So to follow up on this matter, as of today our controversial Comment is indeed under review at BMC Research Notes. Hooray! (And thanks again to Ged for the suggestion).

Something interesting I learned in the process that may be worth sharing:

BMC was concerned that our re-use of data published in the criticized paper would violate copyright of the original paper's publisher, so we would need permission for our re-use. I do disagree with that, because of the obvious Fair Use nature of our Comment and also because we are only using the same image data, but not entire figures from the original paper. Much like you can use the "Lena" image without asking permission from anyone else who's used the same image in their paper before.

But that is not the interesting point - the interesting thing I learned is this: the original paper was published in an IEEE journal, and as an IEEE member, one of my membership benefits is that I can get permission for re-use of any IEEE-copyrighted materials for a processing fee of, hold your breath, $3.50.

You may of course think of copyright and licenses and copyright transfers to journals what ever you will. But I think that's kinda cool, especially that it took literally 5min to get the re-use permission online rather than argue with BMC over the merits of our Fair Use defense.

Now let's hope BMC is asking people for reviews of our Comment who actually understand the subject matter...

Cheers!
  Torsten

Originally posted by Torsten Rohlfing:
That is indeed a bargain....:-)

Worth of a blog post for the Summer of Data Sharing !

Congratulation for bringing your paper up to BMC.

Luis

Hi Torsten--I think it is a certainty that in some circles publishing research papers has become much more about propelling the careers of the scientist forward, rather than contributing to scientific advancement in the field! It is a very depressing reality to face, but good for you challenging the status quo!  As you move forward with this, I would tread lightly--as it will most assuredly annoy the authors, and could be responded to with some backlash!!
Best of luck--Christine 

Hi Christine -

Thanks for the advice, but I am afraid you're probably too late. Have you seen my too most recent papers?

T. Rohlfing and J.-B. Poline, "Why shared data should not be acknowledged on the author byline," NeuroImage, vol. 59, no. 4, pp. 4189-4195, 2012.

T. Rohlfing, "Image similarity and tissue overlaps as surrogates for image registration accuracy: Widely used but unreliable," IEEE Transactions on Medical Imaging, vol. 31, no. 2, pp. 153-163, 2012.

Clearly, treading lightly is not what I do. And frankly, people treading lightly in the presence of bad science is not something we as a community should encourage.

Best,
  Torsten

... which reminds me of an experience a colleague of mine once had. When the first paper about the "Mozart effect" came out (you know, that BS that kids get higher IQ by playing them classical music when they're babies).

Anyway, my friend ran an experiment to confirm the effect and found that there isn't one. They wrote up the study and sent it to the same journal that published the original article. Their manuscript came back right away, with a note from the editor thanking him for his "amusing contribution."

Which leads me to point out that overcoming bad science is only part of what needs to be done. Overcoming ignorance and arrogance is equally important, and putting it out in the open is probably the best way to do that.

TR

And the winner is... BMC Research Notes!

At long last:

The Comment that IEEE-TPAMI did not want you to read!

https://www.biomedcentral.com/1756-0500/...


So to finally fill in all the blanks - about
a year ago, I saw a terribly invalid "registration" paper, in
IEEE-TPAMI of all places. Contacted Brian Avants; he had seen it too and
shared my concerns.

So we obtained the original data and
confirmed our suspicions that the proposed method was not doing
registration at all. To correct the record, we then wrote a Comment,
which we sent to PAMI.

Much to our surprise, our Comment was
rejected without review as "technically valid but insignificant." We
appealed but were turned down, with some rather interesting arguments.
Such as: "you are ignoring the assumptions made by the authors" (we were
using their own data!), and "the algorithm assumes intensity constancy -
it says in the title, 'intensity-based registration'."

Long
story short, we took the Editor in Chief's friendly advice, sent our
Comment elsewhere, and here it is. For your further entertainment and
education, you may also want to look up "Dunning-Kruger Effect" on
Wikipedia. :)

Very Nice, 

It is like doing science again.   :-)

This is yet another illustration of why Journal should NOT be taking the liberty of skipping the VERIFICATION of REPRODUCIBILITY, as part of the review process.

Peer-review is useless and dishonest when it does not include an actual verification of reproducibility.

It is very disappointing to see how the large majority of Journals and Conferences are satisfied with the mediocrity of doing peer-review based on opinions, instead of practicing Science honestly and demanding the verification of reproducility as part of the review process of every publication.

On the bright side, at least, a handful of Journals are doing real science by embracing the Reproducibility Initiative of the Science Exchange
https://www.scienceexchange.com/reproducibility

Everybody else, should be ashamed of publishing, reviewing or citing any paper that has not been replicated. Such mediocrity should not be associated with the scientific endeavor.

For those who care about actual scientific work, (e.g. REPRODUCIBLE), you may find interesting the upcoming workshop on:

"Reproducibility in Computational and Experimental Mathematics" 

http://icerm.brown.edu/tw12-5-rcem


   Luis

Forgot the link to the paper in the original post:

https://www.biomedcentral.com/1756-0500/...

Sorry :)