Saturday, June 26, 2010

PyFilesystem and "filepath": abstracting the file-system in Python

PyFilesystem 0.3 released

Will McGugan: I am pleased to announce a new version of PyFilesystem (0.3), which is a Python module that provides a common interface to many kinds of filesystem. Basically it provides a way of working with files and directories that is exactly the same, regardless of how and where the file information is stored. Even if you don't plan on working with anything other than the files and directories on your hard-drive, PyFilesystem can simplify your code and reduce the potential of error.

PyFilesystem is a joint effort by myself and Ryan Kelly, who has created a number of new FS implementations such as Amazon S3 support and Secure FTP, and some pretty cool features such as FUSE support and Django storage integration.

[MMMG: Compare this to "filepath 0.1" from Jp Calderone]

Jp Calderone: I'm happy to announce the initial release of filepath.

filepath is an abstract interface to the filesystem. It provides APIs for path name manipulation and for inspecting and modifying the filesystem (for example, renaming files, reading from them, etc). filepath's APIs are intended to be easier than those of the standard library os.path module to use correctly and safely.

filepath is a re-packaging of the twisted.python.filepath module independent from Twisted (except for the test suite which still depends on Twisted Trial).

The low number of this release reflects the newness of this packaging. The implementation is almost entirely mature and well tested in real-world situations from its time as part of Twisted.

You can find the package on PyPI or Launchpad:

MMMG: This is all great stuff. From what I saw, the API of PyFilesystem seems like the winner, at least to my eyes. I will steal the best code from "filepath" to augment my personal version of PyFilesystem, then I will see what I can contribute back to these two wonderful projects.
Enhanced by Zemanta

Friday, June 25, 2010

Is climate science unique? Do the scientists need to be protected from themselves?

I discussion of bias, or is it a license to dispense with scientific results that are displeasing?  Where are the outside mechanism to help scientists when they cannot help themselves, because of their own bias.

Why is the mechanism of publication insufficient, when in all other scientific fields it has been exactly the way that the scientific community/mainstream dispensed with invalid results and replaced them with valid ones?

Collide-a-scape » Blog Archive » Collide-a-scape >> The Unbearable Lightness of Bias

If these (don’t-you-dare-call-them-deniers) “bias-busters” could point to successes in other scientific fields for other scientific questions, all thanks given to their “bias-busting” scrutiny, a lot of people would see the benefit and drop suspicion.
Otherwise, we must pre-suppose that climate science is a “nonesuch” science — the only scientific domain where the practitioners must be protected from themselves.
Where is the rich history of success of outsiders providing “bias-busting” services for scientists? The outsiders that really help scientific progress get to know the practitioners and labs and journals so intimately, they end up having choices about how they ultimately publish and add to the literature — and that is, in fact, how it ultimately plays out successfully. And *not* by providing outsourced “bias-busting” services.

reply to Hector M. #14
I don’t deny that some assert that climate scientists must be protected from themselves.  I am curious to learn of the historical cases where outsiders’ free-lance bias-busting was found to be successful.
Because similar arguments were made that a cabal of Darwinists suppress papers that show intelligent design, and a cabal of medical regulators suppress papers that disprove tobacco carcinogenicity, and a cabal of dentists suppress facts about fluoridation’s mild control properties.
If I could consider the historical cases of success of outsiders’ free-lance bias-busting, when scientific practitioners needed to be protected from themselves, I could discern between clear-eyed skeptics and delusional denialists, and see if the proposed cure fits the illness.
Much like the judgement of the need for carbon taxation may be informed by the history of regulatory prohibitions of  fluorocarbons.  (Or the regulatory prohibitions on the sale of alcohol in the United States, they don’t all have to be positive examples of regulation.)
I am genuinely interested to learn more about outsiders’ free-lance bias-busting in science, where it had an effect, maybe good, maybe bad.  You overlooked part of my original remark in your reply.

It would be much easier see the good intentions of these denialists, if they didn't demand that everyone assume that climate science is a nonesuch science.

[Ugh, more]

reply to Judith Curry #24

"Scientific biases are challenging enough, but when these are augmented by political bias and a policy agenda, then the bias issue becomes the overwhelming challenge for the science."

This is asserted, but not demonstrated.

As a process that embraces self-correction, it is hard to come up with examples of errors that science has allowed to let stand, for all time, because of political bias and a policy agenda. I can't think of any.

[Aside: The closest I can think of is the denying of the possibility of a numerical intelligence quotient in the polite company of scientist, even though it is at least as well established as the Big Five Personality model, which is uncontroversial. But does that even count? Nobody is barred from publishing and the truth is available to the motivated. I am genuinely curious - Are there any examples of errors that science has allowed to let stand, for all time, because of political bias and a policy agenda?]

Is there really a need for a unique outside agent to police bias in climate science, alone?  The same process of refereed journal publication, that serves the scientific community/mainstream in other fields to dispense with invalid results and replace them with valid ones, breaks down *only* for climate science?  And where is the model for successful outside policing of scientific bias?  Does one even exist?  That would be even more controversial than the already controversial remedy of carbon taxation, because there is already research on the economic effects of taxation, and a history of taxation to counter perverse externalities.

Or is the issue of bias - a bias so insidious science should not dare to leave the extermination to mere scientists - a red herring?

The demand to treat climate science as a "nonesuch" science is the rub.  How can I distinguish a prickly intolerance for the possibility of bias from an excuse to discard results that are unwelcome from an excuse to disregard consequences of status quo behavior?

Reply to Bill Stoltzfus #26

Part of the reason that Judith Cury's "Team B" idea went over like a lead balloon was that the name "Team B" had an unfortunate historical connotation to the "Team B", commissioned by Director of Central Intelligence George H. W. Bush, of "outside experts" who attempted to counter the positions of intelligence officials within the CIA.  In reality, the CIA failed to represent USSR as enough of a military threat for the liking of defense hawks, so Team B manufactured Soviet military capabilities from whole cloth.

I would rather 100% of published scientific results be attempted for replication by third parties, rather than 20% of funding be spent on fishing expeditions.  Because, if this is a mechanism outside of current scientific publishing guidelines, a group of outsiders would choose where the 20% was directed.  Leaving aside the issue that the comparative invulnerability to political and other agendas of this extra-scientific group is being simply asserted without basis.

Then there is the strange issue why science is singled out for hobbling.  What percentage of business, family, or personal decisions (the analog of scientific results) must be legislated for scrutiny by third parties?  Because is the track record of science worse than the tract record of business, family, or personal decision making?

[double ugh, even more]

Reply to Steve Fitzpatrick #29

I am looking for ways to distinguish accusation of insurmountable bias in climate science from such claims made against in evolutionary biology, tobacco carcinogenicity, vaccine research, etc.

You bring up FDA regulations, but then conflate the issue by stating "Note that these regulations exist in good part to eliminate as much as possible the influence of biases", when in fact these are strictly to try to prevent direct harm to patients by the treatment intervention under investigation.  Demonstrated by how they do nothing to prevent biases that don't strictly harm patients directly -- how else could the bias for over-reliance on pharmaceuticals stand?...  if these were, by design, to try to eliminate bias?

"But a more relevant question is “Are there examples of errors that science has allowed to stand for a significant period of time?”  A search for these kinds of examples might not be so difficult."   Then humor me.  The remedy, to fit the topic, must be an extra-scientific mechanism to eliminate bias that scientists are fundamentally otherwise unable to.  To best understand the beast being discussed, I would appreciate examples.  If it never existed, and never will, then that is the very definition of a red herring to discard results that are unwelcome and an excuse to disregard consequences of the status quo.  I would like to see this is a good faith argument.

I don't deny that the public demands more from climate scientists than protein folders.  But I want to see the distinction of the demand from the demands put on evolutionary biologists by members of the public, from the demands put on mathematicians by those who trisect the angle or square the circle, from the demands put on medical animal researchers by the property destruction and harassment by anti-vivisectionist protesters, etc.  Are the demands, for the most part, strictly to promote scientific truth found in the least time in the efficient way?  Or not?
Enhanced by Zemanta

Tuesday, June 15, 2010

From the Ladies of Reddit: how "nice" guys are usually "not nice"

Growing up, I had the mistaken belief that I was a "nice guy", and that my lack of luck with women was because "women don't like nice guys".
This Reddit posting hit home, and felt like a slap across my face.  It is absolutely true:

2X - Can I rant about this "Girls don't like us nice guys" shit? : TwoXChromosomes:
Once again, a 'girls are stupid for not liking us nice guys' link is #1 on the comics subreddit and #1 on my main reddit homepage.
These things piss me off.
Why? Because I like nice guys. I like shy introverts. I mean, I'm a geek girl myself, so yes, I heart nerds.
And despite all that it was nearly fucking impossible to get a date in high school or college because I'm (frankly) ugly. I mean, ugliness and shyness/introvertness/nerdiness normally go hand-in-hand, right? These guys were never handsome, but by my being just as unpretty as they were, I was invisible. I was unclean.
So every time a guy is bitching and moaning that girls aren't flocking all over them, I want to slap them, because I showed interest in his type, time and time again, and got rejected because I lost the facial genetic lottery, just like he did.
Stop fucking whining that girls suck for not looking past your shy awkward exterior and seeing you for who you are on the inside; you never gave me that courtesy either.
[another commenter]
AHHH, I just read this from /r/bestof and I was hoping someone would have posted about it on 2XC.
Some of the comments (like that one) just don't get it. Women like nice guys.
The problem is that most people who think they're "nice guys" just aren't. In many respects, they're just as shallow as all the promiscuous "assholes" they detest: they'd fuck around if they could, and they put just as much weight on looks as anyone else. The main difference between "assholes" and "nice guys" is that the "nice guys" a usually a combination of: uninteresting, awkward, uncharismatic, not confident, not relatable.
The worst part is that they're (inherently!) jerks. Women actually tend to like nice men! If you, Mr. "Nice Guy," are going to assume that they don't, then that's just insulting and presumptuous. It's your way of saying, "women are dumb enough to not like men who are genuinely nice people." With a presumptuous attitude like that, it's no wonder that these "nice guys" don't tend to do so well.
I think it's a matter of evolutionary psychology. Evolutionarily speaking, it makes sense for men to detest the guy who gets women, hence guys tend to call promiscuous men "assholes." But the truth is, most of these assholes simply aren't assholes. They tend to be fun, interesting, sociable, and kind of nice people. Yes they're promiscuous, but the majority of men would be if they could. Not all of them, but the majority of them. (Men literally do have a larger sex drive than women.)
A rule of thumb: If you think you're being "nice," then you're actually being a jerk. If you think you're "the nicest person in the world, holy shit I am putting my heart and soul into pleasing everyone and not being annoying," then it means you're being kind of nice.
There's a lot of other things I could say... but I feel it would all come off as extremely presumptuous.
[another commenter]
The problem is that most people who think they're "nice guys" just aren't. In many respects, they're just as shallow as all the promiscuous "assholes" they detest: they'd fuck around if they could, and they put just as much weight on looks as anyone else. The main difference between "assholes" and "nice guys" is that the "nice guys" a usually a combination of: uninteresting, awkward, uncharismatic, not confident, not relatable.
Totally agree with all of this, except for the last sentence. Let me fix it for you...
The main difference between "assholes" and "nice guys" is that the "nice guys" expect that doing nice things for hot women alone should make the hot woman attracted to him, they think that "being nice" ought to be the only quality necessary to entitle them to a hot girl of their own.
And this is absolutely, something totally different from actually being a genuine nice guy. Genuine nice guys are nice because they are actually nice, not because they want to get something out of it. By contrast, the nice guys we are talking about are usually just as asshole-ish as the rest of us, except when in the presence of a hot girl that they want. That's the only time that they feel any real need to be any nicer than usual.
The problem is when you are nice as a manipulative move to get something. That's not truly nice. Truly nice is when you are kind for the benevolent sake of helping your fellow man, with no expectation to receive in return. If that's not what is motivating you, then you aren't a nice guy, and stop describing yourself as one just because you keep offering your shoulder for hot girls to cry on.
[another commenter]
I think a good rule of thumb is to try to judge how nice you are being based on how you treat someone you are not attacted to. If you are only nice to girls you lust after, you are not a nice guy.

Shamefully, as a young man, I was guilty of exactly this.  At least I have a mature viewpoint to offer my own daughter.
Enhanced by Zemanta

A juicy example of The Art of Controversy

God bless the true skeptics - they exist, and they hold themselves and their allies to high standards.

This post was interesting to me for this wonderful technique of The Art of Controversy:

So here's a pea-and-thimble [shell game] strategy one might try out in a discussion:

- When confronted with general, collegial, and interpersonal criticisms, look and ask for specifics;
- When being responded to on specifics, return to the general mode, but on another subject, in a more collegial and interpersonal voice.
The reader can easily see that it creates an unwinnable position for the opponent.

From Willard - Very good catch of the specific technique of "The Art of Controversy" -- ducking and weaving between the general and specific. Thank you Willard.

A reminder to myself that my general point and my specific point should skip down the lane holding hands, so I don't fool _myself_ with the same technique (skipping down the lane being a metaphor for presenting an argument).

Also, a good technique for speaking about morality.  Have the specific moral action followed immediately by the general moral concept.  Have the grand abstract moral scheme followed immediately by suggested congruent specific real-world moral actions.
Enhanced by Zemanta

Friday, June 4, 2010

Hierarchies of Concern... and avoiding being the Boy who Cried Wolf

Maslow's Hierarchy of Needs. Resized, renamed,...Image via Wikipedia

Michael Tobis has been blogging lately about possible hysteria about the BP Gulf oil spill.  With the context being that (possibly) falsely predicting catastrophe with the BP Gulf oil spill may lead people to think that global climate disruption will also turn out to be less severe than the scientists are predicting.

Mega-Disaster in the Gulf? Or Not?
Horizon - Backing Off Optimism
Reality vs Model

> But I'm starting to see the problem that many people have with others' eagerness toward predictions of catastrophe.

I wish there was an easy way to talk about and visualize a hierarchy of concerns. Capable humans naturally understand a hierarchy of concerns, where lower concerns only come into play once higher concerns are reasonably satisfied (like my concern for hydration is below my concern for respiration, and both of these are above my concern for oil-free aquatic bird life). And sibling concerns may be substitutable or complementary and have a ranking or command a fraction of a whole.

(This is like Maslow's hierarchy of needs ['s_hierarchy_of_needs], but more individually specific and more detailed.  Also, it is inverted: the "base" concerts are at the top.  Also, it is not entirely clear that "self-actualization" is at the extreme: I see global/group concerns as being predicated on successful self-actualization.  A hierarchy of concerns could have no set limit on levels deep.)

The mainstream media handles concerns like a preschooler on a sugar rush - one big shiny new concern can push out all others, irrationally and ineffectively, rendering the news consumer incapable.

Maybe a little web widget that displays the hierarchy of concern, interactively. A hierarchy, running from the mundane but individualistic corporally viscerally urgent all the way down to the global political abstract. And is easily updatable with new information. And tools to meaningfully compare between hierarchies, of different people or the same at different times.

And tools to draw attention, to yourself (and embarrassingly to others), that inconsistencies exist (I am much more interested in finding out inconsistencies and conflicts between my concerns than I am in maintaining the illusion of perfect consistency from yesterday to today and the illusion of omniscience). Because mundane actions have global political abstract consequences, and the global political abstract imply certain mundane actions, and it is tempting to conveniently compartmentalize these two extremes, and conveniently ignore contradictions and inconsistencies.

Getting back to lacking models leading to overstating the potential for catastrophe -- it is hard to put appropriate demands on models without the context of this hierarchy of concern. Because the actionable decisions based on predictions of the models is a function of the demands put on the model during construction and manipulation and evaluation.

Reblog this post [with Zemanta]

Tuesday, June 1, 2010

Israel attack Turkish flotilla - When the sane leave, the insane double down

Flag of Israel with the Mediterranean sea in t...Image via Wikipedia
The "new, improved" Andrew Sullivan quotes Stratfor's George Friedman:

Andrew Sullivan - The Daily Dish - Israel Meets Its Own Fist

Stratfor's George Friedman, not a natural antagonist to the Jewish state, wonders if this could be a turning point against the Netanyahu-directed assisted suicide of Israel. After all, Israel opened fire on a ship from a NATO member, Turkey.  Should NATO treat this as an attack on every member nation?
The tougher Israel is, the more the flotilla’s narrative takes hold. As the Zionists knew in 1947 and the Palestinians are learning, controlling public opinion requires subtlety, a selective narrative and cynicism. As they also knew, losing the battle can be catastrophic. It cost Britain the Mandate and allowed Israel to survive. Israel’s enemies are now turning the tables. This maneuver was far more effective than suicide bombings or the Intifada in challenging Israel’s public perception and therefore its geopolitical position (though if the Palestinians return to some of their more distasteful tactics like suicide bombing, the Turkish strategy of portraying Israel as the instigator of violence will be undermined).
Israel is now in uncharted waters. It does not know how to respond. It is not clear that the Palestinians know how to take full advantage of the situation, either. But even so, this places the battle on a new field, far more fluid and uncontrollable than what went before. The next steps will involve calls for sanctions against Israel. The Israeli threats against Iran will be seen in a different context, and Israeli portrayal of Iran will hold less sway over the world."
Note that the flag on that ship was Turkey, a NATO member. Will Turkey demand invocation of NATO's Article 5?
And this will cause a political crisis in Israel. If this government survives, then Israel is locked into a course that gives it freedom of action but international isolation. If the government falls, then Israel enters a period of domestic uncertainty. In either case, the flotilla achieved its strategic mission. It got Israel to take violent action against it. In doing so, Israel ran into its own fist.
My own comments...
Why is Israel becoming so ham-handed of late?  The Dubai Hamas assassination was childishly carried out, so badly that todays Mossad was shown to be a shadow of its former self.  And now this.  This was a gift from Israel to its worst enemies.
The more intelligent Israelis must have moved out of participation with armed response to Palestine/Hamas rocket attacks, leaving numb-nuts in charge, who are screwing things up.  The more intelligent Israelis must have moved out of participation with armed response because they could see it only spiraling out of control, to Israel's doom.  But the majority of Israeli voters are too fearful to elect anyone except the right wing - a right wing now full of idiots without check.
This is all at the worst possible time, when Israel's protector, the United States, no longer can work unilaterally, or even keep up the illusion.
Reblog this post [with Zemanta]