I now pronounce you nub & roofline

I now pronounce you nub & roofline

50 States of McMansion Hell: Fairfax and Loudoun County, Virginia

Howdy folks! This post has been months in the making. Scouring the hell that is the McMansion Trenches of Virginia for only the best (worst) houses for your viewing pleasure generated some truly awful contenders. Of all the counties in Virginia, it was the wealthiest DC suburban counties of Fairfax and Loudoun that delivered. I won’t leave you hanging longer than I already have, so without further ado, the countdown:

#10: The Trellis Terror (Loudoun County)

The scrunched up piles of roof and narrow windows are a casualty of trying to squeeze the biggest possible house complete with not one but two garages into the smallest imaginable lot. The second story over-the-garage trellis aims to invoke the Tuscan countryside, but ends up looking like a bad strip mall Olive Garden instead. 

#9: Tricorn Turret (Loudoun County)

The consistency of cladding materials and window shapes make this house more well put-together than most McMansions. However it made the list for obvious reasons: a substantial and precipitous roofline, a rare triple turret dormer assembly, and that bizarre skeletal stone porch thing transform this house from country estate to ridiculous Hummer house. 

#8: Fort Void (Loudoun County)

Usually the problem of McMansions is too many large windows, in this case it’s too many small windows, all of them different from one another as if this house was just a front for the Pella Window showroom. The monotonous brick swallows the windows giving the house a fortress-y aura. The juxtaposition of pastoral rolling farmland with an equally ugly house right next door is particularly choice. 

#7: Mt. Nub’s Revenge (Loudoun County)

This house is a perfect example of how, even when they try really really hard, McMansions are incapable of symmetry. The more you look at this house the more “spot the difference” elements you find: the weird short colonnade vs the five-bay picture window; the length of the two wings, the roofline of the right wing is for some reason broken up because God is dead. And then there’s that nub. 

#6: Sticker Shock (Loudoun County)

This robust residence is absolutely chaotic. No two gables are the same. Stone is applied liberally and without logical consistency. Gutters trail down columns and crevices. Every window antagonizes its neighbor. The only thing over which any control has been exerted is nature itself, repressed and dominated by a monocultural expanse of grass. Normally I am not so blunt, but I will be today: I hate this house. 

#5: Chonky Corinthian

There is a certain type of house that is very popular in Fairfax County. It consists of a hulking range of hipped roofs punctured by a central (?) portico supported by columns that can only be described as thicc. This is one of these houses. The people who built this house could not decide when they were done building it. One can only assume that the myriad plans for this house were saved with file names like “House″ “House 2″ “House 2 final” “House 2 final final” “House 2 FINAL FINAL FOR REAL THIS TIME” 

#4: Mad Hatter (Fairfax County)

First of all, this home is way too long. It just keeps going. It’s like six different houses stitched together. Roofs begin and end. Porches come and go. Two stories somehow transform into one. By the time the eye reaches the front entrance, one is already exhausted. Finally, whoever decided to take the phrase “nesting gables” and apply it in this way deserves a trial at the Hague. 

#3: Tragic “Tudor” (Fairfax County)

This is the house equivalent of an identity crisis. Elements of French, English, and Donald Trump commingle to produce a truly formidable facade. All of the landscaping choices in this post are sad, but this house takes the cake for most depressing scenery, and not just because it was photographed in winter. Stubby shrubs appear to be gasping for breath, what trees exist are mere, unstable sticks; beside the pergola, a fallen cypress. 

#2: Foaming at the Mouth (Fairfax County)

This is a classic McMansion: it does its best to look dignified and imposing and instead appears cartoonish and cheap. Every element of this house except perhaps the wooden door is derived from petroleum products. The massive transom screams “climate denialism.” The grand entrance is overdone and top-heavy to the point of parody. In short: I hope this house melts. 

#1: Brick Behemoth 

If you combine all of the insipid elements of the other houses: mismatched windows; massive, chaotic rooflines; weird asphalt donut landscaping; pompous entrances, and tacked on masses; you’d get this house. The more one looks at this house the more upsetting it becomes. The turrets don’t match. The roofline is truly mountainous. The windows are either too small or too big for the walls they are housed in. The carhole is especially car hole-y. What sends this one over the top is its surroundings: lush trees and clear skies that have been desecrated in order to build absolute garbage. At least it doesn’t have shutters. 

Well, that’s it for Virginia! Stay tuned for another installment of “The Brutalism Post” - this time about what Brutalism actually is. 

If you like this post, and want to see more like it, consider supporting me on Patreon!

There is a whole new slate of Patreon rewards, including Good House of the Week, Crowdcast streaming, and bonus essays!

Not into recurring donations or bonus content? Consider the tip jar! Or, Check out the McMansion Hell Store! Proceeds from the store help protect great buildings from the wrecking ball.

Copyright Disclaimer: All photographs are used in this post under fair use for the purposes of education, satire, and parody, consistent with 17 USC §107. Manipulated photos are considered derivative work and are Copyright © 2019 McMansion Hell. Please email kate@mcmansionhell.com before using these images on another site. (am v chill about this)

Class of 2020: How Kate Wagner Is Making Everyone Question Everything About DesignHowdy!! I’m...

Howdy!! I’m very pleased to announce that Apartment Therapy has included me in their Class of 2020 Design Change Makers!

The Brutalism Post, Part 2: What Brutalism is Not

Why open a series about Brutalism by discussing what is not Brutalism? The answer is simple: of all of the terms in the history of architecture, Brutalism is perhaps the most misused and misunderstood by the general public. 

Pictured: Citizens Bank Tower, 1958-66. Not Brutalism (it’s just plain ol’ International Style modernism). Source

The main issue here, as we will discuss later, is not that people are ignorant for using the wrong definition of the word Brutalism, but that the word Brutalism has come to mean or refer to a variety of architectural phenomena that are linked to one another via overarching similarities, the most important being an expanded set of buildings that elicit a specific emotional response in the viewer.  

“Brutalism”, a specific architectural movement with its own ideology and history, has come to encompass a wide range of colloquial meanings. Some of these meanings are common misconceptions that reflect a need for broader architectural education (the purpose of this series), and some of these meanings reflect a deeper, more philosophical, interrogation into how we perceive and discuss architecture and the complex emotions it arouses within us, exposing a need for new means of communicating a common architectural sentiment

Let’s start with the most common misuse of the term. 

Brutalism is not: every single building made primarily of reinforced concrete. 

Blame this one on the dictionary. The term Brutalism, while being derived from the French term beton brut, meaning raw concrete, does not apply to all buildings made from reinforced concrete. Developed in the 1870s, reinforced concrete is one of the most commonly used building materials in the world. Because of its inexpensive price, its structural integrity, and its ability to be cast into a variety of shapes and forms, many buildings were - and continue to be - made from it. 

Let’s look at three examples of buildings I have found labeled “brutalist” in various places. 

Not Brutalism: (from left to right): Tadao Ando, Vitra Conference Center (1993) Photo by Wojtek Gurak (CC BY NC 2.0); Le Corbusier, Villa Savoye (1929) Photo by Scarlet Green (CC BY 2.0); Albert Kahn, Highland Park Ford Factory (1910) Photo by Thomas Hawk (CC BY NC 2.0)

All of these buildings are constructed primarily from reinforced concrete. As you can see, they are all very, very different from one another. In these three cases, the key piece of information discrediting these buildings as being Brutalist is when they were built. Brutalism was a specific architectural movement from spanning a defined period of time (1940s-late 1970s). Buildings constructed outside of this time window are rather unlikely to be Brutalist

Let’s look at why these buildings might be mislabeled brutalist. 

Our first example is the Vitra Conference Center by Tadao Ando, which was built in 1993. Even though 1993 is far outside the time frame that brutalism spanned, this building has many characteristics that are “brutalistic,” specifically its extensive use of unpainted reinforced concrete, its heavy, geometric massing, and its intense visual weight. Ando’s architecture falls under the term “critical regionalism” - which is best understood as being modern in form (but not in dogma), with a heightened focus on the surrounding ecology and landscape as well as other geographical, cultural, and social contexts.  

The Villa Savoye by Le Corbusier, finished in 1929, is one of the most iconic 20th century houses and works of modernist architecture in the world. This house, though made of reinforced concrete, belongs to the movement known as the International Style, which was developed in Western Europe after World War I, is known for its rejection of ornament, flat surfaces (especially roofs), extensive use of glass, and visually lightweight and repetitive forms. While the International Style makes use of concrete, it differs from Brutalism in its visual lightness - the Villa Savoye seems to float effortlessly above the landscape - very unlike Brutalist architecture, which is characterized by its massive scale, hulking forms and visual heaviness. 

The Highland Park Ford Plant, built in 1910 by noted factory architect Albert Kahn, was once the premiere factory building in America, helping to advance not only the Fordist system, but the city of Detroit, Michigan as being the automobile capital of the world. It is a touchstone of factory design, notable for its pioneering use of the assembly line to facilitate mass production, a concept that remains central to factory design today. Although made of reinforced concrete, the Highland Park plant is not a brutalist building. It frequently is mischaracterized as being brutalist because of its massive side, imposing features, and the close association that has developed between brutalist architecture and urban exploration photography (More on this later). 

TL;DR: All brutalist buildings are made of reinforced concrete (or heavy masonry), but not all reinforced concrete buildings are brutalist. Moving on.

Brutalism is not a catch-all term for Late Modernist architecture

Architecture got so weird and complicated in the period from the 1960s through the early 1980s that it inspired the architectural theorist Charles Jencks to create the first of several delightful and mind-bending charts to try and categorize it: 

Yeah. 

What is Late Modernism? The concise definition is that it is an umbrella term encompassing the various architectural movements that transpired after International Style/Mid-Century Modernism but before Postmodernism. (For more on what Late Modernism is and why you should care, see my post from 2016.) Brutalism elides with Late Modernism, but not all Late Modern buildings are Brutalist. Because Brutalism is contemporaneous with Late Modernism, the distinction can be confusing. Often the case is that Late Modern buildings that are described as ‘Brutalist’ should be recategorized or reassigned to a different, equally obscure and hyper-specific architectural sub-movement happening around the same time. This might seem nitpicky, but look on the bright side: now you get to correct your friends. 

Late Modernism encompassed a lot of smaller architectural movements, most, but not all of them ending in -ism. Some, like Brutalism and High Tech, are more well known; others, like Metabolism, Structuralism, Critical Realism, and Neo-expressionism, not so much. Some buildings don’t fit into any of these categories and must (frustratingly) be referred to as simply “Late Modern” or “Transitional” (referring to the transition from Modernism to Postmodernism.) 

Here are three Late Modern buildings that are not Brutalist:

Left: Richard Rogers, Lloyd’s Building (1986) Photo by Lloyd’s Insurance (CC BY 2.5); Top Right: Kisho Kurakawa Nakagin Capsule Tower (1972) Photo by scarletgreen (CC BY 2.0); Bottom Right: Herman Hertzberger, Centraal Baheer (1972) Photo by Apdency (CC BY-SA 3.0) 

Lloyd’s Building, the headquarters of Lloyd’s of London, located in, unsurprisingly, London, was designed by Richard Rogers and completed in 1986. Despite the relative lack of reinforced concrete, the building is frequently categorized as being Brutalist. The fact that it lacks reinforced concrete as a defining architectural feature is all one needs to eliminate Lloyd’s from the Brutalism category - why it is put there in the first place we will discuss more in depth in the next section of this post. Lloyd’s - along with most of Rogers’ work - is part of the architectural movement known as “High Tech” because it is, well, High Tech. 

High Tech buildings are the apogee of the modernist mindset in terms of glorifying the functions of a building and the technological elements of structural engineering. They take what are usually internal systems such as structural frames, circulation systems (such as stairs and elevators) and services (electrical, plumbing, etc) and integrate them into their external architectural form. (Lloyd’s is colloquially known by Londoners as the “inside out building”). High Tech was relatively short lived because it turns out that when you decorate the outside of your building with its internal services, when winter comes, your water pipes, exposed to the elements, tend to freeze. 

The Nakagin Capsule Tower, built in 1972 by Japanese architect Kisho Kurakawa (one of my favorite architects ever who more people should know about), is one of the buildings most commonly labeled as Brutalist. This building illustrates the gray area that arises when one uses vague aesthetic attributes (concrete, visually heavy, geometric massing) to designate a building as Brutalist instead of the actual history and context of the building in question. The Nakagin Capsule Tower belongs to a different (if coexistent) architectural movement that, frankly is a lot weirder than Brutalism: Metabolism. Take the formal concept of organic biological growth and systems and combine it with the urbanistic concept of megastructures (an entire city contained in a single continuous structure or via several interconnecting structures) and you get Metabolism. Because of the practical issues with building an entire city within a single building, Metabolism lived mostly on paper, however a few built examples were executed, the most famous being the Nakagin Capsule Tower. 

The Centraal Baheer office building was built by Dutch architect Herman Hertzberger in 1972. Like the Nakagin Capsule Tower, it satisfies many of the aesthetic signifiers commonly attributed to Brutalism: it’s made of reinforced concrete, composed of large geometric massing, and it’s visually heavy. Also like the Nakagin Capsule Tower, it belongs to a different, coexisting architectural movement, primarily developed by the Dutch, called Structuralism. Structuralism is a complex set of architectural ideologies developed in the 1960s and 70s, centered around a few key concepts: the rationalist idea that people’s behavior can be directly changed (or manipulated) via design; designing built structures that correspond in form to social structures; an emphasis on cultural and geographical context; an urbanism and design approach based, like Metabolism, on a biological growth analogy (called Aesthetics of Number); the integration of a variety of uses and programs within the same overall structure; and, finally, the aim to architecturally reconcile the needs of both “high” and “low” culture. 

Brutalism, Metabolism, and Structuralism arose from a similar origin, and are ideologically more alike than different, something we will talk about in the next installment of this series. 

Brutalism is not a feeling. 

But also, it kind of is. It is, as the folks say, “a big mood.” A large reason why buildings are incorrectly labeled Brutalist is because they bring forth a very specific emotional response to architecture shared by many people across the world. Some of the buildings that cause people to feel this complex and nuanced set of emotional and aesthetic reactions are, in fact, Brutalist, but many are not. To me, what this demonstrates, is a broader need for architectural education and discourse that goes beyond the most common system for classifying architecture: stylistic labels. 

To talk about this, we’ll bring back Lloyd’s Building, Kahn’s factory and present it alongside a few other examples. 

Left to Right: John M Johanson, Stage Center Oklahoma City (1979) Photographer unknown; Yuri Platonov, Russian Academy of Sciences (1968) Photo by Raita (CC BY 2.0); Boris Magasto, Haludovo Hotel, Krk, Croatia (1972) Photographer unknown; Kevin Roche, The Pyramids (1972) Photo by jikatu (CC BY SA 2.0)

All of these buildings (and all of the photographs of these buildings) are very different from one another, and yet, they have all been classified mistakenly as being “Brutalism.” The only real link between them is emotion. 

Like many folks in the late aughts/early 2010s, I nurtured my then-juvenile love of architecture through spending hours lurking in the Skyscraper City forums looking at thread after thread of pictures of 20th century architecture. Why? Because those images made me feel powerful emotions that I still find difficult to put into words. 

When talking about Brutalism as a feeling, perhaps the closest idea comes from the English philosopher Edmund Burke in his 1756 treatise “A Philosophical Enquiry into the Origin of our Ideas of the Sublime and Beautiful”. I am, of course, talking about the sublime. The Burkean sublime is emotionally complex. To quote Burke directly: 

“Whatever is fitted in any sort to excite the ideas of pain, and danger, that is to say, whatever is in any sort terrible, or is conversant about terrible objects, or operates in a manner analogous to terror, is a source of the sublime; that is, it is productive of the strongest emotion which the mind is capable of feeling.”

But the sublime isn’t just negative. It overwhelms us with its awesome power and in this moment, “the mind is so entirely filled with its object, that it cannot entertain another.” Burke’s concept of the sublime was initially applied to such things as the ocean or the Alps - natural features that are so large, massive, and inherently dangerous that they put us in a state of awe-inspiring disbelief - and yet, and despite their mass and their danger, they give us feelings of deep pleasure and joy

To the people (including myself) who love Brutalism - it does engender feelings of unknowing, of mystery, and sometimes, especially when said Brutalist building is in disrepair or photographed at a particularly menacing angle, of fear or grief. It shares this, rather than a stylistic label, with the buildings featured in this post. 

Because of Brutalism’s association with the State, such as in the case of the former Soviet Bloc, East Germany, the welfare state in England, or its use in governmental buildings around the world, lingering political sentiments can also contribute to this complex mix of emotions - whether one longs for the halcyon days of eras past or fears them as being domineering or totalitarian. It can also cause people to associate buildings that are not Brutalist with buildings that are because they share a same political history. Similar to how the post-industrial society left behind a trail of industrial ruins along the American Rust Belt, so too has neoliberalism gutted and left for dead the monuments of these modernist utopias. 

An actual Brutalist building: Paul Rudolph, Endo Laboratories Headquarters (1964). Photo via Library of Congress

Brutalist or not, these are enigmatic buildings - their forms are strange and unusual, alien even; their contents and even their purposes remain mysterious. Their siting makes them seem either imposing relative to their surroundings or isolated and alone. There is something dark and lonely, sad and longing about them. They are beautiful, partially because of their striking, form-bending architecture, and partially because they once lived different lives in times so unlike ours. 

If you like this post, and want to see more like it, consider supporting me on Patreon! 

There is a whole new slate of Patreon rewards, including Good House of the Week, Crowdcast streaming, and bonus essays!

Not into recurring donations or bonus content? Consider the tip jar! Or, Check out the McMansion Hell Store! Proceeds from the store help protect great buildings from the wrecking ball. 

Copyright Disclaimer: All photographs are used in this post under fair use for the purposes of education, satire, and parody, consistent with 17 USC §107. Manipulated photos are considered derivative work and are Copyright © 2019 McMansion Hell. Please email kate@mcmansionhell.com before using these images on another site. (am v chill about this)

important announcement

i met the lawyer to my foyer
the nub to my roofline
the car to my carhole…
what i’m trying to say is i’m engaged now

edit: we will not be buying a mcmansion for obvious reasons but if you’re feeling the love tonight, you can donate a coffee to the kate wagner & partner tiny and modest wedding fund: www.ko-fi.com/mcmansionhell ! 

(i’m a writer and he’s a high school math teacher so we could use all the help we can get.)

50 States of McMansion Hell: Campbell County, Wyoming

Howdy, folks! It’s time for us to go home, (big, ugly) home on the range. That’s right: It’s time for Wyoming. Now, when I think Wyoming, I think of rolling hills, ranches, incredible landscapes, and also that book about cowgirls that Dick Cheney’s wife wrote.

On that note:

This beautiful 5 bed/4.5 bath 6,000 square foot house was built in 2002. It can be all yours for around $700,000 USD.

Let’s mosey on down through this here estate.

Divorce Lawyer Foyer/Living Room

Now, I’ve been an architecture enthusiast for about a decade of my short, two and a half decade life and never have I seen something as both absurd and patently useless as whatever the hell that thing separating the foyer and the living room is. Does it have a structural purpose? No. Does it have an aesthetic purpose? Also no.

Kitchen

(Running for president voice) “Folks, when I become president, I assure you that the injustice - the absolute tyranny -  that is the island stovetop/wall oven combo will finally–” (crowd cheering) “– and I mean finally, be put deep into the ground where it belongs.”

Dining Room

The aged bronze chandeliers must have been on ultra clearance. Personally, as someone who loves a good deal, I cannot shame them for this.

Master Bedroom

What I can’t determine is what they were going for with the sponge paint on the walls. My closest approximation is tree bark, in which case what we see before us in fact is deeply offensive to the trees, who are innocent and don’t deserved to be involved in any of this.

Master Bathroom

I don’t care how nostalgic we as a culture get for the late nineties/early aughts, sponge paint should never come back. One, it’s sponge paint. Two, it is ridiculously labor intensive and frankly we could all be spending our last few years on this still-habitable earth doing something more worthwhile with our time, such as going outside or falling in love or destroying our brains on the internet.

You are seriously not ready for this

When making this picture I realized I desperately need to log off.

Bathroom 2

Computer, enhance.

Ok but you can’t even read that in the shower. If you’re that desperate for prophetic, non-digital bathroom reading material, buy a bottle of Dr. Bronners or, like, a copy of War and Peace.

Basement

Poor Little Julie. :(

Well, that wraps up our interior. Now, for the final frontier, on the frontier:

Rear Exterior

Somehow this house looks more like it’s made out of cardboard than the many, many houses on this blog that also look like they’re made out of cardboard.

Anyways, that does it for Wyoming! Stay tuned for next week’s Brutalism Post, Part II: What Brutalism is Not. Have a great weekend, folks.

If you like this post, and want to see more like it, consider supporting me on Patreon, ESPECIALLY if you also like donoteat01′s content on YouTube, because we will be doing a livestream collab on Patreon on August 25th, 2019 from 8-10PM EST!!!

There is a whole new slate of Patreon rewards, including Good House of the Week, Crowdcast streaming, and bonus essays!

Not into recurring donations or bonus content? Consider the tip jar! Or,Check out the McMansion Hell Store ! 100% of the proceeds from the McMansion Hell store go to charity!

Copyright Disclaimer: All photographs are used in this post under fair use for the purposes of education, satire, and parody, consistent with 17 USC §107. Manipulated photos are considered derivative work and are Copyright © 2019 McMansion Hell. Please email kate@mcmansionhell.com before using these images on another site. (am v chill about this)

Kate Wagner - McMansion Hell Live at Motorco!

Kate Wagner - McMansion Hell Live at Motorco!:

Howdy folks! If you’re in the NC area next Tuesday, come and see me talk about McMansions as well as do a live house roast at Motorco in Durham. The ticket sales go to fund NC Modernist Houses, a nonprofit that works to preserve the heritage of architectural modernism in North Carolina. 

When: Tuesday, July 23, 6pm

Where: Motorco, 723 Rigsbee Avenue, Durham

Tickets: Advance - $29
At the door - $39
VIP - $99
(you get to throw axes at pictures of McMansions with me afterward lol)

See y’all there! 

The Brutalism Post, Part One: Introduction

This is part one of a five-part post about Brutalism. 

University of Massachusetts at Dartmouth campus by Paul Rudolph. Photo via Library of Congress

No style of architecture so passionately divides even the most good-natured and level-headed people as Brutalism. The discourse surrounding Brutalism being “good” or “bad” is fierce and polemical. The “for” crowd lobbies on both aesthetic grounds – posting pictures of incredible and obscure structures and saying “I mean LOOK at this” – as well as political ones, citing in particular, how Brutalism was used to house thousands of people during the postwar period. 

On the other hand, the “against” crowd brings up the failed urbanism of Le Corbusier that gave us the freeways and slum clearance that split and displaced entire swaths of city fabric, proclaiming that only architects or architecture enthusiasts like Brutalism, and that this is a testament to how out of touch they are with everyday people. “If you had to live or work in these buildings,” they say, “you’d feel differently.” 

Unité d'Habitation by Le Corbusier. Photo by Thomas Nemeskeri, via Flickr (CC BY-NC-ND 2.0) 

I’ve been a spectator to this debate since I first lurked in the Skyscraper City forums as a high school freshman, ten years ago, when Brutalism itself sparked the interest in architecture that brings me here today. I have, as they say, heard both sides, and when asked to pick one, my response is unsatisfying. Though my personal aesthetic tastes fall on the side of “Brutalism is good,” I think the actual answer is  it’s deeply, deeply complicated. 

Still, what is it about Brutalism that makes it so divisive? Why does a short-lived substyle of modern architecture elicit such vehement passion in so many people? What does it even mean for a style of architecture to be “good” or “bad”? You can see why I’m drawn to finally sitting down and penning this series, which has been simmering at the back of my mind since I started McMansion Hell three years ago. (By the way, Happy Birthday to this blog!!!) 

Brutalism has a special way of inspiring us to ask big and difficult questions about architecture. “Is Brutalism good?” is really a question of “is any kind of architecture good?” - is architecture itself good? And what do we mean by good? Are we talking about mere aesthetic merits? Or is it more whether or not a given work of architecture satisfies the purpose for which it was built? Can architecture be morally good? Is there a right or wrong way to make, or interpret, a building? 

Ferrier Estate, a now-demolished social housing complex in South London. Photo by Tim Slessor via Flickr (CC BY-NC 2.0)  

I have bad news for you: the answers to all of these questions are complicated, nuanced, and unsatisfying. In today’s polemical and deeply divided world of woke and cancelled, nuance has gotten a bad rap, having been frequently misused by those acting in bad faith to create blurred lines in situations where answers to questions of morality are, in reality, crystal clear. This is not my intention here. 

For centuries, the philosophical discipline of aesthetics has tried in vain to articulate some kind of clearly defined standard by which we can delineate whether or not a work of art is good, bad, moral or amoral. Architecture makes this even more complicated because unlike literature, painting, music, or art, we have to live, work, and exist in architecture. Not only does the question of whether or not we can separate the art from the artist exist in architecture, so to do questions of whether or not we can separate the building from the politics, from the culture, from the time period, from the urbanism, from the socioeconomic system, from the entire contents of everyday life in which it exists. 

Orange County Government Center, Perspective Drawing, by Paul Rudolph. Photo via the Library of Congress.

Existential questions aside, there are other reasons to write about Brutalism. First, while we’ve been hemming and hawing about it online, we’ve lost priceless examples of the style to either demolition or cannibalistic renovation, including Paul Rudolph’s elegant Orange County Government Center, Bertrand Goldberg’s dynamic Prentice Women’s Hospital in Chicago, and the iconic Trinity Square, Gateshead complex, famous for the role it played in the movie Get Carter. My hope is that by bringing up the nuances of Brutalism before a broad and diverse audience, other buildings on the chopping block might be spared. 

On an even broader note, I think Brutalism is worth writing about simply because a lot of people are rightly confused as to what it even is.  The common practice of identifying Brutalism by the presence of a material - reinforced concrete - too broadly defines a style that belongs to a specific era and architectural praxis. There are so many buildings and styles called Brutalist that are not Brutalist that I’ve devoted the first two installments of this series to the subject “What Brutalism Is Not,” followed, of course, by “What is Brutalism?” The goal is that these two essays will be educational and interesting (with the added bonus of providing the reader with an arsenal of information that will make them as insufferable at dinner parties as I am.)

The third part in this series is devoted to the people of Brutalism - the architects, politicians, planners, writers, and philosophers, who signed their names to an architectural movement that spanned the globe. Finally, the last installment gathers all this information together and answers the question we’ve all been waiting for: is Brutalism good? 

The Kyoto International Conference Center, designed by Sachio Otani. Photo by Chris Guy, via Flickr. (CC BY-NC-ND 2.0) 

This is a series on Brutalism, but Brutalism itself demands a level of inquiry that goes beyond defining a style. Really, this is a series about architecture, and its relationship to the world in which it exists. Architects, as workers, artists, and ideologues, may dream up a building on paper and, with the help of laborers, erect it in the material world, but this is only the first part of the story. The rest is written by us, the people who interact with architecture as shelter; as monetary, cultural, and political capital; as labor; as an art; and, most broadly, as that which makes up the backdrop of our beautiful, complicated human lives. 

If you like this post, and want to see more like it, consider supporting me on Patreon!

There is a whole new slate of Patreon rewards, including Good House of the Week, Crowdcast streaming, and bonus essays!

Not into recurring donations or bonus content? Consider the tip jar! Or,Check out the McMansion Hell Store ! 100% of the proceeds from the McMansion Hell store go to charity!

Copyright Disclaimer: All photographs are used in this post under fair use for the purposes of education, satire, and parody, consistent with 17 USC §107. Manipulated photos are considered derivative work and are Copyright © 2019 McMansion Hell. Please email kate@mcmansionhell.com before using these images on another site. (am v chill about this)

50 States of McMansion Hell: Top 10 Waukesha County, Wisconsin McMansions

Howdy Folks! We’re continuing our out-of-order-for-dramatic-effect tour of the 50 States of McMansion Hell today with perhaps one of the most underrated McMansion counties in the country: Waukesha County, Wisconsin. These houses were so bizarre it was hard to choose just one to do a takedown of. So, without further ado… 

#10: Doom McGloom

This 2002 estate, thanks to the clever machinations of whoever took these photographs, looks less like an enticing investment property and more like a prime candidate for the Chernobyl ripoff set in America that has 2 stars and is only available on Amazon Prime. 

#9: Headquarters of Tree-Haters Anonymous

This 2004 manse is $1.4 million dollars and yet its creators couldn’t afford more than a single (invasive!) tree. I don’t know what kind of sociopath wakes up in the morning and actively hates everything taller than a malnourished shrub. Whoever they are, this is certainly the house for them. 

#8: Roofer’s Paradise 

A post-recession 2011 McMansion, this house clearly didn’t learn anything from the recent past. With many McMansions, I can conceive of ways to improve them to make them better. With this house, I simply do not know how to rectify its main problem: it’s, like, 90% roof. In my head I refer to houses like this as “turtle houses” but frankly this does a disservice to the noble turtle. 

#7: Haunted Geometry

This house was built in 2014, a time when people should definitely have known better. Its inclusion in this list is solely due to the absolutely bizarre geometry of its roof, a kind of geometry formerly unknown to mathematics until this time. Bonus points for the continued animosity to trees found in the wealthy populous of this county. 

#6: McEscher

Nothing about this house makes sense. I’m serious. I’ve looked at it from several different angles and have yet to perceive any coherent spatial logic to how it comes together. This is house is an SCP. It’s an X-Files case. House of Leaves was actually based on this house. It’s an Escherian nightmare. 0/10 would not go inside even if you paid me. 

#5: Obligatory Beigehaus

You know when a bad stand up comedian tells a joke that just keeps going way too long? The audience is like, okay, we get it, you need therapy, but he (and it’s always a he) just keeps going on and on. Well, this is the house equivalent of that. 

#4: House of Lumps

Whoever built this house was utterly incapable of picturing in their minds eye what a house should look like. The very conception of a house is foreign to them. They have never seen a children’s book with houses in it. They probably didn’t even have a childhood. 

#3: Play-doh Playhouse

This house made it so far in the countdown because it is, frankly, weird. I don’t know why it is painted the color of jaundice, or why they have transformed every gable into a hollow cavity longing for death. Lots of things are happening here, though none of them could appropriately be called “architecture.”

#2: Farmhouse Freak

Let your eyes glaze over as you look at this “farmhouse” - the more you look at it the less sense it makes. What are they farming, you ask? Why, turf grass of course! Bonus points for this image in which the house appears through a haze of ozone or something. 

And now, the moment you’ve been waiting for…

#1: Corinthian Catastrophe

It’s one thing to have oversized Corinthian columns on your absurd McManse, but it’s a whole new level of extra to spray paint the capitals gold. This house takes all the elements found in the other houses (treeless sociopathy, turret lust, garish mismatched windows, foam) and ramps it up to 11, which is why it earns the number one slot in the county. Also, as a bonus, I find it incredibly funny that they embossed the letter “C” everywhere. I guess whoever buys it either has to have a name starting with C or has their work cut out for them. The C represents the grade they got in home design class. 

Anyways, that’s it for Wisconsin, folks! Stay tuned for a special essay on whether or not brutalism is good, as well as the next installment of the 50 States: Wyoming. Have a great weekend. 

If you like this post, and want to see more like it, consider supporting me on Patreon!

There is a whole new slate of Patreon rewards, including Good House of the Week, Crowdcast streaming, and bonus essays!

Not into recurring donations or bonus content? Consider the tip jar! Or,Check out the McMansion Hell Store ! 100% of the proceeds from the McMansion Hell store go to charity!

Copyright Disclaimer: All photographs are used in this post under fair use for the purposes of education, satire, and parody, consistent with 17 USC §107. Manipulated photos are considered derivative work and are Copyright © 2019 McMansion Hell. Please email kate@mcmansionhell.com before using these images on another site. (am v chill about this)

McMansion Hell Cross Stitch Patterns

Howdy folks! My internet has been funny this week, thus, I had a lot of time to myself with no internet which invariably leads to…crafting. I made some McMansion Hell-themed cross stitch patterns in MacStitch and wanted to share them with all of you in case there are other folks out there who love to stitch. 

These pictures are just the previews - to download the full patterns, click here

Happy Stitching! See you soon with Wisconsin’s McMansion! 

Why Marie Kondo’s method is ideal for my ADHD

Why Marie Kondo’s method is ideal for my ADHD:

I got to write about how Marie Kondo helped with my ADHD! 

Frank Lloyd Wright Quarterly | Frank Lloyd Wright Foundation

Frank Lloyd Wright Quarterly | Frank Lloyd Wright Foundation:

Art by Ellen Surrey

Howdy! The lovely folks at the Frank Lloyd Wright Foundation let me take the reigns as guest-editor for the new edition of their quarterly, about FLW and pop culture! It was a great opportunity for me to revisit my love of Wright’s work, to invite some of my favorite architecture writers around to share their talents, and to draw arrows and text on top of a very famous and handsome building instead of the ugliest and dumbest houses imaginable. 

Topics include why FLW’s work and life remain so influential after all these years; a reverse McMansion Hell roast of Taliesin West, and a look at FLW in tv (including GoT!!), movies, and the entire genre of science fiction. (There’s also some really cute drawings of FLW sitting on the Iron Throne, cavorting with robots, etc.)

Stay tuned for the next installment of this blog, which will drop Thursday! 

I wasted a whole day making this chart for architects and so now you have to see it too

I wasted a whole day making this chart for architects and so now you have to see it too

interior design is my passion

interior design is my passion

Kate Wagneri made a website for myself (kate wagner)

i made a website for myself (kate wagner)

Tottenham 3-0 Burnley: Son

By /u/ennuihenry15

submitted by /u/ennuihenry15 to r/soccer
[link] [comments]

Tornado decimates school gymnasium

By /u/hjalmar111

Tornado decimates school gymnasium submitted by /u/hjalmar111 to r/ThatsInsane
[link] [comments]

This guy truly understands “mind over matter”

By /u/I_Am_Err00r

This guy truly understands “mind over matter” submitted by /u/I_Am_Err00r to r/Unexpected
[link] [comments]

Stop-motion animation using wool

By /u/eternalrefuge86

Stop-motion animation using wool submitted by /u/eternalrefuge86 to r/interestingasfuck
[link] [comments]

LAW

By /u/antoshkalebedev2018

LAW submitted by /u/antoshkalebedev2018 to r/PoliticalHumor
[link] [comments]

Trump complains about light bulbs making him look orange and people flushing toilets 15 times in rambling monologue. 'It doesn't make you look as good... being a vain person, that's very important to me'.

By /u/chelsea707

Trump complains about light bulbs making him look orange and people flushing toilets 15 times in rambling monologue. 'It doesn't make you look as good... being a vain person, that's very important to me'. submitted by /u/chelsea707 to r/politics
[link] [comments]

Future World leaders are probably going to have embarrassing videos on the internet of them playing lets play and trying to be youtubers

By /u/hitlers-sister

submitted by /u/hitlers-sister to r/Showerthoughts
[link] [comments]

The expression reveals the suckiness

By /u/superfrelon

The expression reveals the suckiness submitted by /u/superfrelon to r/Wellthatsucks
[link] [comments]

Thinking outside the box

By /u/Palifaith

Thinking outside the box submitted by /u/Palifaith to r/gifs
[link] [comments]

The American style of tipping is a toxic culture that should be abolished

By /u/Niz99

Now, before I continue, I would like to say that I ain't American, but I have read about how tipping works there from reddit. Thus, according to Internet law, I'm an expert on the matter. (In all seriousness though, just correct me if I'm wrong)

  • First of all, you guys say that tipping is necessary in order for workers to earn above minimum wage. However, isn't it a rule in most establishments that if the employee doesn't make enough through tips, their boss should cover the rest? What do you know, apparently untipped waiters aren't suffering from lack of cash. You might say that if tipping is abolished, the cost of food in restaurants will rise. So what? You are already paying extra through your bloody tips, just divert that cash to a higher price of a meal. Just make things a bit more simple.

  • Apparently, there is also cases of employers 'stealing' their workers tips or forcing them to 'share' it. Like how certain restaurants force their waiters to equally share their hard earned tips with their fellow workers. What the fuck is that? So if I work hard, treat my client extra super special awesome and received a glorious 50% percent tip for my hard work, I must share it with lazy Harry who sleeps in the storeroom every chance he get? And don't tell me that waiters have to share their tips with the cooks for the sake of 'fairness'. After all, don't cooks earn above the min wage?

  • Unlike most people apparently, I love maths. Damn, how I loved it, and I was pretty good at it. Nevertheless, it doesn't mean that I want to do a freaking mental calculation everytime I finish a meal. Like wtf people, I wanna chill after I finish dinner. And don't you people start giving me pointers on simple ways to calculate how much to tip a person. I find it twisted that tipping is so ingrained in American culture that you guys make a literal rule for it. Japanese student bow to their teachers respectfully before starting class, but any American asshole who gives below 20% for a tip is the scum of the earth and is a reserved a special VIP room in hell.

  • Plus, tipping culture apparently breeds a sense of entitlement among some in the service industry. I've read stories about waiters and waitresses who turn their nose away from what they presume is a 'low' tip and even 'subtly' hint that they should get a higher amount. Like bitch, you are literally getting money from someone whom isn't entitled to pay you squat and you got the audacity to be choosy about it.

  • Why only waiters get to be tipped? Why not the janitors, the maids or the garbage men? Some of them have even tougher jobs then waiters. They don't get tipped. I mean, when have you guys even said thank you to the guys who scrubbed your toilet bowl squeaky clean? Not to mention how some waiter may be lucky enough to earn double or triple above the minimum wage if they work at a particularly affluent restaurant. A bit unfair isn't it?

  • I feel like the easiest way to solve this problem is for the American legislation to get amended so that the workers in the service industry get paid according to the minimum wage. So why are you all so stupid that you can't figure out this simple solution?

  • Now, I'm not saying that tipping is bad. However, it should be used to show appreciation rather than being something mandatory. Hell, some restaurants even include tipping in the bill as a gratuitous fee. How presumptuous is that? Crap, I'm straying of point. Anyway, like I was saying, tipping should be a choice on the customer's behalf, to pay for some particularly outstanding service from their waiter. It shouldn't be freaking corrupted by being made socially compulsory.

You guys might be wondering why the hell I decided to post this. I don't live in America, nor do I plan to. This has nothing to do with me whatsoever, so why am I being a busybody? Well, it's because this annoys me. It's really, really annoying on how this whole issue is so stupid and unnecessarily overcomplicated. Of course, this ain't the only time America decided to be 'different' in order to separate themselves from the rest of the world only to end up looking like idiots. Remember the imperial system crap? Similarly, this whole issue is such a big pain in the ass for everybody that I internally cringe everytime I think about. So, I decided to vent about it by bashing the Americans. That's all.

Edit 1: Most of you talk about how tipping is a good thing because servers get more money than if they totally depend on min wage. That working as a server is so horrible that nobody will do it if not for the tips. Well, guess what? There are other shitty jobs out there and they don't get tipped for it so this seems pretty unfair to me. Basically, servers want this culture to stay, despite they getting less than minimum, because its better for them compared to the customers.

Edit 2: What's up with Europe and shitty service? You guys keep talking about how tipping incentives servers in America and you use your experience with Europe to prove that. This is not a thing in Southeast Asia, where I actually live, cuz people provide good service here WITHOUT EXPECTING TIPS. You Westerners are seriously dissapointing me here.

Edit 3: My god, you servers are salty af clinging to your precious tips just because you want to enjoy a lavish lifestyle that other minimum wage workers don't enjoy. You guys are a bunch of whiny entitled crybabies.

Edit 4: One thing I feel like I have to remind you guys is that I'm not hating on tipping. Instead I feel that we should only tip when we get great service, not average service. Unfortunately, though its purely voluntary, Americans are socially pressured to tip, even for shitty service. This is what I dislike.

submitted by /u/Niz99 to r/unpopularopinion
[link] [comments]

Your boss is not your family

By /u/2DeadMoose

Your boss is not your family submitted by /u/2DeadMoose to r/BlackPeopleTwitter
[link] [comments]

[Postgame Thread] Oregon Defeats Utah 37-15

By /u/CFB_Referee

Box Score provided by ESPN

Team 1 2 3 4 T
Utah 0 0 15 0 15
Oregon 10 10 3 14 37

Made with the /r/CFB Game Thread Generator

submitted by /u/CFB_Referee to r/CFB
[link] [comments]

[Post Game Thread] The Milwaukee Bucks (20-3) blow out the Los Angeles Clippers (16-7), 119-91 extending their win streak to 14 games

By /u/ASkittlez

LAC Min FG FT 3PT +/- OR Reb A Blk Stl TO PF Pts
I. Zubac 24:26 6-10 2-2 0-1 -2 6 12 2 0 0 1 0 14
P. George 31:14 5-12 0-0 3-6 -25 0 7 0 1 0 5 4 13
P. Beverley 29:53 2-7 0-0 2-5 -32 2 4 1 0 1 2 4 6
K. Leonard 29:30 5-14 6-6 1-2 -18 1 5 4 2 2 5 0 17
M. Harkless 21:50 1-6 0-0 1-4 -8 0 3 2 0 2 1 1 3
M. Harrell 23:34 3-7 5-7 0-0 -26 2 7 2 0 0 0 4 11
L. Williams 23:13 5-13 0-0 1-4 -33 0 1 3 0 0 5 2 11
J. Robinson 19:37 3-12 0-0 0-3 -6 0 3 2 0 1 1 1 6
T. Mann 10:07 1-1 0-0 0-0 +3 0 1 0 0 1 1 1 2
D. Walton Jr. 9:47 0-2 3-3 0-1 +4 1 1 1 0 1 0 0 3
M. Kabengele 9:25 2-7 1-1 0-3 +6 0 2 0 0 1 0 0 5
P. Patterson 7:23 0-3 0-0 0-3 -3 0 2 1 0 0 0 2 0
L. Shamet 0:00 0-0 0-0 0-0 0 0 0 0 0 0 0 0 0
J. Motley 0:00 0-0 0-0 0-0 0 0 0 0 0 0 0 0 0
R. McGruder 0:00 0-0 0-0 0-0 0 0 0 0 0 0 0 0 0
J. Green 0:00 0-0 0-0 0-0 0 0 0 0 0 0 0 0 0
A. Coffey 0:00 0-0 0-0 0-0 0 0 0 0 0 0 0 0 0
Totals 240 33-94(.351) 17-19(.895) 8-32(.250) - 12 48 18 3 9 21 19 91
MIL Min FG FT 3PT +/- OR Reb A Blk Stl TO PF Pts
B. Lopez 24:42 2-12 2-2 0-5 +9 0 3 0 2 0 0 2 6
G. Antetokounmpo 26:20 11-20 4-6 1-4 +7 3 11 4 0 1 4 3 27
K. Middleton 25:29 7-9 0-0 3-3 +14 0 6 5 1 0 1 3 17
E. Bledsoe 22:55 4-10 0-0 2-4 +28 0 1 5 0 2 2 2 10
W. Matthews 22:44 3-6 0-0 3-5 +18 2 7 3 0 0 1 1 9
D. DiVincenzo 22:09 2-7 6-6 1-4 +15 2 5 1 0 3 2 1 11
P. Connaughton 21:08 5-5 1-1 2-2 +35 0 4 1 0 0 1 2 13
K. Korver 18:50 0-1 0-0 0-1 +13 0 5 3 0 0 1 2 0
G. Hill 16:09 2-3 2-2 0-1 +8 1 2 4 0 3 0 1 6
R. Lopez 11:49 1-5 0-0 0-2 -15 3 4 1 2 0 1 0 2
E. Ilyasova 9:53 5-11 2-2 1-3 +24 2 4 1 0 0 0 4 13
D.J. Wilson 8:56 1-3 0-0 0-1 -8 0 0 0 0 0 1 1 2
S. Brown 8:56 1-2 0-2 1-1 -8 0 1 1 0 2 1 1 3
T. Antetokounmpo 0:00 0-0 0-0 0-0 0 0 0 0 0 0 0 0 0
F. Mason III 0:00 0-0 0-0 0-0 0 0 0 0 0 0 0 0 0
D. Bender 0:00 0-0 0-0 0-0 0 0 0 0 0 0 0 0 0
C. Reynolds 0:00 0-0 0-0 0-0 0 0 0 0 0 0 0 0 0
Totals 240 44-94(.468) 17-21(.810) 14-36(.389) - 13 53 29 5 11 15 23 119
nbaboxscoregenerator.com by /u/Obi-Wan_Ginobili
submitted by /u/ASkittlez to r/nba
[link] [comments]

Group of students learning the strength of a proper shield wall

By /u/unnaturalorder

Group of students learning the strength of a proper shield wall submitted by /u/unnaturalorder to r/interestingasfuck
[link] [comments]

AITA for replacing a picture of my husbands late wife?

By /u/annoymous7892

Three years ago, my husband lost his wife of 18 years. She passed from a brain tumor, only 11 months passed between her diagnosis and her death. She was only 41 years old and left behind 4 children ranging from the ages of 5-16.

I have actually known my husband for about 8 years, as we work together. I always found him to be a handsome guy, but never did I start falling for him until a year after his first wife passed. He even introduced me to their kids before we officially started dating. They all accepted me with open arms and have all been nothing short of amazing.

We got married last November and welcomed our daughter in September. About a month back, we had some family photos done with all 7 of us. They finally arrived in the mail recently and I decided to start framing & hanging them throughout the house. There are 4 photos with my husbands late wife hanging around the house. 3 of them are old family photos and the other is a shot of just her.

There are only so many places to hang photos in this house, so I decided to move one of the old ‘family’ photos and place our newly done photo there. The old frame was a lot smaller than the new one I had put it, so I moved the old one to a more tight spot in a less visible area of the house.

I soon came to the realization that this may have come off a bit tacky, so I talked to my husband and children about it upon their arrival from work/school. They all said it was no big deal and encouraged me to leave the newly framed photo in this spot, which I did.

Over the weekend, ‘her’ parents came over to visit with the kids. They, especially her mother, have always seemed to have a blatant disdain for me. She discouraged my husband from marrying me because I was ‘too young’ to help raise 4 children . Then, she yelled at me for referring to myself as the kids mother. Despite the tense past, I have never seen her as angry as she was about the changing of pictures.

She accused me of trying to replace her daughter by giving rid of all evidence that she once lived in the house. I assured her that was not the case. She demanded that I put the original picture back in its place and find another spot to put my photo. I refused, telling her that my husband and her grandchildren had all approved of the way I did things. She then when on a long rant about how I have no respect for her or her husband. I replied by saying that this is my house and I will do things the way I want just as she can arrange the things she wants in her own house. They angrily stormed out of the house and we haven’t heard from them since.

Perhaps I came off a little too hard, but this woman has been making my life miserable from day one. I had simply had enough and felt that I had to put my foot down and stop letting her get away with the ‘grieving’ excuse.

submitted by /u/annoymous7892 to r/AmItheAsshole
[link] [comments]

Error'd: Press Any Key...EXCEPT THAT ONE!

By Mark Bowytz

"I'm guessing this is a case where there are keys and then there are KEYS," writes Guy G.

 

Eric G. wrote, "Based on this Apple News from the Future, I can't tell if George Lucas will live forever, or if he found a way to keep tweaking the Star Wars movies from beyond the grave."

 

"How anyone can claim that Valve is a money-grubbing company when they offer discounts this amazing is beyond me," Chris A. writes.

 

"I feel almost as if someone behind the Google News algorithm was like, 'You know what, with all this impeachment hearing drama, maybe you need a drink?', writes Hans H.

 

"Well, on the bright side, at least we have full transparency into their methods," wrote Frederick S.

 

Drew W. writes, "I had a sneaking suspicion that my Silver Airways flight was going to be canceled when I tried to check on its status and got this as a result."

 

[Advertisement] Ensure your software is built only once and then deployed consistently across environments, by packaging your applications and components. Learn how today!

CodeSOD: Failure To Process

By Remy Porter

Karl supplies us with an unusual bit of code. In the vein of a "true confession", it's code Karl wrote. In the vein of a good WTF, it had to be written like this because of bad choices made earlier in the pipeline.

But the code itself isn't a WTF. It's not good, but… well…

public override bool DirectoryExists(string dir) { //OH GOOOOD, DISGUSTING //UGHHHH UGHHHH try { FtpWebRequest ftpRequest = CreateRequest(dir); ftpRequest.Method = WebRequestMethods.Ftp.ListDirectory; ftpRequest.GetResponse().Close(); return true; } catch { return false; } }

As you can see from the comments, Karl feels very bad about writing this code. Karl wrote it because, when examining the FTP access library he discovered that there wasn't a DirectoryExists method, and Karl wanted to check to see if a directory existed.

Now, if you carefully examine the FTP commands, you'll note: there is no "exists" command. All you've got are a couple variations on list commands- the LIST command (which has no standard format for how information is returned) and the MLST/MLSD commands, which do.

Karl's code takes a simple approach then: try and list the directory. If we can, return true. If we can't, that triggers an exception, we'll catch it and return false.

If there's anything bad in here, it's that we're not being selective about the exceptions. We want to catch whatever exception represents a file-not-found error, but let any other exception bubble up. Otherwise, this will return false if the server is down, which is probably not what we want to have happen.

Given the features available in FTP, I'm not sure there's a more elegant way to do this.

Speaking of "inelegant ways to do this", Lothar recently sent us a different FTP WTF. Specifically, a customer was trying to bulk download JPGs from at FTP directory, and when they traced the FTP commands executed, they got this trace:

---> NLST *.jpg <-- 150 Accepted data connection <-- 226-Out of memory during globbing of *.jpg <-- 226-(This probably means "Permission denied")

The ---> is the command sent, the <-- tells us what the server replies with. Much like HTTP, server responses lead with numeric status codes, and a 2xx code represents a success. Obviously, this isn't a success, but we got a 226 response, which is explicitly a "we succeeded and we're closing the connection since we're all done here."

It's a weird message, and with a little googling, sometimes it's actually a memory issue, sometimes it really is a permission denied error, and sometimes it's apparently a certificate issue.

Which brings us to the real WTF, which isn't Karl's code or Lothar's error trace. It's FTP.

FTP is a legacy protocol which has been supplanted by more secure, more flexible, and definitely more usable file transfer protocols. But it's also still widely deployed, especially in enterprise spaces, because there are mainframe systems that need to send files and don't have a usable SFTP implementation (or, if they do, no one knows how to install it or set it up), or there's an EDI pipeline that was setup in 1986 that no one understands how it works but it is 100% central to the business process.

FTP is a legacy technology, and legacy technologies are like herpes: you might have had fun when it was getting installed, you probably went years without thinking about it, but when it goes wrong you regret everything, and you are never going to get rid of it.

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

Process Oriented

By Bobby T. Johnson

Andre was finishing writing documentation before he clocked-out for a much needed, 2-week vacation. He had stocked up his fridge with beer, energy drinks, and cola. He planned on working on raids with his gaming guild. He hadn't been as active as he liked lately, and was really looking forward to the break.

Andre's phone buzzed. He looked and saw Bob was calling. Bob struggled with the most basic of tasks, but worked in a large enterprise. His department contracted out to Andre to help offset the problem of their sales department.

“Hi Bob, how’s it going?” Andre asked.

“Hi, Andre thanks for taking my call. I have an unusual request.” stammered Bob.

“Yeah, shoot. I tend to enjoy the unusual.” said Andre.

“Well, uh…this is outside my department” Bob started, “and it’s rather personal. But, uh, you see, I left my car keys at the garage and they have my token...I need a login reset for the day, but because of company policy I could get, uh, disciplinary action for not having my token.”

“Yeah, sorry Bob. I can’t break the rules. You know I would.”

Bob sighed, “Ok, I understand. It never hurts to ask.”

“I don’t always agree with the rules but sometimes they are there for reasons we don’t know.”

After Bob’s problem, Andre went back to planning for his gaming weekend when he received an email. This was from another client, Initech Insurance. Initech used public databases, spreadsheets, and access for requesting information to financial advisors. The financial advisors sent updates back to Initech.

Angela, from Initech Insurance had bumped heads with Andre in the past. She was "process oriented", which is to say, she didn't care about the end results so long as you let her micromanage you. Once, she requested Andre send about 1000 emails out, but refused to let him use BCC and that every email had to personally sent. As long as they paid for his time, Andre only cared so much about their stupidity.

Andre looked at her email. It was a request to fix 16,000 records, in a shared Access database. The data, according to Angie, was "randomly shifted by a row". Ever a stickler for process, Angie explained that someone had already built an Access Form to manage the data, and someone simply needed to go through and manually copy/paste the data in that form.

Andre took a quick look at the dataset and saw that some of the data wasn't properly delimited, and on import had mashed some of the wrong data into the wrong columns. Glancing through the rest of the email chain, he saw that this had started over a month ago, when the account manager had asked Angela to fix this.

Andre clicked reply. He added the Project Manager. “Hi Angie, I think the best way to solve this challenge would be to use SQL to move the data between fields than using Access and copying and pasting. It’d also be faster and cheaper.”

He loaded up a game launcher and started to download a update and newly purchased games. He pulled out a energy drink and started to drink it when she responded, “NO! NO SQL, we are doing this in the Access Form.” He looked through the previous chain between her and the product manager.

The product manager asked her why she included Andre. Angie said she was working on it but wanted to ask Andre for the fastest result. He opened up Discord and messaged his friend, “Hey, I’m gong to be a few minutes late this consultant is trying to use me unofficially to fix a problem.” His friend responded with “K”

She responded “Please do the first 8000 records, and I’ll work on the next 8000 records. With both of us working on it, we should be done in a few days.” Attached was a spreadsheet of 8,000 row IDs that needed correction.

Andre sighed and looked at his rubber ducky. “Yeah, I know, but it should be quick and easy if I could use SQL. It won't matter if she doesn’t know.”

It was easy work in SQL. A careful select with a few case statements quickly created a new table with the corrected rows. It took Andre 15 minutes.

He resent an email to Angie, “Hi Angie Here is an updated list of the request changes.” He didn’t cc the product manager, because at the end of the day, he wanted to get paid and didn't care about the credit.

Angie immediately responded “That’s IMPOSSIBLE. Let me check.” After a few minutes. “Well, very good, maybe you can do the 4000 other records. I only managed to do 250.”

He sighed, “Yeah just send the rest of the records and I’ll clean them up.” A few minutes later he received the rest of the list and imported them into the previous database from before. Andre ran the same script. After fifteen minutes, he sent her an invoice and the data. He turned off his email notifications, and logged into his game. He looked at his rubber ducky and said, “Some rules are stupid and need to be broken.”

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

CodeSOD: An Utter Mockery

By Remy Porter

Today's submitter gave us their name as simply ImminentBurnout. IB works at a company that uses Python and has strong opinions about unit testing. They don't have much understanding to go with those opinions, but they definitely have opinions.

One opinion is that every object- every object must have a stub version to facilitate unit testing. Now, if you're familiar with Python, you know the MagicMock library is built-in in Python 3 and is available as a dependency in 2.7, so problem solved. A MagicMock can act as a stub for every class or method. Plus, it has patching operators to dynamically swap out implementations.

And if IB's workplace used MagicMock, we wouldn't have much to say.

Instead, they had an in-house generic module which would generate the boilerplate for you. IB doesn't tell us much about how this module is actually used- how you mark a class to have a mock generated.

But IB did share with us the implementation of the mock. Python's a flexible language, and there are a million ways you could accomplish this (even though MagicMock or a related library is probably the "right" answer).

One thing that's important to note is that, in Python, you can include arbitrary code in the class body. So something like this is perfectly valid:

>>> x = 5 >>> class Foo: ... if x: ... def bar(self): ... return x ... else: ... def bar(self): ... return 2 ... >>> f = Foo()

In this example, the definition of bar will change depending on the value of the variable x. This is generally not a good thing to do, but it's important to note that you can add loops, declare variables, and call functions from inside of the class body.

So, inside the body of the mock-generating class, how exactly did they leverage this feature of the language?

impl_def1 = ' retval = self.extend_f( "%s" %s %s )\n' \ ' try:\n' \ ' if retval == None:\n' \ ' retval = %s\n' \ ' except:\n' \ ' pass\n' \ ' return retval' \ % ( method_name, comma, ', '.join( input_defs ), ', '.join( output_defs ) ) impl_def2 = ' _ = self.extend_f( "%s" )\n' \ ' return' % ( method_name ) impl_def = impl_def1 if len( output_defs ) else impl_def2 stub_def = 'def method( %s ):\n%s' % ( ', '.join( input_defs ), impl_def ) exec(stub_def)

They do a pile of string-mungning then invoke exec, which as you might guess, executes a string as if it were Python code. That's the WTF, but they somehow managed to make string-munging more awkward and uglier than it needed to be.

There are a few things we can see in this code. First, the heavy use of the % operator implies that it started life in an older version of Python, where string formatting worked more like: "This %s is %s" % ("code", "bad"). So that could excuse not using MagicMock, perhaps, but there are still far better ways to do this. The bonus of using the older-style string formatting is that it helps make it basically impossible to parse out what's actually getting injected in each block- which values exactly end up in the line retval = %s\n?

You'll also note that they used the most awkward possible option for doing multiline strings, as Python supports (and has nearly always supported) using """ as the boundary for multiline strings.

IB has this to add:

It's so bad that it cured my impostor syndrome. I may not be the best, but at least I don't write this kind of crap.

And that, by the way, is the cure for imposter syndrome: realize that the world is full of successful morons, so no matter how much of a moron you think you are, you're entitled to whatever success you have.

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

CodeSOD: List Incomprehension

By Remy Porter

Loads of languages, like Python, have some sort of "comprehension" as a form of syntactic sugar. Instead of doing something awkward like:

my_list = [1, 2, 3, 4]
res = []
for x in my_list:
  res.append(x*x)
# res contains: [1, 4, 9, 16]

You can instead do:

my_list = [1, 2, 3, 4] res = [x * x for x in my_list] # res contains: [1, 4, 9, 16]

Used correctly, it's not just code golf, but it can make the intent and purpose of your code more clear. Used incorrectly, you can accomplish the exact opposite.

Vincent took over a product with a lot of modules which had, at one time, been very important bits of functionality, but now were deprecated. For example, there used to be an lxml-based parser which loaded data from an XML-based web-service. That webservice was long dead, the parser thus was no longer needed, but the code wasn't so well organized that you could just delete the module without doing a review.

That's how Vincent found this:

def scrape_ext(root, split_by): return '\n'.join([ ' '.join([b.strip() for b in c.split()]) for c in [_f for _f in [ y.strip() for y in root.text_content().split(split_by)] if _f]])

This is the impressive triply-nested comprehension, with useless variable names and a bonus bit of awkward indentation to help keep in unreadable and unclear. So much for Python's whitespace-as-syntax helping developers keep their code blocks properly indented.

Let's see if we can make sense of this by taking it from the inside out. First:

[y.strip() for y in root.text_content().split(split_by)]

This is easy, on its own: take the text of an HTML element, and create a list by splitting on some character, but also stripping whitespace. This, alone, is a pretty textbook example of a simple comprehension: it iterates across a list and manipulates each item in the list in a small way. The next comprehension, wrapping around that:

[_f for _f in split_and_stripped if _f]

This highlights another feature of Python comprehensions, filtering. You have an if _f at the end, which selects only the elements that are truthy values- any empty strings will be filtered out.

There's only one problem with that filter: it's not necessary. Because the next compression is for c in [_f for … if _f], so we could just as easily have done for c in split_and_stripped if c. And what do we do with c anyway?

Another nested comprehension:

[b.strip() for b in c.split()]

Split the string on whitespace, strip the whitespace… that we just split on. Python's split will remove all the whitespace characters, making the strip unnecessary.

Then we ' '.join([b.strip() for b in c.split()]), which shows us Python's unusual approach to joins (they're string methods, not array methods- this joins the array using a space between each element).

Then we join the results of all the other comprehensions with a \n.

So the real purpose of this code: turn all the whitespace into single spaces, then replace an arbitrary character (split_by) with a newline. But you wouldn't get that by just reading it, and I'm not entirely certain that's what the original developer actually realized they were doing, because this isn't the kind of code written by someone who understands the problem they're solving.

Like so much bad code, this was fortunately unused in the program, and Vincent was free to dispose of it.

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Error'd: Perfunctory Yet Functional

By Mark Bowytz

"This system is scheduled for a reboot at 26:00 hours on Monday. Or, as it's more commonly known, 'Tuesday'," Peter G. wrote.

 

"So I guess one person was just THAT torn on who to vote for," writes Joseph D.

 

David G. wrote, "To dash or not to dash, that is the question!"

 

"Yeah! How exactly does Lorem Ipsum work to...clean my house?" writes Yudderick D.

 

Rob G. wrote, "Maybe a file description is incoming, you know, after it starts responding."

 

Pete H. writes, "How 'nice' to have my special day recognized exactly as I'd expect it in an auto-generated email from a gigantic, international corporation."

 

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Classic WTF: Manager of the Data Dump

By Alex Papadimoulis

It's a holiday in the US, where we catalog the things we're thankful for. I'm thankful that developers collectively learned to understand how databases work, and didn't start releasing databases that stored flexible documents with no real schema and could just be used as a data dump. That would be terrible! This classic WTF illustrates that. Originally. --Remy

J.T. is not well liked amongst the developers at his organization. As a Database Administrator, it's J.T's job to make sure that database structures and queries maintain data integrity and do not put an unnecessarily load on the server. This often gets in the way of the developers, who prefer to think of the database as a giant dump site where data gets thrown and is rummaged through to be retrieved. Things like "indexes," "valid data," and "naming conventions" are merely obstacles put in place by J.T. to make their life harder.

Generally, the submission-review-rejection procedure happens once or twice with most of the developers. But one particular developer -- a newly hired ".NET Wizard" named Frank -- turns the procedure into a daily cycle that drags on for several weeks. Following is Frank's reply to the first in a chain of rejections on a project that Frank was leading up ...

J.T.,

> I cannot find the "DtxSurveys" table you asked to import this
> into. Are you sure this is the right table?

This is a new table we need. Please just use DTS to import the 
CSV file I sent and SQL Server will generate the table and the 
columns. That's how we're doing it in dev.

Frank

J.T. replied, explaining explained that, per the database development guidelines, he must submit a CREATE TABLE script that explicitly defines the table.

J.T.,

Fine, here is a script:

CREATE TABLE [DtxSurveys] (
[Status] varchar (8000) NULL, 
[Resp] varchar (8000) NULL, 
[Last] varchar (8000) NULL, 
... snip ...
[Supervisor_Name] varchar (8000) NULL, 
[Supervisor_CUID] varchar (8000) NULL, 
[Supervisor_UID] varchar (8000) NULL, 
[File_Data_Date] varchar (8000) NULL)

Frank

J.T. wasn't quite sure where to begin. The script was useful, as in a catch-all dumpster sort of way. He replied to Frank explaining that the table had no primary key, no typed data, and exceeded SQL Server's maximum row size of 8060 bytes. Frank wasn't too pleased and replied back:

J.T.,

When we run the script in dev, all we get is a warning that 
the row is too long. It's not an error, but whatever. Here
is the revised script:

CREATE TABLE [DtxSurveys] (
[Status] [nvarchar](255) NULL,
[Resp] [nvarchar](255) NULL,
[Last] [nvarchar](255) NULL,
... snip ...
[Supervisor_Name] [nvarchar](255) NULL,
[Supervisor_CUID] [nvarchar](255) NULL,
[Supervisor_UID] [nvarchar](255) NULL,
[File_Data_Date] [smalldatetime] NULL)

Frank

It was a slight improvement, as in a catch-almost-all garbage can sort of way. The only thing Frank changed was VARCHAR(8000) to NVARCHAR(255) and the File_Data_Date field. I'll spare you the rest of the back-and-forth on the CREATE TABLE script, but suffice it to say that it took several more revisions before it represented the actual: "Status" was a single letter, "Supervisor_UID" was a globally unique identifier, "Universal_ID" was an eight-character numeric identifier, etc.

I'll leave you with one of the last things that Frank sent over for review. It was the following query:

SELECT Product_ID, Product_name, 
case WHEN 
      (SELECT SUM(ProductRequest.ProductFaceValue) 
      FROM Requests WHERE (ProductRequest.ProductID = Product_ID) 
      AND (ProductRequest_createdByUserUID = @uid) 
      GROUP BY ProductRequest_productName) is null 
then 0 
ELSE 
      (SELECT SUM(ProductRequest.ProductFaceValue) 
      FROM Requests WHERE (ProductRequest.ProductID = Product_ID) 
      AND (ProductRequest_createdByUserUID = @uid) 
      GROUP BY ProductRequest_productName) 
end 
as TotalDollarAmount FROM Requests WHERE (product_ID IN (@pid))

When J.T. mentioned that they will have to optimize the query because it ran for 2000 milliseconds, Frank explained that it's already optimized and can't run any faster. J.T. updated the query to use an ISNULL and increased the run time to 52 milliseconds. It was a small, 3800% decrease.

[Advertisement] Otter - Provision your servers automatically without ever needing to log-in to a command prompt. Get started today!

CodeSOD: Repeat and Rinse

By Remy Porter

The challenges of doing a national integration continue to plague Sergio. More specifically, the “solutions” left behind by his predecessors continue to annoy.

Sergio has inherited a system which needs to plug in to a national database. As the national integration was something which was added after the business processes were already determined, that means that certain terms/descriptors/captions/etc. are used internally than are required externally, and vice versa. So, for example, one laboratory test Sergio’s company performs might be called “QD1” internally, but is known by the government as “F3+”.

As you might imagine, Sergio’s predecessors solved this with a database table called MAPPINGS. It contains all the mappings, so it might map lab test names, city codes, units of measure, group codes- just anything that could have a name that is possibly in conflict with the government’s requirements is in there. So the table has three key fields:

COD1 is the internal “code” for a thing. COD2 is the government’s code. And MAP_GROUP is a category tag, probably hastily added after a naming conflict between two different TLAs.

So far, so good. That all makes sense.

So let’s take a look at how they query the database.

	     public static String MapLabTests(String original) {
	        String result = "";
	        Logger logger = new Logger("ThisClass.MapLabTests");
	        StringBuffer sbQuery = new StringBuffer();
	        AccessBdNonXa accessBdNonXa = new AccessBdNonXa();
	        HashMap<String, String> parameters = new HashMap<String, String>();
	        try {
	            sbQuery.append(" SELECT  COD2 FROM MAPPINGS ");
	            sbQuery.append("  WHERE COD1= ?");
	            sbQuery.append("  AND   MAP_GROUP= ?");
	            parameters.put("1", original);
	            parameters.put("2", "LAB_CODES");

            	@SuppressWarnings("rawtypes")
	            Vector vectorBD = accessBdNonXa.lookup(sbQuery.toString(), parameters);
	            logger.debug(" Query bbdd" + sbQuery.toString() + " " + parameters);
	            if (vectorBD != null && vectorBD.size() > 0) {
	            	@SuppressWarnings("rawtypes")
	                HashMap hData = (HashMap) vectorBD.get(0);
	                result = (String) hData.get("COD2");
	            }
	        } catch (Exception e) {
	            logger.error("Exception: " + e.getMessage());
	            e.printStackTrace();
	        } finally {
	            if (accessBdNonXa != null) {
	                accessBdNonXa.close();
	            }
	        }
	        return result;
	    }

So, first off, the AccessBdNonXa class is the completely in-house, rolled from the ground-up data access wrapper. Sergio didn’t provide any of that code, but it’s a reinvented wheel.

Beyond that, this isn’t good code, but it’s hardly a true WTF. One has to wonder why they use a StringBuffer to construct a one-line query. I like the leading whitespace there, as it shows that there’s probably some copy-paste in its history, and the one character has been propagated everywhere that block is needed.

In fact, I know it was copy-pasted in a bunch of places.

That’s where this line comes in: parameters.put("2", "LAB_CODES");

A hard-coded value for a SQL query parameter should instantly make you suspicious.

What happens if you want to, I dunno, map city codes?

	    public static String MapDetGroup(String original) {

	        String result = "";
	        Logger logger = new Logger("ThisClass.MapCities");
	        StringBuffer sbQuery = new StringBuffer();
	        AccessBdNonXa accessBdNonXa = new AccessBdNonXa();
	        HashMap<String, String> parameters = new HashMap<String, String>();
	        try {
	            sbQuery.append(" SELECT  COD2 FROM MAPPINGS ");
	            sbQuery.append("  WHERE COD1= ?");
	            sbQuery.append("  AND   MAP_GROUP= ?");
	            parameters.put("1", original);
	            parameters.put("2", "GROUP_CITY_CODE");
            	@SuppressWarnings("rawtypes")
	            Vector vectorBD = accessBdNonXa.lookup(sbQuery.toString(), parameters);
	            logger.debug(" Query bbdd" + sbQuery.toString());
	            if (vectorBD != null && vectorBD.size() > 0) {
	            	@SuppressWarnings("rawtypes")
	                HashMap hData = (HashMap) vectorBD.get(0);
	                result = (String) hData.get("COD2");
	            }
	        } catch (Exception e) {
	            logger.error("Exception: " + e.getMessage());
	            e.printStackTrace();
	        } finally {
	            if (accessBdNonXa != null) {
	                accessBdNonXa.close();
	            }
	        }
	        return result;
	    }

There is one version of this method, copy/pasted, for each kind of category that could be looked up- at least eight at the moment, but these methods also aren’t quite centralized in any one place, so there honestly may be more, and as new codes and categories for codes are added, more of these methods will need to be added.

If only there were some tool that they could use, some way to parameterize this query, so that they could write the method once and pass different parameters to it. If only…

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

Newly Singleton

By Remy Porter

Shery was brought on to help with a project which was “going well”. “Going well” meant that all the little icons on the project management dashboard were green, which is one of the most effective ways to conceal a project which is struggling.

Shery’s component was large, and it was complicated, and it had a well defined interface to the rest of the application. Specifically, they had a documented JSON message format which her code would receive via JMS. That meant she could do the vast majority of her work in isolation, without talking too much to the existing team, so she did.

But the day came when they were going to send her messages. And the very first message her code received was a JSON document which looked like this: {}.

That wasn’t correct. Shery’s code dutifully handled and logged the exception, and she took it on herself to diagnose the problem. She pulled up the code from the other part of the team.

The first thing Shery noticed was all the code copy/pasted from StackOverflow. She could tell it was copy/pasted because that was the only code with any sort of sane indenting. All the code developed in-house used indenting stochasticly. One group of developers had clearly turned off their autoindenting in the IDE, another group hadn’t, and the result was a mishmash.

Most of the code was clearly done via copy/paste. If someone wrote a block of code in one section of the application, and someone else needed that functionality, they’d just copy/paste it around. There were miles of unused imports at the top of pretty much every file, there were statements following the pattern if (someCondition) { } else if (theExactSameConditionAsTheIf) { }. Suffice to say, there were nearly as many warnings as there were lines of code.

Shery decided she wasn’t going to debug or alter their code. Instead, she raised the issue she was seeing- empty messages- and politely suggested that she had noticed some “non-compliance” with the company’s coding standards which should probably be addressed, at some point.

While she was busy looking at the other team’s code, someone from the other team was looking at her code. And when she checked source control, there was a fresh commit at the head of the branch where they “fixed” some of her issues.

Shery had an object which provided a service. This object was itself stateful. That state should be the same everywhere in her component. So Shery created a Singleton.

Setting aside the concerns of managing any sort of global state, even in a Singleton, they were doing this in Spring. Spring, like most Java containers, has all sorts of features and functionality to manage object lifecycles. In “pure” Java, if you wanted a Singleton, you might do something like this:

public class Singleton {
    private Singleton() {}
    private static Singleton instance;
    public static Singleton getInstance() {
        if (instance==null) {
            instance=new Singleton();
        }
        return instance;
    }
    public void doSomething() {...}
}

It’s a load of boilerplate, but now you can interact with it by calling: Singleton.getInstance().doSomething(). No matter where you are in the application, you’ll always be interacting with the same instance.

Java frameworks, like Spring, exist, in part, to hide this boilerplate from you. Spring lets you make components, and assumes that the default scope for that component should be singleton, so this:

@Component
public class Singleton {
  public void doSomething() {...}
}

Is the same as before, but with a lot less garbage code.

Even better, if a class wants an instance, they can define the property like this:

@Autowired
private Singleton svc;

At runtime, the container will managed the instance, and when anyone requests it (using decorations like @Autowired) the container will inject the correct instance. This is the “right way” to do this sort of thing in Spring.

When Shery checked the commit her peers had written, they had done this instead:

@Component
public class Singleton {
    private Singleton() {}
    private static Singleton instance;
    public static Singleton getInstance() {
        if (instance==null) {
            instance=new Singleton();
        }
        return instance;
    }
    public void doSomething() {}
}

In essence, they mixed the pure Java approach and the Spring approach to accomplish nothing. For bonus points, they kept the autowired svc variable, but would interact with it like this:
svc.getInstance().doSomething().

Shery never got a chance to have a conversation with the developer responsible for that change. The project continued to “go well”, you see, so well in fact that half the team was sacked and the project was moved to another department to be taken to completion. Don’t worry, the dashboard continued to show green for all of their key-performance indicators.

[Advertisement] ProGet can centralize your organization's software applications and components to provide uniform access to developers and servers. Check it out!

CodeSOD: A Very Personal Role

By Remy Porter

Nohemi has a program which needs to apply role-based security. Due to their organizational needs, the rules for role names are a bit unusual. Some roles have to be a case-insensitive match. But some roles have a more flexible pattern they need to have. This is how her co-worker implemented this:

public static String decodeRole(String role) { String decodedRole = ""; if (role != null && !role.trim().equals("")) { if (role.trim().equalsIgnoreCase(ROLE_1_STRING)) decodedRole = CODE_ROLE_1; else if (role.trim().equalsIgnoreCase(ROLE_2_STRING)) decodedRole = CODE_ROLE_2; else if (role.trim().equalsIgnoreCase(ROLE_3_STRING)) decodedRole = CODE_ROLE_3; else if (personalContains(role.trim(), ROLE_4_STRING)) decodedRole = CODE_ROLE_4; } return decodedRole; }

Here's the key method which does this translation. Roles 1, 2 and 3 must be an exact match. Role 4, on the other hand, has to apply their special rule, a rule so complicated it can only be evaluated via a regular expression:

private static final String REGEXP_SUFFIX = ").*$"; private static final String REGEXP_PREFIX = "^.*("; public static boolean personalContains(String fatherString, String toSearchString) { Pattern p = Pattern.compile(REGEXP_PREFIX + toSearchString.toLowerCase() + REGEXP_SUFFIX); Matcher m = p.matcher(fatherString.toLowerCase()); boolean matchFound = m.matches(); if (matchFound) return true; else return false; }

It's a contains check. If your search string were, say, ROLE_4, the constructed regex would be ^.*(ROLE_4).*$, which is "the start of the string, followed by zero or more of any character, followed by ROLE_4, followed by zero or more of any character, followed by the end of the string.

It's a contains check. This isn't even the right way to do this is regular expressions, since ROLE_4 would be a regex which matches if any substring is ROLE_4, but regexes aren't the right way to do this in the first place, since Java has a lovely String.contains method already. The entire personalContains method could be removed.

The real WTF, though, is that instead of returning m.matches(), they have to do this:

if (matchFound) return true; else return false;
[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

Error'd: BSOD with a Side of Fries

By Mark Bowytz

"Yes, I'd like to have a Quarter Pounder meal with a Coke and a Blue Screen of Death on the side. To go," Bruce W. writes.

 

"Am I going to be highly satisfied with my XPS system temperature? Well, Dell, I just don't know yet!" writes Ben S.

 

Steve M. wrote, "According to Chrome-based Microsoft Edge Dev channel, it's got 0 x 256 problems, but revealing my name ain't one."

 

"Dear Micsrosoft - Change text to 'bowling alley', please and thank you," writes Maxwell D.

 

Kosti J. wrote, "Wow! Some great deals in this AliExpress campaign, such as ...a cycling clothing set at a 138% markup!"

 

"At work, we use a pretty filthy iPad to work our coffee machine. Right now, there's an iOS update between me and my caffeine...and, well, I don't like it," Roger G. writes.

 

[Advertisement] Continuously monitor your servers for configuration changes, and report when there's configuration drift. Get started with Otter today!

Representative Line: What Am I?

By Remy Porter

Object oriented programming is the weapon of choice for many programmers, and when wielded properly, you can often rely on a mix of convention and strong types to make it clear what type of object you’re working with. Sometimes though, you need to check. In a language like Java, you have the instanceof operator, a boolean comparison which answers if obj instanceof SomeClass. Seeing a lot of that in a codebase is a clear code smell.

Sometimes, though, not seeing it is the code smell.

Chris S spotted this pattern repeatedly in their codebase:

  setHasTypeCorporateAccountOrTypeCustomerAccount(hasTypeCorporateAccount() || hasTypeCustomerAccount() || hasTypeCorporateAccountOrTypeCustomerAccount());

This code, inside of the Account class, is calling a set method to set the value of hasTypeCorporateAccountOrTypeCustomerAccount. It determines that based on related boolean values- hasTypeCorporateAccount and hasTypeCustomerAccount, and of course hasTypeCorporateOrTypeCustomerAccount (the value being set).

As you might have guessed, this creates a situation where the boolean flags are used for a weird version of polymorphism elsewhere in the codebase, e.g., if (acct.hasTypeCorporateAccount()) {…}. That’s already ugly, but as implemented, the underlying fields are just boolean values, and the set methods just directly set the values, with no checks. So it’s entirely possible to have hasTypeCorporateAccount() return true, while hasTypeCorporateAccountOrTypeCustomerAccount() return false. And sometimes, that happens, and is the root cause of a bunch of different bugs.

Chris can change the getters to make this behave rationally, but fixing the root cause- the person who wrote this- might be a little harder.

[Advertisement] ProGet can centralize your organization's software applications and components to provide uniform access to developers and servers. Check it out!

The Support Game

By Ellis Morning

PDP-11 (459312210)

In the 1970s, shortly before our friend Argle dared to do exactly what his boss asked of him in an efficient manner, he worked at the computer lab of a local community college. When his friend Terry was hired on as a new assistant, Argle sat down with her at the Tech Support desk for a run-down of hard-earned knowledge and best practices.

"The real trick to this job," Argle wrapped up, "is to realize that the only four answers you ever need to give are 'Yo,' 'Oh,' 'So,' and 'No.'"

"What?" Terry's brow furrowed. "That would never work!"

As if on cue, a student came up to the help desk just then. "Excuse me?"

"Yo!" Argle greeted, turning to face him.

"Professor Goddard wants me to use the astronomy tutorial," the student said.

"Oh?" Argle prompted.

"Well, I don't have time today."

"So?"

"Can I have it on a floppy disk to do it later?" the student requested.

The tutorial in question was a slick program written for the PDP-11 that ran on purely text-based dumb terminals. It was ingenious for the technology of the day, but not compatible with an Apple ][ or Commodore Pet, the likely targets for the student's use.

Argle wasn't about to over-explain anything to someone who wouldn't get it. "No, I'm afraid that's not possible," he said. "Sorry."

The student looked disappointed, but shrugged. "OK, thanks."

As he walked away, Terry doubled over in her chair and bit her lip, fighting off hysterics.

With her training complete, Terry was now ready to run the help desk herself. When her first client—a professor—approached, she knew exactly how to handle the matter. "Yo!"

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

CodeSOD: Never Refuse a Fifth

By Remy Porter

Sometimes, you want your dates to look like this: 3/3/2019. Other times, you want them to look like this: 03/03/2019.

There are plenty of wrong ways to do this. There are far fewer right ways to do it, and they mostly boil down to “use your language’s date library.”

And then there’s this, which “QuakePhil” found.

    function convertSingleDigitMonthOrDayToTwoDigitMonthOrDay($date) {
        $month = '';
        $day = '';
        $secondChar = substr($date, 1, 1);
        $thirdChar  = substr($date, 2, 1);
        $fourthChar = substr($date, 3, 1);
        $fifthChar  = substr($date, 4, 1);

        if ($secondChar === '/') {
            $month = '0' . substr($date, 0, 2);
        } else if ($secondChar !== '/') {
            $month = substr($date, 0, 3);
        }

        if ($thirdChar === '/' && $fifthChar === '/') {
            $day = '0' . substr($date, 1, 2);
        } else if ($secondChar === '/' && $fourthChar === '/') {
            $day = '0' . $thirdChar . '/';
        } else if ($secondChar === '/' && $fifthChar === '/') {
            $day = substr($date, 2, 3);
        } else if ($thirdChar === '/' && !in_array('/', array($secondChar, $fourthChar, $fifthChar))) {
            $day = substr($date, 3, 3);
        }

        return $month . $day;
    }

It’s worth noting that this project had just undergone a massive refactoring effort, and somehow this particular method snuck through.

It’s an… interesting approach to the logic. Instead of splitting the string on "/" (which would still be wrong, but at least make sense), we check which position the "/" falls. But we don’t do that by searching the string or indexing the string, but by doing a pile of substr calls and saving the result.

Phil suggested changing the code to something more reasonable, but since there had just been a massive refactoring project, nobody want to make any code changes which weren’t new features.

[Advertisement] Ensure your software is built only once and then deployed consistently across environments, by packaging your applications and components. Learn how today!

An Excelent Start to a Career

By Charles Robinson

Hal was a wiz kid computer programmer at age 15 in 1976. He could make the toggle switches and LEDs on his Altair 8800 dance at will. In college, he was class valedictorian after earning his computer science degree in 1984. Hal was destined for greatness and the real world was about to get rocked.

Hal's college friend Victor, who graduated two years prior, was already running his own startup company that made Unix-based financial planning software. Remembering Hal's brilliance, Victor recruited him to join his company the day after graduation. Victor needed the wiz kid-turned-wiz adult to create the equivalent of Lotus 1-2-3 in Unix. It was a tall first project but it paid well, so Hal happily signed up. Besides, everyone knew that spreadsheets were gonna change the world.

Hal was so full of ideas, he felt like he could take Lotus 1-2-3 and make it better. He built Unixus 3-2-1 from scratch and vowed to turn it into a fun and useful program. Surely it was bound to make Victor's company millions upon release.

Victor didn't seem to notice Hal's efforts, though. He was seldom in the office any more, the parking spot for his Porsche frequently sitting empty. Whenever Victor was actually around, he seemed disheveled and claimed to be preparing for an important sales trip. There were myriad rumors going around that he was actually out partying with the cashflow from their first big sales.

One day, Hal came in to work only to find the doors chained shut. A note from Victor was taped to the door. In it, he explained how the company was going bankrupt and he had to sell it for pennies on the dollar to a larger tech firm. All of the company's assets now belonged to the new ownership, and everyone was out of a job. Hal felt crushed that he lost his first job in addition to his source code for Unixus 3-2-1.

Many years went by and Hal moved on to more stable employment. He initially held a strong resentment towards Victor but it gradually faded with each passing year. Victor too had found himself some more stability and his hard partying days were over. Victor and Hal eventually reconnected through a mutual friend.

"Hal! Good to see you, old buddy!" Victor shouted, rising from the restaurant table he reserved for them. Hal shook his hand, less enthusiastically than Victor did. "Hey, I just want to say right away that I'm sorry about the whole company closure way back when. I was young and immature and I felt so bad about costing everyone their jobs," Victor dropped his head, showing that his 80's permed hair was long gone.

"Hey, don't worry about it. It wasn't a big deal," Hal downplayed, failing to mention how he used to throw darts at a picture of Victor. Hal started to open up more while they reminisced about wild college stories. Victor's memory of things… was a bit different. Specifically, he was always the center of every story. He always was the big winner in every bit of college hijinks, and if Hal remembered anything embarassing about Victor, it was Hal's memory that was faulty- Victor was King of the School.

They eventually got around to talking about Unixus 3-2-1. "I know it was my first real project, but I felt like my spreadsheet program was really something. What ever happened to all the source code from that place?"

Victor laughed, "Oh Hal, you won't believe this! Most of our products were scrapped or hacked to bits and repurposed. But they realized Unixus 3-2-1 had potential. They added a few finishing touches then packaged it up and sold it off. Boy, I wish I could have gotten a better deal when I sold the joint. If I had only known…"

Hal failed to find any of that unbelievable. "Well, at least it made someone some money," Hal lamented. "Do you have any idea who they sold it to?"

"Well now, here's the crazy part," Victor paused before drawing a breath. "There was this up and coming software company around that time that rhymes with Bike-rosoft that was eager to get their hands on it. They took what you made and it became the core of a little program called Excel."

Hal instantly started to regret reuniting with Victor. Whether he was full of crap or not, Hal's resentment came flooding back during the rest of their lunch together. He made an excuse to leave right after the check came, hoping to avoid something like Victor claiming he helped Al Gore create the internet.

[Advertisement] ProGet supports your applications, Docker containers, and third-party packages, allowing you to enforce quality standards across all components. Download and see how!

Return to the Vaucluse, 2019

By Jon North (noreply@blogger.com)


One of our first ports of call when we began regular visits to France over 25 years ago was to the southern Rhône valley, the south of the Drôme département (south that is, of our old twin town, Die).  This September we were back in the Vaucluse for a short overnight stay on my birthday.  The photo above is from the road approaching the Domaine de Coyeux, high above Beaumes de Venise, with Mont Ventoux in the distance.  Try as I might I could not convey the height and scale of this well-known obstacle in many Tours de France, but it is often there in views of the area.

When we first visited in the mid-1990s the only way to approach Coyeux was via an unmade road, snaking across the top of the Dentelles de Montmirail, itself already a worrying distance from civilisation.  We've visited the Domaine de Durban, on the road towards Coyeux, quite often over the years, but now that track is closed and the road to Coyeux snakes up the hillside like a Tour de France special, until you arrive on the high plateau with the lace-like rocks that give the Dentelles their name, at closer quarters.

On the way to the Domaine de Coyeux
When we first came in the 90s, Beaumes de Venise was well-known for its sweet muscat wines, as it still is, but in the past 20 years sweet wines have become less popular and the reds from this village have become increasingly good and, compared with the neighbouring Gigondas or Vacqueyras, let along Châteauneuf du Pape, very good value.  Coyeux is noted this year in the Guide Hachette for an excellent red, Praestans, which fulfilled our expectations when we tasted it.  We also fell for dry white and rosé wines from Muscat grapes - a very worthwhile visit with views to match.
   

             

All the white and rosé wines we tried at Coyeux were made from muscat petit grains grapes, the same as are used to make the fortified Muscat de Beaumes de Venise.  When we firt came, dry wines were only made when the harvest was plentiful enough.  Now, dry white wines are increasingly normal, and I was surprised to discover rosé made from the same grape (this is not really surprising - the skins are reddish though the juice is 'white', so leaving ths skins in contact with the juice for a short time makes a nice pale rosé).  Mary is always pleased to find dry muscat, and the Coyeux rosé and white were both excellent.  We also have more and more dry muscat around us in Lunel, and one of our best winemakers tells us he decreases the sugar content of even his sweet wines year by year as people's tastes change.

As we often do when travelling in France, we chose to stay in a Logis de France hotel - almost always with good restaurants attached.  This one in Sablet, the Domaine de Cabasse, lived up to the usual good standards and is as its name suggests also attached to a wine domaine.  However, with so many good wines around we didn't buy here this time - just drank some of their good red with our meal.  The pool was also a welcome attraction in the afternoon heat.





The hotel is among the vines midway between the small sleepy villages of Sablet and Séguret.  
As you emerge onto the little road, across the fields you can see two huge buildings typical of the caves coopératives you come across right across the south of France, but if possible even bigger.  This I guessed, rightly, was the Gravillas Coop which we were aiming for after our night in the hotel.  It has a good reputation, and a Rosé in particular which got outstanding marks in the latest Guide Hachette.  The previous afternoon we'd stopped in another small sleepy village, Violès, on the plain between the Rhône and the Dentelles.  There we visited a charming family-run domaine, the Bastide Saint Vincent, another Hachette recommendation, whose red Florentin is a splendid example of the newish 'Plan de Dieu' (plain of God, I guess) Côtes du Rhône Villages appellation.



               
 The story of our link to Violès also goes right back to our twinning visits 25 years ago.  At that time our local organic shop in Derbyshire had branched out into wines, and had a nice red Côtes du Rhône and we decided to see if we could find the producer.  At that time Violès was even more of a backwater than it is now - small, dusty and not really used to tourists, but we found the winemaker, bought some wine and had a meal in the little restaurant at the village crossroads.  The raising of quality and acquisition of the plan de Dieu tag has brought new fortune to Violès, which now has more commerce, several well-set-up winemakers and a beautiful new public library among other things.  All very enouraging

To finish here are a few photos of the area, taken on the top of the Dentelles but a little further south and west around the Domaine de Durban, with some better views of Mont Ventoux too




Our dog Camel

By noreply@blogger.com (Jon North)

Over 250 years the English poet Christopher Smart wrote the poem For I will consider my cat Jeffrey.  Now we think constantly 'for we will consider our dog Camel', the new friend at 74 rue de la Brechette following Evie's sad death last autumn.  Her replacement is several times her size!!


Last week we went to Figeac, 3-4 hours' drive west of Lunel, to collect our new companion, a 10-year-old male dog called Camel.  He had lived for 4 years in the refuge in the remote hilltop locality Nayrac a couple of km outside Figeac on the river Lot.  The refuge is operated (since the late 80s) by  MASAQ, the Mouvement Associatif pour la Sauvegarde des Animaux du Quercy, a very caring and committed group which seems to be struggling to cope with demand - the 100+ dogs have to be housed, fed and walked by volunteers and they are completely full up on the cat side, and the refuge has shelter and some limited facilities, but little warmth or protection for the many animals in the woodland setting surrounded by rubbish processing plants at the end of a long winding dead end road.

Camel is a placid, friendly animal, underweight and in need of more walks and more food, carefully controlled to allow for his rather shrunken stomach, and now just starting to get used to our warm dry house!  He was chipped and checked by their vet before we could collect him and sign the adoption papers, and given a thorough check by our own vet the day after he arrived.  But if he seems bemused to find himself in our home it isn't surprising - he'll still be getting used to being warm.

Figeac itself is a really nice little town with a really lovely medieval centre which we explored despite the annoying closure of the Tourist Office for staff training!  But going to the Mairie instead to get a town plan was a bonus, because the young woman at the desk turned out to be a volunteer at the refuge and of course knew one of their longest-staying residents!  She and we were very glad to say hello before we went to find him.

Here is an album of photos of Camel in his first few days with us.

Road trip

By noreply@blogger.com (Jon North)

Interrupting my enjoyable accounts of meetings with family and friends in England, a trawl around out 3500 km plus round trip to England from the S of France.

We decided to drive because we thought of taking French wine to our family.  In the process we kept some Burgundy for ourselves from our favourite Jacob vineyards - crémant, Chambolle Musigny and some white Hautes Côtes de Beaune matured in Acacia barrels which is less easily available so we were lucky to find it!  Sadly, on our return through France we could not find a chance to revisit Echevronne but we'll be back, no doubt.

Perhaps the highlights of the route and the roads we travelled were or will be the Millau viaduct which will more or less welcome us back into the Languedoc when we return home down the A75 tomorrow, and the quiet journey from north of London up the A5 towards Staffordshire as we started our visit in December.  the Narrow Boat pub on the Grand Union Canal has been an enjoyable stopping place over the years.  We returned south on the A5 but sadly the pub was not yet open for coffee as we passed.

But the most surprising pleasure is the moment on the M25 near Rickmansworth when it passes neatly under the Chalfont Viaduct - as the Atlasobscura website relates "Those impressive Edwardians thoughtfully left enough room between the arches of this grand 1906 railway viaduct for an eight-lane motorway to pass through".  I can't help thinking the motorway designers had to back off a considerable distance to line up the M25 through those excellent brick arches.

Our almost daily journeys from near Burton to Wirksworth, between the houses of our sons, gave us time to refine our route - the picturesque but windy one via Ashbourne was rapidly replaced by the smoother but still enjoyable route via the A38 and A50, quicker and less demanding!

We have learnt to pace ourselves, shorter stages with hotel stops, but still on these short winter days the end of each day is a challenge as streams of headlights approach us in the gathering darkenss as we attempt to decipher the directions to the next hotel.  And even welcome sunshine in France today presented problems as we negotiated ring roads and roundabouts.  But we enjoyed the sights we could spare time to see - Chartres cathedral against the skyline, and a sliver of crescent moon as we drove towards Orléans.

The car ferry is a gentle respite on the route, and the Channel Tunnel is so quickly over in a car that you scarcely remember it before you are on the road again.  But on short winter days perhaps the tunnel is a better bet, leaving more daylight hours to drive safely in.  Changing sides of the road no longer presents too many challenges - I always feel that the signage and road markings are clues enought to which side of the Channel and of the road you are currently on.  And we saw no gilets jaunes at all on the way up,though several stood by their roadside fires and did not bother us as we passed Dieppe, Evreux and Dreux on the way down.

An anniversary and other Christmas moments

By noreply@blogger.com (Jon North)

Our 40th wedding anniversary was in October, but since we could not meet the family then we arranged a meal together just after Christmas.  It was lovely to have all our sons and their families with us for an evening of Italian treats in Oakerthorpe, where the Peacock pub has been rebranded Pesto at the Peacock!


We shared our time over the festive period between the homes of Jeff & Fi and Sam and family, enjoying several exchanges of presents, walking around the Staffordshire countryside and taking occasional trips to shops in Burton and Ashbourne, as well as catching up on tv and sleep!

Jeff & Fi's cottage across the fields
 






Some Christmas moments at Marebrook

By noreply@blogger.com (Jon North)

More photos from our stay at Jeff & Fi's









Waiting for Christmas in Staffs and Derbys

By noreply@blogger.com (Jon North)

Our days leading up to Christmas this year have been spent with the usual mix of shopping, wrapping, watching telly and spending time with our family.  This is a little photo diary of our first few days in rural Staffordshire.

  




 




Christmas trip part 2 - to the Midlands

By noreply@blogger.com (Jon North)

NB photos of Staffs taken on an earlier winter visit
 I'm writing in the early morning of Saturday 22 Dec in a cosy cottage near Burton-on-Trent, home our son and daughter-in-law Jeff & Fi, and our temporary resting place in England.  We've driven nearly 1500 km to get here with amazingly little difficulty - not only avoiding hold-ups in France but making the journey from Kent to London and then on to Staffs smoothly on quiet roads.  All the more amazing on the brink of a holiday weekend when all reports are of heavy traffic and jams.  They all seem to have happened after we passed!

Our short stay in London with old friends Ruth & Claus was comfortable and warm in every sense.  We had a chance to wander in Highgate village and introduced two sets of friends from different parts of our lives over lunch - if there was a common thread beyond a general enjoyment of culture it was the appreciation of good wine: we'd brought with us wine Gaynor & Ed had bought in Bordeaux during our summer visit there together, and we enjoyed both good Bordeaux red and delicious Chablis with lunch.  I keep straying in this blog into the territory of the wine one!

Highgate scenes

So we set our yesterday from London, aiming for the A5.  This lust be one of the least diverted of the old Roman Roads in the country, once you get onto it north of St Albans.  Of course it has new bits like the Milton Keynes bypass that Romans would have found it hard to imagine (even without the concrete cows) but mostly it passes through delightful undulating rural landscapes all the way to Leicestershire where we turned off towards Burton.  On the way up we stopped for some lunch at a really good raodside pub, the Narrow Boat at Weedon (now we discovered with Motel-like rooms fringing the car park).  We visited this quite often on our trips to and from Derbyshire some years ago.  The grounds are right next to the Grand Union Canal and it's altogether a good place to visit.  We arrived at Jeff & Fi's at 3 pm and enjoyed a cosy evening with a blazing stove, good wine(!) including some excellent Aldi organic prosecco we'd discovered on our last visit and some beautiful pinot noir we'd brought from the Limoux vineyard of Jean-Louis Denois.  A good start to our holiday.

Christmas in England - arrival

By noreply@blogger.com (Jon North)

With the uncertainties of Brexit on the horizon, this seems like a more than usually important time to visit our family in England, and we are really looking forward to being with Jeff & Fi, Sam, Sas, Heather and Ben and also seeing Ed, Isla and Karen for a while.  Lots to look forward as I sit in our Folkestone hotel which was a nightmare to find in the wet and dark, but it all felt better after a drink in the bar!

The past two days started early, very early, as we drove up a dark and almost deserted A9 from Lunel to our overnight stop in Burgundy, not quite knowing if the gilets jaunes would add to the uncertainty of Brexit.  Weekend reports were not encouraging - we read of fires on and by motorways and toll booths, a whole section of the A7 from Avignon to Valence closed, numerous entrances and exits closed as our own local ones had been on and off for the past few weeks.  We started out full of anxiety, just hoping that after the weekend the demonstrators would all heed to colder damper weather and go home.  

All sorts of thoughts have crowded in on us as we've driven 1,000+ km up France.  We drove without seeing a single gilet jaune. Tant mieux, and of course there are fewer in the north than in the south, fewer in midweek than at weekends, fewer now than there were 2 months ago. But we were reminded that until recently a frequent electronic message on motorways here used to be 'respect the men in yellow’, and today it says something like ‘agents anticipés’. Does not trip off the tongue as the other used to, but no danger of mistaking workmen for protestors, at least in the wording of messages!  

One final thought about these demonstrations - we read in a local paper in Burgundy that local groups of gilets jaunes are hoping to form associations so that they can be eligible for grants from public authorities!!  So people who began by demonstrating against too high taxes are now hoping some taxation will be spent on their groups.  In fact, merssages are mixed - some want lower taxes; others want to preserve services which are being cut.  On top of that, some have destroyed lots of things, motorway toll booths and sometimes the roads themselves with fire.  Vinci (one of the motorway companies) reckons they will need to spend millions repairing things.  Not easy to square with keeping up tax-funded services.  But enough, we don't understand everything and no doubt there are important causes to be fought.  The rest of this will be, more pleasantly, concerned with our doings.

Our first port of call in Burgundy was our friends Jean-Michel and Christine Jacob, whose wine will accompany us to England and (some of it) back home.  It's always a pleasure to see them again, but their story really belongs in my wine blog where I've often written of them (here for example), so suffice it to say that we were warmly received and delighted with the wines we found and could take away, as we hope our family and friends will be!  We had a good simple lunch in Beaune, a pleasure to revisit this lovely town, and then sought our our hotel.


lunch in Beaune - Mary appears twice thanks to the mirror next to our table!
The hill of Corton from our hotel terrace
We'd chosen a Logis hotel in Ladoix Serrigny, on the south-eastern flank of the famed Hill of Corton (origin of grand crû wines which we did not seek out this time, both red and, unusually north of Beaune, white (Corton Charemagne).  It was a simple but comfy overnight stop and we set out at leisure for the north today.  The roads were quiet, the weather mostly dry and often bright, and the middle stretch in particular as we approached Troyes across the Fôret d'Orient was full of the amazing variations of colour and texture of winter scenery, branches, fields and hedgerows.  The last stretch past Reims and on up to the coast is long and less interesting to look at, but we made the Tunnel hours ahead of the time we'd planned and so crossed into wet Kent and our first night on English soil, looking forward to seeing people as we travel north tomorrow and after.

September wine fairs

By Jon North (noreply@blogger.com)

The French 'rentrée' is also the start of an interesting period for wine enthusiasts - the season of wine fairs in supermarkets.  In all, these span nearly 6 weeks, and these days the wines are not only good value but also carefully selected.  As with medal competitions, you always have to bear in mind that makers who already have a good market for their wines need not participate, but with the aid of review articles you can usually buy good wines at good prices.  Since the best bargains are usually snapped up early, the secret is to arrive at the supermarket at opening time on day 1. 

I did this several times in September, partly for our wine tasting circle here in Lunel and partly to find wines as presents.  The wines I bought are listed below, but first a few notes on the different places I visited and on some of my choices.  I was aided by the comprehensive article in the Revue in August, which highlighted a dozen or so top picks from each chain.

The earliest wine fairs are in the cut-price supermarkets - Lidl, Aldi, Netto and Leader Price.  One could add BioCoop but their wine fair sas so chaotic that they could not even agree on a start date, so lost out as I arrived early on the first day of each.  That’s what you have to do to snap up the best bargains.  Lidl is the most impressive, with three aisles dedicated to a huge range of French and a few foreign wines.  As you can see, I found plenty of choice even without the usual array of Bordeaux reds.  Rhône reds and a nice Touraine white were my picks here.

For the rest, the budget stores ranged from the chaotic Netto (lucky to find any of the wines listed beforehand) to the interesting but slightly disorganised Leader Price and the very nicely organised Aldi, whose range I’ll explore more next year.  Although the major supermarkets’ fairs start later in September, or into October, I managed to find one of my star buys in Intermarché and (as I have done often in previous years) some good buys in Leclerc.

Two personal stories link to my wine fair visits this year.  The first is a red from the flat lands between Orange and the Dentelles de Montmirail in the southern Rhône, from the village of Violès.  When we first discovered this it followed a purchase from the organic shop Beanos in Matlock Bath, which we used a lot during our time in Derbyshire.  At that time Violès was on the bottome rung of the Côtes du Rhône, a sleepy village en route to more celebrated places like Vacqueyras, Gigondas and Beaumes de Venise.  Since then the village has acquired Côtes du Rhône Villages status itself, and the Tour des Abbesses we found in Inter was one of the best reds we’ve come across recently for  everyday drinking.  I seek it out when I visit Intermarché stores in the Cevennes, but I doubt it will be sin stock for much longer.

Second, one of the highlights of my visit to Aldi was the Bonnezeaux from a well-known producer, Château de Fesles.  The sweet chenin blanc wines from this area (the Layon valley south of Angers) are exceptional and long-lasting, one of the few white appellations outside exalted Burgundies to keep more than a few years.  But is is for this very reason that makers divest themselves of wines 15 years old or more.  We first discovered this in the 1990s when we bought 1979 Bonnezeaux from the very same producer from 3D wines, who introduced us to some of our favourite makers elsewhere.  Because these wines last so long, makers hold onto stocks but in the end have to sell them to make room in their cellars, and we profit.  This Bonnezeaux was not a great wine, but a delicious wine to drink now all the same, and it will keep a few years yet.

By the way, a late purchase not on my list below is a Savennières, another Anjou wine but this time dry but equally longlasting.  Sadly the bottle we bought from Leclerc will not be tested for longevity because we finished it between us at lunchtime today, but there is another lurking and I’ll be getting back to Leclerc in the vain hope that there may still be some left - utterly delicious.

Here is my list of purchases, 7 whites first, the rest red.


Saint Véran Louis Dailly 2017 Leader Price 8.04 €
Macon Villages Cave d'Azé 2017 Netto 4.99 €
Touraine Sauvignon Caves Gilles Gobin 2017 Lidl 3.99 €
Menetou Salon Patient Cottat 2017 Intermarché 9.95 €
Sancerre Les Fossiles, dom Roblin 2017 Intermarché 13.75 €
Alsace Riesling Rittimann Celliers de Romarin 2016 Aldi 5.99 €
Bonnezeaux Château de Fesles (50 cl) 2001 Aldi 13.99 €
CdR Villages Dom de Tavans 2017 Leader Price 5.87 €
Cahors Malbec du Clos 2015 Leader Price 5.33 €
J L Baldès
Saint Joseph Dom de Blacieux 2017 Lidl 9.99 €
Vacqueyras Terroir des Dentelles 2016 Lidl 6.99 €
SCA Rhonéa, Beaumes de Venise
Juliénas Collin-Bourisset 2017 Lidl 5.99 €
Vinsobres Dom Croze-Brunet 2016 Lidl 5.49 €
Côtes du Rhône Vill. Dom la Tour des Abbesses, Plan de Dieu 2017 Intermarché 4.49 €
Gaillac Gd réserve de Labastide de Lévis 2016 Aldi 3.99 €

Visiting the Val du Séran

By Jon North (noreply@blogger.com)

We come to the Ain almost every year to play and sing music at Val du Séran, a huge converted farm building in the lovely countryside between the Bugey area and the upper Rhône.  It's quite isolated, surrounded by green hills with some of the cows that produce milk for Comté cheese, and with more distant views of the pre-Alpine mountains.  But apart from the stimulating guidance of our host Stéphane and the marvellous cooking of his wife Chantal, we always enjoy the wines they choose to accompany our evening meals, usually showing a sure touch  in matching food and wine.

This year, for example we've had a Bordeaux Clairet accompanying tarragon chicken.  We came across this on our trip to Bordeaux earlier this year - a light-coloured red wine (like a dark rosé) usually made from merlot grapes.  Then, a Californian cabernet sauvignon full of sweet vanilla fruit in a style which is definitely not French, a really good match with a scented but mild beef curry; a smooth Fitou (from the area south of Narbonne, not far from us) with a veal and ginger dish; a beautiful red Côte du Rhône  with barbecued rissoles made from a Croatian recipe; and a Fleurie (one of the Beaujolais crûs, usually regarded as relatively light and floral) with a pork mignon served with a haricot purée.  A couple more nights of this before we return to our own less diverse choices at home.

When we come to this area, we also enjoy the local wines, particularly the white Roussette de Savoie and Seyssel, made from local grape varieties Altesse and Molette.  These are lively flavourful wines which work well as apéritifs, as Mary proves most evenings after a full day of cello playing.  AT Stéphane's recommendation we visit the maker Bernard Aimé in the village of Corbonod near the town of Seyssel itelf and close to the river Rhône which is very picturesque in this area.

We have good lunches here of salads, cheese and fruit, but (unlike our home habit) no alcohol at lunchtime so that the music is not marred by sleepiness or lack of attention!  But the evenings are a time to relax and feast, as you can see!

Living with pain

By noreply@blogger.com (Jon North)

The past year for me has seen a slow ratcheting up of pain from sciatica.  Before that I would not have believed its all-consuming force.  Even so, I am well aware that I am lucky not to have worse, and the exercise of trying to stay positive is both constructive and interesting.  When we arrived in France Mary spent months with severe tendinitis in a shoulder which she would otherwise have used to bow her cello - upsetting and frustrating but ultimately thank goodness cured.  Since then we have both had joint pain, and sod's law says that at our age this is usually in a place linked to your most creative and important activity.

Treating pain is a complex obstacle course full of blind alleys and treatments which have worked for other people, enthusiastically advocated, but they seem to have no effect when you try them yourself.  Ultimately painkillers are OK, but they kill more than the pain you are trying to treat, and if they do not upset the digestive system or attack the stomach lining they are more or less addictive.  I take them anyway, and they work at least for periods of time.  'Alternative' treatments work if you believe in them, and I tend not to which makes me a poor patient!

With the sciatica I have also been through the gamut of steroid treatments which are very good for a day or three, then just stop working, back to agonising mornings.  Surgery is something I've had with one knee replacement, with mixed results though I can walk - it is like having a posh new hinge on a door, but someone forgot to oil it properly.  Since sciatica involves eroded discs in the spine it is another level of complexity and risk altogether, and I'll go some distance to avoid it despite our appreciation of the talents of Supervet Noel Fitzpatrick.  The animals he treats are after all much less heavy than I am!

But exercise and physical treatment are another matter.  I enjoy cycling, and riding my bike around our lovely flat town or using my exercise bike at home are both constant parts of my life.  The pain goes down, and either I can read and listen to music, or enjoy passing scenery, whhile I finish a session or ride with less.

I've just got back from my latest session with a physiotherapist.  I have had several of these over the past 5 years or so, before and after knee surgery, and my expectations were low.  I'm not the only person, patient or doctor, who recognises the description of being left hitched to a machine (pulley, electrodes, whatever) for 20 minutes while your therapist attends to 2 or 3 other clients.  An hour in a treatment room with only 10 minutes' direct treatment is not uncommon.

I'm not writing to moan about bad examples of physiotherapists because this time I think I have finally found a good one.  A young man who spent 29 mins of the allotted half hour with me, pushed and pulled me both to check my limits and push them a bit, and most importantly left me with exercises I should be doing several times a day at home.  To be fair to the last man he proposed something similar, but not very well explained.  I know now what stretching I need to do and how often, and I'll do it even though my rubber mat on the floor is less convenient and comfortable than his therapy couch.

I'll see him regularly over the next 6 weeks or so.  But in the end the answer is going to be in my own hands, or legs maybe - profiting from my enjoyment of cycling outdoors and in, putting together the advice of my GP, the rheumatologist, an osteopath who tweaked my vertebrae once, and my new physiotherapist that the best treatment is going to be more exercise.  And overcoming my innate laziness to get down on the floor and do the spinal stretches every day.

Incidentally, I was puzzled about the link between the term rheumatology, all things connected with rheumatism, arthritish and pains generally in joints and muscles, and the original French/Latin root rhume meaning cold or sniffles.  Apparently ancient medicine regarded these painful conditions as linked to watery humours - who knew?  The Oxford Dictionary certainly did and does!

At the same time I remember all the time that other people (family, friends and those I meet around the place) have more serious difficulties.  I was reminded of this again this morning seeing others arriving at the physio centre as I left to ride my bike home in the sunshine.

Reading

By noreply@blogger.com (Jon North)

Well here I am again, the inconstant blogger.  I have more time to write just now, and more time to read, so this will be about reading, and specifically in praise of the Kindle.  I've nothing against other brands but I'm not the BBC so need not display impartiality and I have little experience of other electronic readers.

I am a librarian and booklover, so of course I understand and share many people's feelings that Real Books are still of value.  Some would say they are far better - Mary usually reads them, and I often do, but often I turn to my Kindle and here's why.

First of all, I read in bed.  A heavy book can be difficult to manage, and if like me you often read when your partner is asleep, a Kindle is light enough to hold in one hand and its inbuilt light is a boon.

Secondly, and I appreciate this as a librarian, you can search text.  As I get older I often lose track of the names of people and places, and find it helpful to check back to the first or a previous reference.  Who was that?  When did the place crop up or what did the person do?  On a Kindle I can check quickly back and resume reading.

Thirdly, in my frequent need to read in a foreign language (my French is improving, but slowly) if I come across a word I don't remember or have never seen before I can quickly check its meaning through the built-in dictionary.  This is a real boon, and there is a way of noting newly discovered words in a vocabulary builder which, after a while, is a good read on its own!

Fourthly I get my daily paper first thing in the morning.  Paper copies are among the most famously cluttering things in many households I have known, often piled high in garages and spare rooms when their owners cannot bear to throw them out 'in case', but they are scarcely ever re-read after a day or two.  The online papers are not there for ever but for a couple of weeks you can search, then clip and store things that interest you.

People who cling fiercely to books and never read electronic text talk of the feel and form of a real book.  I share some of this feeling, especially having worked with older out of print books, but I realise that these become fragile and should finally be conserved and consulted via scanned copies or they will be damaged beyond repair.

I don't think I shall ever stop liking having books around, but they take space and collect dust - logically a Kindle is more environmentally friendly option, and it's amazing to think that there can be scores, hundreds, of books available in one small device, more easily stored and downloaded at will.  And as travelling with heavy books is ever harder, a single device with months of good reading in it is a welcome asset.

Thinking back to our holiday in the Caucasus

By noreply@blogger.com (Jon North)


I have been wondering, thinking more or less non-stop, and dreaming as well, how to round up our extraordinary fortnight beyond the Black Sea and on the boundary between Europe and Asia.  We went knowing so little, and we have found out so much about these extraordinary small countries in the mountainous triangle between the Russian, Turkish and Iranian great powers.

We visited Armenia and Georgia, for only a week each, and could hardly have contemplated visiting Azerbaijan which is the third of the trio because its relations with Armenia are so poor.  Indeed, on a trip north and wesst around Armenia we were advised to avoid the road near the western border in case of snipers.  Since our return we have reflected as much on the shared history of the countries we visited as on the differences which struck us most forcibly during our visit.

It was only as we started to read good guidebooks (we'd especially recommend the Bradt Guide to Armenia by Deirdre Holding, and the Georgia companion by Tim Burford (Bradt also) is not bad either), and then discovered a wonderful history and contemporary survey The Caucasus: an introduction by Thomas de Waal, that we began to understand the interwoven complexities of this fascinating area.

The Genocide Memorial in Yerevan
Armenia, our first port of call and now a landlocked country of only 3+ million people, was perhaps historically the most significant, having for centuries stretched from the eastern Mediterranean through Turkey across almost the whole of the area now divided between Armenia, Georgia and Azerbaijan.  Its current identity is marked by loss - loss of territory to the Turks (who now claim the sacred Mount Ararat once the centre of the Armenian highlands from which the ethnic group originates), loss of people to the awful genocide of the early 20th century, and loss of stability in the face of war (still smouldering with Azerbaijan over the disputed territory of Nagorny Karabach) and of natural disaster in the shape of earthquakes, since the area is on a geological fault line.   One town, Spitak, the size of Lunel was almost destroyed in 1988 and most of its inhabitants killed.  The damage was still all too evident, with temporary homes in containers and railway carriages, as we travelled through the area.

Our second week in Georgia was primarily to meet again our friends Leo and Marika, whom we'd hosted in Wirksworth in the early 2000s.  We added organised tours to fill our time during the week when they had to work.  We had a quaint but comfortable hotel in the old town of Tblisi, whence we could walk around the centre and visit museums and churches, and even attend a marionette show in the charming puppet theatre (in Georgian but with English surtitles, a very entertaining folk tale that ended up with the principal characters in Paris!).
Pirosmani's Doctor on a donkey

Karen Hakobyan's instruments
Highlights of our visit included artistic visits to the beautiful small museum in Yerevan dedicated to the well-known film director Sergei Parjanov - we loved it and went back to take in more detail before leaving Armenia - and the beautiful small gallery of the artist Niko Pirosmani in Signagi.  We also met Karen Hakobyan, a maker of traditional musical instruments including the duduk, in Yerevan - a real inspiration, and I have an alto shvi, a traditional fipple flute, as a lasting memento.  The instruments are made of stained apricot wood, the traditional material.

The separate identities of the two countries were really only crystallised under Tsarist and particularly Soviet influence, one effect of which was to precipitate huge population shifts to create 3 ethnically distinct countries (it would have been diffficult if not impossible to visit Azerbaijan even if we had had time).  I read that at one time in the 19th century almost no ethnic Armenians lived in present-day Armenia, huge numbers residing in what is now Iran and many also living in Georgia (our Armenian driver Ashot, for example, was brought up in Tblisi).

But communist rule (and particularly the longs shadows of Stalin and Beria) served to sow distrust between ethnic groups which separated like oil and water.  Armenia now has a population 98% ethnically Armenian, and although Tblisi has a more cosmopolitan feel (with a mosque where Shia and Sunni worship together and Christian churches of all the various traditions), there is a feeling of bravado and of Georgian national pride as strong in its way as the Armenian one.

The distrust, and the story of the south Caucasus, revolves round ethnic and cultural minorities, not only in Nagorny Kharabakh where de facto Armenian control has left a seemingly permanent impasse between the opposed positions of Azerbaijan and Armenia that have resisted repeated international conferences all over the world and which makes any reconciliation between Armenia and Turkey even more difficult than the genocide left it, but in the north of Georgia where two autonomous regions, Abkhazia and South Ossetia,, have been set up with Russian support.  The history of these two areas is long and complicated, but it has left most Georgians with a deep distrust of Russia and a strong preference for territorial integrity.  This in turn means they support Azerbaijan in claiming back Nagorny Kharabakh which sets Georgia at odds with Armenia, although luckily the antipathy is not extgreme so you can still travel as we did from one to the other, and Armenian commerce has access to the Black sea through the southern province of Georgia where there is still a high Armenian population that has resisted ethnic cleansing.

Traditional bread-making in Garni, Armenia
Two final things for now.  First food and drink.  We had excellent food throughout, with lots of fresh fruits, vegetables and salads particularly in Armenia and some lovely soups in Georgia as well as the ubiquitous and stomach-lining khachapuri, variations on warm flat bread filled with cheese, one delicious variant of which has an egg lightly poached on top.  The Armenian wine industry is less developed than the Georgian but both use indigenous grape varieities to good effect, particulalry the red Areni (centred round a village of that name in the south) in Armenia and some good dry whites in Georgia.  But Georgia also prides itself on good sweet red and white wines made from late-picked grapes.  And then there is Armenian brandy, reputedly Churchill's favourite tipple, and the ubiquitous vodka which testifies to the strength of the Soviet influence.

Drink at the Botanic Gardens, Tblisi
Roadside food stall in Tblisi old town
Wine tasting at Château Mukhrani west of Tblisi
Second, Stalin.  An odd note to finish on, maybe, and our Georgian friends doubted if we really needed to visit the museum in Gori which he himself established.  The little town still retains his statue in hte centre, and his childhoold house is preserved under a kind of huge bus shelter.  He adopted the last Tsar's railway carriage to get around, more or less without modification.  But we were helped by a suitably irreverent English-speaking guide, himself a native of Gori.  Above all this was a reminder (in a world where Brexit and Trump remind us how near the surface divisive extremism can be) that the fragile free countries we visited are there partly as a result of Stalin who, for better or worse, permitted the development of separate states with separate identities, and partly despite the cruel bloodshed that has surrounded them throughout their history and virtually until the present.

View over Tblisi to the mountains north-west of the city
Perhaps the most powerful influences that have shaped them and allowed them to survive and develop, though, are the mountains - the High Caucasus to the north which so hampered easy Russian access to the southern countries, the mountains between Turkey and Armenia which while sacred to the latter are also a brake on easy passage, and the mountains within the countries which have allowed separatist enclaves to survive.  It also makes for lovely scenery for us tourists, even if there is a risk of the early snow we ran into twice in our fortnight.

The larger Mount Ararat with Khor Virap monastery in the foreground
Mount Ararat from the 7th century circular cathedral of Zvartnots south of Yerevan


A week in Georgia

By noreply@blogger.com (Jon North)


 I had hoped to write this sooner, but getting back home I had to catch up on life in France, so this has been the first chance I've had to publish this post about our second week away.

It was of course the original reason for our Caucasus trip - to visit the couple of Georgian friends Leo and Marika whom we had hosted while Leo was a postgraduate architecture student in Nottingham.  But they both have to work, so having spent a weekend with them we joined up with the guide Eva (also seen here) and driver Ilya for a few days' tours to places outside the capital.  The weather turned wet and cold until our last day, which spoiled a lot of views, but we still saw a lot.

Our hotel in the Old Town was comfortable and convenient for walking around the tourist centre of Tblisi, which we did often during our week's stay.  The travel agent had arranged daytime trips during the week, but over the weekend Leo drove us east to an area, Khakheti, which produces most of the country's wine.  We visited monasteries and a town wine festival, and also a gallery including works by Pirosmani, one of the country's most popular modern artists who liked his wine as you can see).

                                   

                          

We spent the week, while Marika and Leo were at work, exploring the area west of Tblisi, including the old capital of Georgia and its cathedral, a monatery overlooking a lake and a snowy mountain pass, then later the château Mukhari, a winery of some quality.

Probably the outstanding visit of the week was to Mtskheta, the old capital of Georgia, which is just west of Tblisi at the confluence of 2 rivers.  The 6th century Jvari monastery is on a promontory overlooking the rivers, and the 11th century cathedral of Mtskheta just below.  Both were impressive - photos to finish this brief post, and more to come as I reflect on the complex, intertwined history and culture of the two countries we visited over this memorable fortnight.


 




















A week in Armenia 14-20 October 2016

By noreply@blogger.com (Jon North)

We'd thought of visiting our friends in Georgia for a few years, and Mary finally made the arrangements this summer as we celebrate 40 years together.  She found a travel agent in Montpellier which does Asian tours including an Armenian trip with an extension into Georgia, and since the season is nearly over they offered us an individual fortnight's holiday for 2.  This week we were in Armenia, and next we'll be in Georgia whence we fly home to France.

So this is a bit about our first week.  The best story is told through photos, but although I can easily put these on Facebook from this tablet, getting them into a blog will be much easier once I'm home with my laptop.  For now, some verbal impressions.  We travelled to Yerevan via Vienna, arriving in the early hours of Friday last.  Our guide Marietta and driver Ashot met us at the airport and have accompanied daily with friendly and professional care - they usually leave us evenings to recover, but this evening they've invited us for a farewell meal before they both accompany us to the airport at 5 tomorrow morning.

We have spent most of the time based in Yerevan, a sprawling mixture of Soviet blocks, shacks, building sites and splendid modern plazas and developments, home to around a million people, with very busy roads, and few bikes or motor bikes.  Roads outside the capital, and even back roads in Yerevan, are mostly in a poor state, and Ashot has needed all his skill to keep us out of most of the potholes in his Mercedes minibus.

Marietta, who is proud of her French, is even prouder of her country and people.  In fact, many people we met were anxious to get our opinion and approval of Armenia.  We had little real knowledge of it before we came, and only found a really good guidebook (in the Bradt guides series - http://www.bradtguides.com/destinations/europe/armenia.html) after our arrival, but we rapidly realised that this small nation with its quirky and unique alphabet derives its intense national pride from its history, up to the present day, of defining itself in the melting pot of the Caucasus, between the powerful political jaws of Russia and Turkey and as a Christian country under pressure from Muslim neighbours.

We have visited Christian heritage in many monasteries and saw incredible and ancient remains including countless lovely carvings, and we witnessed briefly the Sunday mass and beautiful singing at the cathedral.  We discovered local traditions - musical instruments like the doudouk, bread making - and extraordinary artists like Sergei Paradjanov.  We have been to wine and brandy tastings and learnt about local grape varieties.  And we have eaten good food, based hugely around fresh fruit and veg and salads and cooked vegetables prepared in countless ways, interesting local cheeses, creams and yoghurts, plus nice meat especially in stuffings for vine leaves.  And drunk a lot of good coffee.

We have ventured out of Yerevan to do all this on day excursions, mostly on day excursions but once on a 2-day trip via the huge lake Sevan north near the Azeri and Georgian borders, a journey plagued by rain then snow which forced us to alter our route, all of which Ashot handled well despite dropping temperatures.  On that trip we were reminded of the latest catastrophic earthquake in the north-west - Armenia lies on a tectonic fault-line.  Tragedies near the epicentre were heightened by poor Soviet-era building standards so that whole buildings collapsed killing thousands.

Such tragedies, and wars, pale into insignificance beside the 1915 genocide which has occupied our minds increasingly towards the end of our visit.  Wikipedia and the official genocide memorial website give much valuable information but nothing quite prepares you for the terrible graphic reality of the museum exhibition witnessing to the enforced death of over a million Armenians expelled from their homes and neighbourhoods.

I could go on but it will suffice to finish with our own testimony to a friendly and hospitable people despite, perhaps partly because of the tragedies and economic hardship they have suffered and despite the difficulties many still face.  We are happy to do our small part to make this place and these people better-known.

Why is the NHS listening to the siren voices of the vape manufacturers? | Victoria Coren Mitchell

By Victoria Coren Mitchell

The calls to make e-cigarettes free on prescription are outrageous when an actual cure for smoking is available

Public Health England has called for e-cigarettes to be made available on the NHS. This makes me so angry that I want to have a fag, although the last one I had was on 27 September 2014.

Ah well. I’ll settle for eating another packet of Minstrels and typing furiously.

Continue reading...

The checkout free shop is a wonderful idea, a machine will never judge you | Victoria Coren Mitchell

By Victoria Coren Mitchell

When we no longer have to impress the cashier – or fellow shoppers – we can buy what we like

Are you frightened of the rising machines? I try not to be. Machines are the future and being horrified by the future is so terribly ageing. Banging on about the misery of automated switchboards, the insecurity of online banking or the impersonality of email puts 20 years on you immediately, like racism or natural light. I try to avoid such things.

So, for me, it’s all “Good news, my local post office has shut down!”, “Ooh, you need a ‘registered account’ to buy cinema tickets, I couldn’t be happier!” and “Hurray! A leaked NHS England report says 111 calls will soon be diverted to a ‘diagnosis app’ instead of a person!”

Continue reading...

I did lose a pair of trousers once but they weren’t worth £1m, Boris | Victoria Coren Mitchell

By Victoria Coren Mitchell

The bankrupt tennis star’s admission that he has mislaid his trophies resonates with all of us who know we put that thing somewhere…

A s I sat with Boris Becker in the Riviera sunshine, each of us clutching a cigarette in one hand and a doughnut in the other, I thought: “This is my kind of sportsman.”

I didn’t know Boris Becker very well but I liked him enormously. Clearly, we both enjoyed the taste of a sugary butt on a spring afternoon. Also, we both loved a hand of cards; we were in Monte Carlo for a €10,000 poker tournament. I always found him approachable, friendly, unpretentious and nice to be around.

Continue reading...

Comparing unlike with unlike – it’s Whitehall’s secret new parlour game | Victoria Coren Mitchell

By Victoria Coren Mitchell

If the culture minister says that ambassadors should be paid more than BBC editors, we should next ask him how to choose between Victoria Beckham and a leopard

Well done Peston On Sunday, last week, for having three interview guests and triggering major news stories with each of them. That’s an amazing hit rate.

First guest Jeremy Corbyn hit the headlines for suggesting that the UK doesn’t really have a special relationship with America. Third guest Miriam Margolyes said “fuck” live on air. And the middle guest, shiny new culture secretary Matt Hancock, said that editors at the BBC should not be paid more than ambassadors.

Continue reading...

Molly tries to shuffle the pack | Victoria Coren Mitchell

By Victoria Coren Mitchell

The more things change the more they stay the same – even in a film about a strong woman

Can Hollywood fix itself? Is that already happening? Let’s go to the cinema together and find out.

It’ll have to be my local Everyman – a genteel chain where they transmit a lot of productions live from the National Theatre and sell yoghurt-coated nuts instead of Minstrels. Might not be your cup of tea. On the plus side, you can also get a cup of tea. It has to be that venue, because the trip has already happened.

Continue reading...

My Christmas present to you

By Victoria Coren Mitchell

Hate gift guides? Here’s an alternative to all those that insult you with dull ideas

The gift guides are out again! Regular readers will know how infuriated I am by those Christmas gift guides. Pages and pages of newsprint, all given over to the advice that we should consider buying our relatives a pair of socks or a bottle of wine.

Adding insult to insult, they invariably divide these “ideas” into relatives (“A lipstick for your wife! A book for your mum! A bottle of wine for Grandpa!”) just to make sure that nobody shops beyond the boundaries of age and gender stereotype. Not only must we buy the same old stuff every year, we must make the same old assumptions. Women love clothes. Kids love sweets. Men love golf calendars.

Continue reading...

Ed Sheeran versus the super-idiots | Victoria Coren Mitchell

By Victoria Coren Mitchell

Attacking the singer’s charity efforts takes a rare type of stupidity. But guess what...

According to a Dropbox survey published last week, most people believe that “only 68% of their work colleagues” are capable of the job.

This is a staggering figure. Why so high? Nobody’s capable of the job. Nobody’s capable of anything.

Related: Ed Sheeran Comic Relief film branded 'poverty porn' by aid watchdog

Continue reading...

Michael Gove, where did our love go? | Victoria Coren Mitchell

By Victoria Coren Mitchell

The environment secretary has shown a flash of his old character but I can’t forgive his betrayal

Did you wake up on Thursday morning expecting to feel a wave of affection for Michael Gove? I didn’t. I’m not sure anyone did. Possibly not even Mrs Gove, who wrote a very interesting joke in her Daily Mail column the day before.

“Kevin Spacey and Harvey Weinstein are reportedly languishing in rehab at an upmarket sex addiction clinic in Arizona,” it began. “No sex for eight weeks, apparently – although quite why they needed to fly to the desert for that is beyond me. Most of us find getting married does the trick.”

Continue reading...

Why are the police copping flak? | Victoria Coren Mitchell

By Victoria Coren Mitchell

Don’t damn officers for a Halloween prank – law and order should show it has bit of heart

It is possible that I have many things in common with Fenland police. You can’t do Only Connect for as long as I have without knowing that one thing can always be linked to another. Four things can be a challenge, but a simple pair, such as Fenland police and me… off the top of my head: we both like to wear blue, we’ve both dealt with a lot of drunks and neither of us has managed to stamp out hare-coursing in the county of Cambridgeshire.

And here’s another: we both tweeted pictures of ourselves dressed up for Halloween.

Continue reading...

Casinos gamble on their credibility | Victoria Coren Mitchell

By Victoria Coren Mitchell

If you’re too smart for gaming houses they’ll find a way to stop you - but more fool them

It is rare to see Phil Ivey, the greatest poker player of our time, losing seriously. This man is a genius. He can get inside other people’s heads.

The first time I played poker against him, I think he found me a little unsettling. People do, the first time. In Phil’s case, I don’t think it’s just that I was female – which is what throws most people – but that I was female and making jokes.

Continue reading...

Faster, higher, twerkier? | Victoria Coren Mitchell

By Victoria Coren Mitchell

Why stop at pole dancing at the Olympics? Let’s have groping and mud wrestling as well

The news that pole dancing has been formally recognised as a sport – and will now be considered for possible inclusion in the Olympics – fills me with delight.

Regular readers may be surprised. You might imagine I would feel weary and suspicious at this development. You might imagine I’d roll my eyes and ask: “What next? A simultaneous men’s event – how many bills can you shove in her bra as she writhes?”

Continue reading...

I’m really not a petrolhead... | Victoria Coren Mitchell

By Victoria Coren Mitchell

... but the thought of driverless cars and the surrender of freedom fill me with gloom

God bless the women of Saudi Arabia and their excitement about a royal decree allowing them to hold driving licences at last.

As we sit in traffic jams, fuming about inexplicable delays and unending roadworks, terrible radio playlists, the utter monotony and hell of it all, we should think of our sisters in the desert who see only the freedom, power and joy.

Continue reading...

Putting Granny online? No thanks| Victoria Coren Mitchell

By Victoria Coren Mitchell

I’m all for doing something for old people. But don’t ask me to put family snaps on the web

Today is National Grandparents’ Day. AgeUK is asking people to tweet, Instagram, hashtag (and other words your granny would not have understood) a photograph of their beloved ancestors, marked #grandpics, with a suggested donation of £5 with every photograph.

They asked me directly to do this. I said no. F*** em, I said.

Continue reading...

Oh, do let’s be beastly to the Nazis | Victoria Coren Mitchell

By Victoria Coren Mitchell

Nobody thinks Paul Hollywood is an actual Nazi; we should let him dress how he likes

A Nazi goes into a pub.

Hang on… that’s not a Nazi! It’s the well-known baker and TV personality Paul Hollywood!

Continue reading...

High street betting must clean up its act | Victoria Coren Mitchell

By Victoria Coren Mitchell

My column about the evils of fixed-odd betting terminals brought almost universal agreement but a few voiced their dissent. Let me answer my critics

Since my column last week about fixed-odds betting terminals, I’ve been getting hundreds of messages a day. Ninety-five percent of them agree with my view that these high-stakes modern slot machines should be restricted. I don’t think I’ve ever written a column that encountered more assent – or less dissent – apart from the one about “funeral crashers” who skulked around misleading the bereaved in hope of free booze. Not many people came out in support of those guys.

But FOBT machines are equal to the funeral crashers in their rapacious appetites, and inspired almost as little praise. The unanimity of the response was heartening and baffling at the same time: left-wingers and right-wingers, Labour MPs and Tory MPs, betting-shop workers, betting-shop punters and people who have never been in a betting shop; old and young, the addicts and the free; all united in agreement that the machines should be capped.

Continue reading...

Weekly Update 167

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

It's summer! Yes, I know it's back to front for many of you but Dec 1 means it's sunnier than ever here. Regardless, this week I've been at DDD in Brisbane, written my 10 year old son Ari and I running kids coding clubs in Oslo (cold) and London (rainy)

Welcoming the Swiss Government to Have I Been Pwned

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

I recently had the pleasure of spending a few days in Switzerland, firstly in Geneva visiting (and speaking at) CERN followed by a visit to the nation's capital, Bern. There I spent some time with a delegation of the National Cybersecurity Centre discussing the challenges they face and where HIBP

Teach Your Kids to Code with Ari in Oslo and London

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

When I first started writing code a few decades ago, it was a rather bland affair involving a basic text editor and physical books for reference. I didn't have an opportunity to create anything usable by others until years later and perhaps most importantly in the context of this blog

Weekly Update 166

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

Kangaroos! I've been trying to line these guys up for weeks to no avail but finally, they've delivered. Speaking of delivering, I actually got 3 blog posts out this week which I've not done for a while, the most significant of which relates to "data enrichment" companies (also often referred

Data Enrichment, People Data Labs and Another 622M Email Addresses

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

Until this month, I'd never heard of People Data Labs (PDL). I'd certainly heard of the sector they operate in - "Data Enrichment" - but I'd never heard of the company itself. I've become more familiar with this sector over recent years due to the frequency with which it's been

When Bank Communication is Indistinguishable from Phishing Attacks

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

You know how banks really, really want to avoid their customers falling victim to phishing scams? And how they put a heap of effort into education to warn folks about the hallmarks of phishing scams? And how banks are the shining beacons of light when it comes to demonstrating security

Welcoming the Norwegian Government to HIBP

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

Over the last couple of years, I've been increasingly providing governments with better access to their departments' data exposed in breaches by giving them free and unfettered API access to their domains. As I've been travelling around the world this year, I've been carving out time to spend with governments

Weekly Update 165

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

Yes, I'm in my car. I'm completely disorganised, rushing to the next event and really didn't plan this very well. But hey, what an awesome little soundproof booth it is! That said, I did keep this week deliberately concise... until I went to edit it and then Adobe Premiere (or

Weekly Update 164

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

It's a late, early in the day, hazy, bush-firey Aussie weekly update with a whole bunch of various bits and pieces of interest from throughout the week. The references below will give you a sense of how much I've jammed into this week so I won't repeat it all here

HSTS From Top to Bottom or GTFO

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

We're pretty much at a "secure by default" internet these days, at least that's the assumption with most websites, particularly so in the financial sector. About 80% of all web pages are loaded over an HTTPS connection, browsers are increasingly naggy when anything isn't HTTPS and it's never been cheaper

Weekly Update 163

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

It's been a pretty full week this one with a couple of talks in Sydney followed by another in Melbourne. Then, to top it all off, getting sick hasn't helped and oh boy did this one hurt. Good news is that even just a few hours after recording this video

Weekly Update 162

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

Ah, impending summer on the Gold Coast! It's that time of year when you can just start to sense those warm beach days and it's absolutely my favourite time of year here. Which means... it's time to head off to other events again. Fortunately it's all domestic this time as

Weekly Update 161

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

It's my first conference back in Australia since probably about May and I'm experiencing a rare luxury - not flying! I'm sticking to driving some big distances just to get a break from the tyranny that is check-in, security and airport lounges. Seriously, it was beginning to do my head

Weekly Update 160

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

Australia! Geez it's nice to sit amongst the gum trees and listen to the birds, even if it's right in the middle of some fairly miserable weather. I'll continue to be here for the foreseeable future too, at least in one state or another. But being back here hasn't stopped

Weekly Update 159

By Troy Hunt

Presently sponsored by: Whois XML API: A lack of domain intelligence causes data breaches. Test our Security Enterprise API & Data Feed packages with free credits!

Well, this will be the last weekly update done overseas for some time as I count down the return to beaches, sunshine and fantastic coffee (yes, I'm confident saying that even whilst in Italy!) It's been a non-stop trip with an attempt of a bit of downtime at the end

The Housing Podcast: what is exempt accommodation and why is it becoming a problem?

By Lucie Heath

The Housing Podcast is a production of Inside Housing magazine, the UK’s leading publication for the social and affordable housing sector. Listen to find out more about the key issues in housing today, with input from the sector’s leading voices

Scotland unveils new fire safety guidance for high-rise owners and residents

By James Wilmore

New fire safety guidance for owners and residents of high-rise buildings in Scotland has been published in an effort to highlight the risks and prevent a tragedy like Grenfell.  

Right to Buy sales fall to lowest level in six years

By James Wilmore

Right to Buy sales of council housing stock have fallen to their lowest level since 2012/13, official figures have revealed. 

Five things we learned from the Housing Hustings

By Nathaniel Barker

Wednesday night saw spokespeople from across the country’s major parties brought to the Royal Institute of British Architecture’s headquarters to take part in a housing hustings. Inside Housing’s Nathaniel Barker was there to note down what we learned

Whether politicians talk about it or not, voters have not forgotten the housing crisis

By Peter Apps

Housing has not grabbed the headlines in this election, but it keeps coming up on the doorstep. Politicians have missed a trick by not making it a frontline issue, writes Peter Apps

London fire chief to step down four months early amid Grenfell criticism

By James Wilmore

London Fire Commissioner Dany Cotton is to step down earlier than planned amid reports of pressure from families bereaved after the Grenfell Tower tragedy.

Short on time? Friday’s housing news in five minutes

By Jack Simpson

A round-up of the top stories this morning from Inside Housing and elsewhere

6 December digital edition of Inside Housing out now

By Inside Housing

The 29 November digital edition of Inside Housing is now available to subscribers

Orbit joins list of housing associations ditching fixed-term tenancies for lifetime agreements

By James Wilmore

43,000-home association becomes the latest to move away from fixed-term tenancies

General election 2019’s housing battlegrounds: South Shields

By Jack Simpson

While the big topic in this election is Brexit, the housing crisis remains a top issue for voters in many areas. Inside Housing has travelled around the country to visit the marginal seats where it may influence the result. In the final part in our series of five, Jack Simpson visits South Shields

Regulator blocked exempt accommodation providers from registering

By Nathaniel Barker

The Regulator of Social Housing (RSH) has previously blocked organisations seeking to provide “exempt accommodation” from registering as housing associations, Inside Housing has learned.

Housing associations looking to offload shared ownership portfolios to institutional investors, says Savills

By Lucie Heath

Housing associations are expressing interest in selling off shared ownership portfolios to free up cash to invest in existing homes and new supply, according to Savills.

Our resident safety campaign winners announced

By Inside Housing

What landlords can learn from the winning approaches to working with residents to raise awareness of safety issues

General election 2019’s housing battlegrounds: Gower

By Lucie Heath

While the big topic in this election is Brexit, the housing crisis remains a top issue for voters in many areas. Inside Housing has travelled around the country to visit the marginal seats where it may influence the result. In the fourth in our series of five, Lucie Heath visits Gower

Government could prevent poor-quality houses being built, says Welsh housing minister

By Lucie Heath

Welsh ministers may intervene in planning applications that are not of high quality or designed to create sustainable communities, the housing and local government minister has said.

US and Iranian men released in prisoner swap

Xiyue Wang was jailed in Iran in 2016 while Massoud Soleimani was arrested at a US airport last year.

Everton 3-1 Chelsea: Duncan Ferguson 'loved' debut as interim manager

Duncan Ferguson says he's "here for Everton for as long as they need me" after their 3-1 win over Chelsea in his first match as interim manager.

Everton beat Chelsea in Ferguson's first game in charge

A much-improved Everton hustle their way to victory over Chelsea in caretaker manager Duncan Ferguson's first match in charge of the Toffees.

General election 2019: Lib Dems pledge help for small business

Jo Swinson's party would scrap business rates and provide greater support for entrepreneurs.

Albanian earthquake: Ronaldo and Buffon meet young survivors

Boys who survived an earthquake in Albania by jumping from a balcony meet stars from Italy's Juventus.

South Western Railway strike: Engineering adds to weekend woe

RMT union members are taking 27 days of strike action on South Western Railway.

General election 2019: Reddit says UK-US trade talks document leak 'linked to Russia'

Jeremy Corbyn claimed the papers proved "the NHS is for sale" when he highlighted them at a press conference.

Ammanford memorial bench for town's beloved dog

Coley the collie's death caused an outpouring of grief, with flowers left for more than seven weeks.

Anderson, Bairstow & Wood return for England's South Africa tour

James Anderson, Jonny Bairstow and Mark Wood return to the England Test squad for the tour of South Africa starting this month.

'Heartbroken' girl wants stolen therapy dog back for Christmas

Stolen Lottie is a therapy dog and "best friend" to 11-year-old Chloe Hopkins.

Revived Briton Audrey Schoeman 'lucky to have second chance'

Hiker Audrey Schoeman's heart stopped beating for six hours - but doctors saved her life.

Peter Pan pantomime stars say it is 'hardest job in show business'

A children's TV star and a Strictly dancer will be putting in three performances a day.

Climate change: Oceans running out of oxygen as temperatures rise

A warmer world means oceans are able to hold less dissolved oxygen, which is bad news for many fish.

OpenBSD bugs, Microsoft's bad update, a new Nork hacking crew, and more

By Shaun Nichols

Meanwhile, the DOJ sets its sights on money mules

Welcome to yet another El Reg security roundup. Off we go.…

What happened this week?

Trump weighs in on the NHS, and the BBC invites Boris Johnson for a chat about trust.

Two malicious Python libraries caught stealing SSH and GPG keys

Two malicious Python libraries caught stealing SSH and GPG keys

Nasty. Two typosquatting libraries were spotted on PyPI - targetting dateutil and jellyfish but with tricky variants of their names. They attempted to exfiltrate SSH and GPG keys and send them to an IP address defined server. npm has seen this kind of activity too - it's important to consider this when installing packages.

flk: A LISP that runs wherever Bash is

flk: A LISP that runs wherever Bash is

This is a heck of a project: an implementation of LISP written entirely in Bash, meaning you can run it as a script on any machine that has a Bash installation.

Via Hacker News

Quoting Michael Lopp

Let’s agree that no matter what we call the situation that the humans who are elsewhere are at a professional disadvantage. There is a communication, culture, and context tax applied to the folks who are distributed. Your job as a leader to actively invest in reducing that tax.

Michael Lopp

datasette-atom: Define an Atom feed using a custom SQL query

I've been having a ton of fun iterating on www.niche-museums.com. I put together some notes on how the site works last week, and I've been taking advantage of the Thanksgiving break to continue exploring ways in which Datasette can be used to quickly build database-backed static websites.

I post a new museum to the site every day, so it was inevitable that someone would ask for a feed. And here it is: an Atom feed for Niche Museums.

This means Niche Museums is effectively a blog now, which is fitting: it's a universal truth that any sufficiently advanced backend technology will evolve to the point where it can power a blog with an Atom feed.

datasette-atom

I built the feed by wrapping up work on the first version of a new Datasette plugin: datasette-atom. It takes advantage of the register_output_renderer plugin hook, which was contributed by Russ Garrett back in May.

The idea with the plugin is to make it possible to construct an Atom feed from an arbitrary SQL query.

This is a really powerful ability. It means that a user with sufficent knowledge of SQL can subscribe to an arbitrary feed of data from any Datasette instance that is running the plugin.

Defining an Atom feed with a SQL query

The plugin works by requiring you to provide a SQL query that produces the following columns in its output:

atom_id, atom_title and atom_updated.

These correspond to the required entry elements defined by the Atom specification.

The plugin can then render the results of the query as an Atom feed.

You can also produce an atom_link column, which will become a link.

And finally, you can produce either an atom_content column which will be treated as text and used as the feed entry body, or an atom_content_html column which will be treated as HTML.

(The HTML from atom_content_html is sanitized through Mozilla's Bleach library to ensure the plugin doesn't act as an XSS vector.)

This means we can define a custom Atom feed by crafting a SQL query! Here's the query I'm using on the Niche Museums website:

select
  'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
  name as atom_title,
  created as atom_updated,
  'https://www.niche-museums.com/browse/museums/' || id as atom_link,
  coalesce(
    '<img src="' || photo_url || '?w=800&amp;h=400&amp;fit=crop&amp;auto=compress">',
    ''
  ) || '<p>' || description || '</p>' as atom_content_html
from
  museums
order by
  created desc
limit
  15

I'm using a couple of extra tricks here.

The atom_id is defined as a tag:uri following this advice from Mark Pilgrim - since created is an ISO 8601 timestamp substr(created, 0, 11) returns the YYYY-MM-DD component.

Not every museum has a photo, which means photo_url is sometimes null. In SQL, concatenating a null value to something else (using the || concatenation operator) produces another null. So this entire expression evaluates to null if photo_url is null:

'<img src="' || photo_url ||
'?w=800&amp;h=400&amp;fit=crop&amp;auto=compress">',

The coalesce(x...) function returns the first argument passed to it. So coalesce('<img ...>' || photo_url || '...', '') returns the empty string if the photo is not available.

Deriving created/updated fields from Git history

For the atom feed to work, I need an atom_updated value. This should be a timestamp representing "the last time the entry was modified in a significant way" - so it's actually more like a created timestamp for my museums website.

My museum data is defined in a YAML file - museums.yaml - which doesn't include created and updated timestamps. So where can I get them from?

Since the YAML file is stored in the site's GitHub repository, I'm deriving those timestamps from the git history. I repurposed code I wrote for my PG&E outages project for this - the full script is annotate _timestamps.py in the museums repo.

It works by looping through the entire history of the museums.yaml file comparing the list of museums in each version to the previous iteration. If a museum is new (it has an ID not seen before) we use the commit date as its created date. If the JSON serialization of the museum differs from the previous version we reset its updated date.

Then at the end the script uses sqlite-utils to update each record with the derived timestamps:

# updated/created are dicts of {"id": "timestamp"}
db = sqlite_utils.Database("browse.db")
for id, ts in created.items():
    db["museums"].update(id, {
        "created": ts,
        "updated": updated[id]
    }, alter=True)

The alter=True parameter to the .update() method causes sqlite-utils to automatically add any missing columns that are referenced in the update.

Defining a feed as a canned query

Now that we've defined the feed as a SQL query, we can assign it a more pleasing URL using Datasette's canned queries feature.

I encoded the query as a JSON string using JSON Escape Text, then added it to the metadata.json configuration file for Niche Museums. I named the query feed, resulting in a URL of www.niche-museums.com/browse/feed.

There's just one catch: Atom feeds should have a name. As a quick and nasty hack I allow the name to be set using ?_feed_name=Niche+Museums. I have an open issue to come up with a less nasty way of defining this.

Also this week

I added a simple search engine to Niche Museums! Here's an example search for "model": www.niche-museums.com/browse/search?q=model.

I used sqlite-utils to configure SQLite FTS (here's the line that calls it in the CI build script), defined another canned query in metadata.json that executes the query and built a custom template to render the results page.

I added press coverage to Niche Museums. Many of the listings now link to articles in the local media about them.

Museum pages now link to other nearby museums. Here's the commit that implemented that feature - it works by embedding a new SQL query in the template page, using datasette-template-sql.

I identified a bug in yaml-to-sqlite and shipped a fix in version 0.3.

I added the following seven museums to the site:

Goal for the next week: make some progress on projects that aren't related to niche museums!

Quoting Google Standard of Code Review

In general, reviewers should favor approving a CL [code review] once it is in a state where it definitely improves the overall code health of the system being worked on, even if the CL isn’t perfect.

Google Standard of Code Review

niche-museums.com, powered by Datasette

I just released a major upgrade to my www.niche-museums.com website (launched last month).

Notably, the site is entirely powered by Datasette. It's a heavily customized Datasette instance, making extensive use of custom templates and plugins.

It's a really fun experiment. I'm essentially using Datasette as a weird twist on a static site generator - no moving parts since the database is immutable but there's still stuff happening server-side to render the pages.

Continuous deployment

The site is entirely stateless and is published using Circle CI to a serverless hosting provider (currently Zeit Now v1, but I'll probably move it to Google Cloud Run in the near future.)

The site content - 46 museums and counting - lives in the museums.yaml file. I've been adding a new museum listing every day by editing the YAML file using Working Copy on my iPhone.

The build script runs automatically on every commit. It converts the YAML file into a SQLite database using my yaml-to-sqlite tool, then runs datasette publish now... to deploy the resulting database.

The full deployment command is as follows:

datasette publish now browse.db about.db \
    --token=$NOW_TOKEN \
    --alias=www.niche-museums.com \
    --name=niche-museums \
    --install=datasette-haversine \
    --install=datasette-pretty-json \
    --install=datasette-template-sql \
    --install=datasette-json-html \
    --install=datasette-cluster-map~=0.8 \
    --metadata=metadata.json \
    --template-dir=templates \
    --plugins-dir=plugins \
    --branch=master

There's a lot going on here.

browse.db is the SQLite database file that was built by running yaml-to-sqlite.

about.db is an empty database built using sqlite3 about.db '' - more on this later.

The --alias= option tells Zeit Now to alias that URL to the resulting deployment. This is the single biggest feature that I'm missing from Google Cloud Run at the moment. It's possible to point domains at deployments there but it's not nearly as easy to script.

The --install= options tell datasette publish which plugins should be installed on the resulting instance.

--metadata=, --template-dir= and --plugins-dir= are the options that customize the instance.

--branch=master means we always deploy the latest master of Datasette directly from GitHub, ignoring the most recent release to PyPI. This isn't strictly necessary here.

Customization

The site itself is built almost entirely using Datasette custom templates. I have four of them:

The about page uses a particularly devious hack.

Datasette doesn't have an easy way to create additional custom pages with URLs at the moment (without abusing the asgi_wrapper() hook, which is pretty low-level).

But... every attached database gets its own URL at /database-name.

So, to create the /about page I create an empty database called about.db using the sqlite3 about.db "" command. I serve that using Datasette, then create a custom template for that specific database using Datasette's template naming conventions.

I'll probably come up with a less grotesque way of doing this and bake it into Datasette in the future. For the moment this seems to work pretty well.

Plugins

The two key plugins here are datasette-haversine and datasette-template-sql.

datasette-haversine adds a custom SQL function to Datasette called haversine(), which calculates the haversine distance between two latitude/longitude points.

It's used by the SQL query which finds the nearest museums to the user.

This is very inefficient - it's essentially a brute-force approach which calculates that distance for every museum in the database and sorts them accordingly - but it will be years before I have enough museums listed for that to cause any kind of performance issue.

datasette-template-sql is the new plugin I described last week, made possible by Datasette dropping Python 3.5 support. It allows SQL queries to be executed directly from templates. I'm using it here to run the queries that power homepage.

I tried to get the site working just using code in the templates, but it got pretty messy. Instead, I took advantage of Datasette's --plugins-dir option, which causes Datasette to treat all Python modules in a specific directory as plugins and attempt to load them.

index_vars.py is a single custom plugin that I'm bundling with the site. It uses the extra_template_vars() plugin took to detect requests to the index page and inject some additional custom template variables based on values read from the querystring.

This ends up acting a little bit like a custom Django view function. It's a slightly weird pattern but again it does the job - and helps me further explore the potential of Datasette as a tool for powering websites in addition to just providing an API.

Weeknotes

This post is standing in for my regular weeknotes, because it represents most of what I achieved this last week. A few other bits and pieces:

Quoting Hyrum's Law

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

Hyrum's Law

How Do You Remove Unused CSS From a Site?

How Do You Remove Unused CSS From a Site?

Chris Coyier takes an exhaustive look at the current set of tools for automatically removing unused CSS, and finds that there's no magic bullet but you can get OK results if you use them carefully.

Via @chriscoyier

Weeknotes: datasette-template-sql

Last week I talked about wanting to take ona a larger Datasette project, and listed some candidates. I ended up pushing a big project that I hadn't listed there: the upgrade of Datasette to Python 3.8, which meant dropping support for Python 3.5 (thanks to incompatible dependencies).

Since Glitch now runs Python 3.7.5 my biggest reason for supporting 3.5 was gone, so I decided to make the upgrade.

Datasette 0.31 was the first version to drop support. Datasette 0.32 is the first to take advantage of it: I switched Datasette's template rendering over to use Jinja's async template support, which requires Python 3.6+.

This has exciting implications for the extra_template_vars plugin hook, which allows plugins to add extra variables (and functions) to the template scope.

Plugin authors can now add custom template functions that are defined with async def ... and make await calls within the body of the function. When the template is rendered, Jinja will automatically await those function calls.

I released a new plugin that takes advantage of this capability: datasette-template-sql. It lets you embed additional SQL queries directly in a custom Datasette template. For example:

{% for article in sql(
    "select headline, date, summary from articles order by date desc limit 5",
    "news"
) %}
    <h3>{{ article.headline }}</h2>
    <p class="date">{{ article.date }}</p>
    <p>{{ article.summary }}</p>
{% endfor %}

This new sql() function takes a SQL query and the optional name of the database to run the query against (in case you have more than one database file attached to your Datasette instance).

I'm really excited about this capability. I quipped about it on Twitter:

This is the great thing about having a plugin system: even if I'm not convinced this is the right technical direction for the core Datasette project, I can still expose this kind of feature in a plugin that people can opt into if they want to have this ability.

The official Datasette website is going to make extensive use of this plugin. I have an early prototype of that up and running now, which inspired me to release datasette-render-markdown 0.2 with a custom template function for rendering markdown directly:

{{ render_markdown(value) }}

Aside from the work on Python 3, my Datasette time this week has mostly involved ongoing refactors of both the query execution code and the core TableView.data() method. Hopefully these will unblock a flurry of interesting new functionality in the not too distant future.

Niche museums

This week's new museums on www.niche-museums.com.

datasette-template-sql

datasette-template-sql

New Datasette plugin, celebrating the new ability in Datasette 0.32 to have asynchronous custom template functions in Jinja (which was previously blocked by the need to support Python 3.5). The plugin adds a sql() function which can be used to execute SQL queries that are embedded directly in custom templates.

Via @simonw

Quoting John Carmack

I have sometimes wondered how I would fare with a problem where the solution really isn’t in sight. I decided that I should give it a try before I get too old. I’m going to work on artificial general intelligence (AGI). I think it is possible, enormously valuable, and that I have a non-negligible chance of making a difference there, so by a Pascal’s Mugging sort of logic, I should be working on it.

John Carmack

5 Reasons It’s Financial Ruin If You’re Not Monitoring Your Online Business

By Joep Piscaer

Poor digital experience can kill your business. 57% of shoppers have left a slow e-commerce site and then bought from a similar retailer; 23% of shoppers subsequently never returned to the site, according to Yottaa. So even if you provide a positive customer experience nine out of 10 times, the one time you don’t could […]

The post 5 Reasons It’s Financial Ruin If You’re Not Monitoring Your Online Business appeared first on Pingdom Royal.

Datasette 0.31

Datasette 0.31

Released today: this version adds compatibility with Python 3.8 and breaks compatibility with Python 3.5. Since Glitch support Python 3.7.3 now I decided I could finally give up on 3.5. This means Datasette can use f-strings now, but more importantly it opens up the opportunity to start taking advantage of Starlette, which makes all kinds of interesting new ASGI-based plugins much easier to build.

My Python Development Environment, 2020 Edition

My Python Development Environment, 2020 Edition

Jacob Kaplan-Moss shares what works for him as a Python environment coming into 2020: pyenv, poetry, and pipx. I'm not a frequent user of any of those tools - it definitely looks like I should be.

Via @jacobian

Weeknotes: Python 3.7 on Glitch, datasette-render-markdown

Streaks is really working well for me. I’m at 12 days of commits to Datasette, 16 posting a daily Niche Museum, 19 of actually reviewing my email inbox and 14 of guitar practice. I rewarded myself for that last one by purchasing an actual classical (as opposed to acoustic) guitar.

Datasette

One downside: since my aim is to land a commit to Datasette master every day, I’m incentivised to land small changes. I have a bunch of much larger Datasette projects in the works - I think my goal for the next week should be to land one of those. Contenders include:

I’m going to redefine my daily goal to include pushing in-progress work to Datasette branches in an attempt to escape that false incentive.

New datasette-csvs using Python 3.7 on Glitch

The main reason I’ve been strict about keeping Datasette compatible with Python 3.5 is that it was the only version supported by Glitch, and Glitch has become my favourite tool for getting people up and running with Datasette quickly.

There’s been a long running Glitch support thread requesting an upgrade, and last week it finally bore fruit. Projects on Glitch now get python3 pointing to Python 3.7.5 instead!

This actually broke my datasette-csvs project at first, because for some reason under Python 3.7 the Pandas dependency used by csvs-to-sqlite started taking up too much space from the 200MB Glitch instance quota. I ended up working around this by switching over to using my sqlite-utils CLI tool instead, which has much lighter dependencies.

I’ve shared the new code for my Glitch project in the datasette-csvs repo on GitHub.

The one thing missing from sqlite-utils insert my.db mytable myfile.csv --csv right now is the ability to run it against multiple files at once - something csvs-to-sqlite handles really well. I ended up finally learning how to use while in bash and wrote the following install.sh shell script:

$ pip3 install -U -r requirements.txt --user && \
  mkdir -p .data && \
  rm .data/data.db || true && \
  for f in *.csv
    do
        sqlite-utils insert .data/data.db ${f%.*} $f --csv
    done

${f%.*} is the bash incantation for stripping off the file extension - so the above evaluates to this for each of the CSV files it finds in the root directory:

$ sqlite-utils insert .data/data.db trees trees.csv --csv

github-to-sqlite releases

I released github-to-sqlite 0.6 with a new sub-command:

$ github-to-sqlite releases github.db simonw/datasette

It grabs all of the releases for a repository using the GitHub releases API.

I’m using this for my personal Dogsheep instance, but I’m also planning to use this for the forthcoming Datasette website - I want to pull together all of the releases of all of the Datasette Ecosystem of projects in one place.

I decided to exercise my new bash while skills and write a script to run by cron once an hour which fetches all of my repos (from both my simonw account and my dogsheep GitHub organization) and then fetches their releases.

Since I don’t want to fetch releases for all 257 of my personal GitHub repos - just the repos which relate to Datasette - I started applying a new datasette-io topic (for datasette.io, my planned website domain) to the repos that I want to pull releases from.

Then I came up with this shell script monstrosity:

#!/bin/bash
# Fetch repos for simonw and dogsheep
github-to-sqlite repos github.db simonw dogsheep -a auth.json

# Fetch releases for the repos tagged 'datasette-io'
sqlite-utils github.db "
select full_name from repos where rowid in (
    select repos.rowid from repos, json_each(repos.topics) j
    where j.value = 'datasette-io'
)" --csv --no-headers | while read repo;
    do github-to-sqlite releases \
            github.db $(echo $repo | tr -d '\r') \
            -a auth.json;
        sleep 2;
    done;

Here’s an example of the database this produces, running on Cloud Run: https://github-to-sqlite-releases-j7hipcg4aq-uc.a.run.app

I’m using the ability of sqlite-utils to run a SQL query and return the results as CSV, but without the header row. Then I pipe the results through a while loop and use them to call the github-to-sqlite releases command against each repo.

I ran into a weird bug which turned out to be caused by the CSV output using \r\n which was fed into github-to-sqlite releases as simonw/datasette\r - I fixed that using $(echo $repo | tr -d '\r').

datasette-render-markdown

Now that I have a releases database table with all of the releases of my various packages I want to be able to browse them in one place. I fired up Datasette and realized that the most interesting information is in the body column, which contains markdown.

So I built a plugin for the render_cell plugin hook which safely renders markdown data as HTML. Here’s the full implementation of the plugin:

import bleach
import markdown
from datasette import hookimpl
import jinja2

ALLOWED_TAGS = [
    "a", "abbr", "acronym", "b", "blockquote", "code", "em",
    "i", "li", "ol", "strong", "ul", "pre", "p", "h1","h2",
    "h3", "h4", "h5", "h6",
]

@hookimpl()
def render_cell(value, column):
    if not isinstance(value, str):
        return None
    # Only convert to markdown if table ends in _markdown
    if not column.endswith("_markdown"):
        return None
    # Render it!
    html = bleach.linkify(
        bleach.clean(
            markdown.markdown(value, output_format="html5"),
            tags=ALLOWED_TAGS,
        )
    )
    return jinja2.Markup(html)

This first release of the plugin just looks for column names that end in _markdown and renders those. So the following SQL query does what I need:

select
  json_object("label", repos.full_name, "href", repos.html_url) as repo,
  json_object(
    "href",
    releases.html_url,
    "label",
    releases.name
  ) as release,
  substr(releases.published_at, 0, 11) as date,
  releases.body as body_markdown,
  releases.published_at
from
  releases
  join repos on repos.id = releases.repo
order by
  releases.published_at desc

In aliases releases.body to body_markdown to trigger the markdown rendering, and uses json_object(...) to cause datasette-json-html to render some links.

You can see the results here.

Releases SQL results

More museums

I added another 7 museums to www.niche-museums.com.

Attackers Used Look-Alike Domains to Steal $1M from a Chinese VC

Comments

Robots in Finance Could Wipe Out Some of Its Highest-Paying Jobs

Comments

File Systems Unfit as Distributed Storage Back Ends: 10 Years of Ceph

Comments

FDA Approving Drugs at Breakneck Speed, Raising Alarm

Comments

HackerOne breach lets outside hacker read customers’ private bug reports

Comments

Great White Sharks Are Terrified of Orcas

Comments

PracticalAI 2.0: A New Platform to Learn ML Using Python, Keras and Tensorflow

Comments

Speed Breeding to Fast-Track Crop Improvement in Africa

Comments

Understanding “Deep Double Descent”

Comments

Systemd Opened Security Hole in Linux, VPNs Could Be Compromised

Comments

Joel Spolsky is (semi-)retired

Comments

The Lesson to Unlearn

Comments

Deploy-your-own-SaaS: “only yours” cloud services for everyday needs

Comments

New Features in ES2019

Comments

Google seen as retreating from the Alphabet experiment

Comments

Montgomery Brewster's 'None of the Above' would walk this election.

By Jackart (noreply@blogger.com)

It's actually quite liberating to follow politics without a team to shout for. I remain a Conservative by inclination. I like free markets, economic liberalism and so forth even if the Conservative manifesto doesn't seem to all that much, Tories, if not their leadership, are mainly for these things. I am also a social liberal, I remain committed to an open and tolerant society. However the Liberal Democrats risk becoming the Church of England does Politics, being stuffed with the kind of dry, shabby inadequate who can't quite get over his (self) loathing of homosexuality. I dislike May. I think she's a narrow-minded provincial bigot who's been promoted way, way above her level of competence. She is however the best of the two candidates for Prime Minister. 

Let's not pretend Corbyn was doing other than palling around with the IRA in the 1980s because the glamour of "anti-imperialist" terrorists excited him. He has always supported whoever was fighting the UK at the time, and doesn't deserve to be an MP, let alone to reverse those letters. Labour's clown-car economics is only marginally less risible than the Tories offer, this time round. The difference is Labour actually believe their silliness, and they're led by a traitor. 

If you live in Scotland, this election is about independence. If you live in NI, then this election is about the tribal headcount. If you live elsewhere this election is whether you want an incompetent nanny-state provincial Tory or an antediluvian Socialist to deliver Brexit. It's a shabby, and dispiriting affair. If you can't work out how to vote, you can always vote for Montgomery Brewster. None of the above is appealing. But if you feel you MUST vote, then I have prepared a handy flow-chart to help you.

If you despise politicians, you get despicable politicians.
This shabby parade of also-rans from which we have to choose on today (without any actual choice on the main, nay only, issue of the day) is the logic of calling decent, capable people like Blair, Cameron and Major "war criminals" and "Traitors", for decades. It pollutes the language for when you actually get some of these things on the offer.
No worthwhile people will put up with the scrutiny and abuse heaped daily on politicians. So you get the kind of bore for whom the scrutiny isn't an issue. They've never done anything interesting in the their lives. At least David Cameron dropped some E and went to a rave or two as a youth. What does Theresa May, who spent her twenties complaining about the promotion of lesbianism in schools, know of fun? As for Corbyn, he looks like the kind of man for whom a perfect saturday night is treatise on Marx (so long as it contains nothing he doesn't already know and agree with) with some lovely mineral water. He is the Labour man Orwell warned you about.

I'll be voting Tory. Why? My local headbanging Leadsomite hard-brexiter has stood down after his colossal act of vandalism, to be replaced by a man with whom I seem to agree.
My expectations are of a  Tory majority around 75, on a low turnout, and they will have half a dozen seats in Scotland.  The Liberal Democrats will take Vauxhall and Twickenham, losing in Sheffield Hallam (the "were you up for...?" moment as Clegg loses his seat), but holding Orkney and Shetland against the SNP, remaining about where they are now overall. Or that's where my betting is at the moment.
What do I want to see happen? I'd like to see May remain PM but in a hung parliament, reliant on Northern Irish politicians for her majority because let's face it, she deserves nothing better.
A rubbish show all round but at least I can enjoy it, whoever loses.

Whales are more Important to Climate change than Donald Trump.

By Jackart (noreply@blogger.com)

Donald Trump has pulled the USA out of the Paris Climate accord. And I don't think this matters all that much. For a start, the USA's emissions are falling. Mostly this is because coal is being replaced by Natural Gas, but also because people are driving less, in smaller vehicles with ever more efficient engines. The motors driving the west's steady fall in carbon emissions are economic and technological, not political.

Next to the steady decline in carbon emissions from the west, is set the Vast increase in emissions in recent decades from Asia. But this represents billions of people using no net carbon energy, tending crops using animal muscle and burning biomass (and occasionally starving to death) Just a few decades ago, to my meeting an indian chap on Holiday in Stockholm with his family and chatting about cricket while we tried to decipher the train times. The rise of the middle class in India and China is a huge flowering of human potential, even if it comes with soluble environmental problems.

Anyway, the level of Co2 in the atmosphere is rising, and this is changing the climate. Reducing emissions is a noble aim, but it must not get in the way of developing economies' economic growth. Fortunately, the solution is already with us. Renewable technology is improving. Cars are getting more efficient, and perhaps moving away from fossil fuel (at least directly). And this process will happen in india and China more quickly than in the west beacaue adopting what will be soon proven and cheap technology will enable them to miss whole generations of poluting technologies.

Which brings us to the great cetaceans. The southern ocean is the world's biggest habitat, with the world's shortest food chain, at the top of which sits the largest animal that has ever existed on earth. Phytoplancton bloom, and are eaten by zooplankton, which are eaten by fish larvae and Krill, which are eaten buy just about everything else. The biggest eaters of Krill are the baleen whales which turn five tons of Krill into Iron-rich shit every day. Sperm whales meanwhile are diving to the abysal deep turning several tons of squid into Iron-rich scat, moving nutrients from the deep to the surface. The limiting nutrient at the bottom of the food-chain is iron, so whale faeces fertilise the ocean, and enable more phytoplanckton to grow which absorb Co2 from  the air, much of which falls to the bottom of the ocean as marine snow, and eventually become rock.

But we killed the whales, and when we stopped doing so, they didn't recover as quickly as we hoped. We didn't just kill the Apex predators, in doing so, humanity reduced the Southern ocean's ecosystem's capacity to create life, and absorb Carbon. The southerm ocean may have settled at a lower equilibrium of Iron circulation. The Atlantic on the other hand, which gets tons of Iron from the african deserts every time the wind blows, has seen whale stocks recover better.

Which is why I want to see more research into Iron seeding the ocean, which may give a leg up to Balaenoptera musculus, as well as possibly solving climate change. Climate change is a problem. But while Trump's petulent gesture doesn't help us solve it, nor does it make the problem any harder. Politicians simply matter less than a whale taking a dump.

Why the Blue Passport Matters.

By Jackart (noreply@blogger.com)

People have spent the day on Twitter saying "why does the colour of a passport matter"? While the Daily Express is cheering the return of the Blue Passport to the rafters. For most people capable of abstract thought, this is a mystifying detail, the importance of which to their opponents is utterly baffling. Of course, I am a remain "ultra". But I did swim in the same intellectual Milieu as the Brexity-Trumpkins for decades and know many serious Brexiters personally. Having spend decades rationalising the EU-obsessed madness of the Tory right as a harmless eccentricity that they don't really mean, I do have, with hindsight, some understanding what these creatures think.



Why does the passport matter?

For the Tory Brexiter, the underlying issue is Sovereignty. They object violently, strenuously and on principle to ANYTHING that comes "above" the Crown in Parliament. The jurisdiction of the ECJ is for them, an insult to the courts and other institutions of the UK. The idea is offensive that any law-making organisation, especially one that Jacques Delors told the trades unions is basically for stopping the Tories Torying, could be "supreme" over parliament.

Of course the ECJ mainly deals in trade disputes and represents an international court to settle international issues and ensure consistent interpretation of EU law. It isn't "making the law of the land" and nor is it a "supreme" court in a meaningful way as far as the average citizen is concerned because it doesn't deal with those issues. If you're up in front of the Magistrate for punching a rotter, you're not going to be able to appeal all the way to the ECJ. Criminal law stops with the nation. Appeals of bad people going up to the European court of Human Rights on seemingly spurious grounds get funnelled into this narrative (shhh, I know), so the impression is obtained that "Crazy Euro-Judges" are "over-ruling parliament", and demanding prisoners can vote or should be allowed hacksaws to avoid trampling on "Human Rights" or whatever the tabloid outrage du jour may be. This then reinforces the narrative that the EU is "anti-democratic" and "makes all our laws". And once you have this narrative, flawed as it is, it's jolly easy to amass an awful lot of corroborating "evidence" because the Tabloids spent 30 years deliberately feeding it.

Sovereignty vs Influence; there is a trade-off. The UK, broadly, wrote the Financial services legislation for the entire continent. In return, the Continent got access to the only truly global city in Europe. The French did this for farming and got the CAP, while the Germans got the Eurozone's interest rates and got to destroy Southern Europe. The EU which contains (rather like the UK and trade negotiators) no-one who CAN write decent financial services legislation legislation, because most of those people are British. Thanks to Brexit, the quality of the legislation on financial services will go down, both in the UK which will be compelled to have regulatory equivalence to keep banks' access to the single market and the EU. The UK will have become a rule-taker rather than a rule maker. I fail to see how this reclaims "Sovereignty". The organisational source of the legislation will remain unchanged, but we loose any ability to influence, let alone write it. Multiply this catastrophe across an economy and you see why the "sovereignty" argument against EU law is, on any rational basis, stupid.

The parliament, the very existence of which takes on the aspect of a supranational government in waiting, rather than a simple means to have democratic oversight of an organisation which employs fewer people than Manchester city council, distributes about 1% of GDP and writes trade law. This unwarranted grandiosity once again suits both the Brussels apparatchiks, and the simian oiks of UKIP whom the British public sent to Brussels as a mark of the National contempt for the institution. The parliament is, to my mind is a risible little potempkin affair, barely worth considering,

So there's the error. Back to the passport.

The International Civil Aviation Organisation sets the dimensions, so the writing was on the wall for the old British hardback passport, fabulous though it was, it didn't really fit in the back pocket of your trousers.  However once you believe that the EU tentacles are slowly creeping into institutions to turn you into a province of the "EUSSR", then you start to see this everywhere. The EU is foolish to seek the trappings of a national Government before they had built a demos, and absent any desire for it from the people. Symbols matter. The UK doesn't have an ID card. So when Brits talk about nationality they might say "Australian passport-holder" rather than "Australian citizen". I am not sure if any other nationalities use this formulation. The passport is slightly more than a document. No? Try losing one abroad.

The EU resolution on Passports is here. For anyone who thinks the EU "made" the UK have a Maroon passport, here's EU Croatia's. .



The EU suggested the Colour be harmonised and the words "European Union" be put First. At the top. Above the crown, First. Symbolising, perhaps inadvertently that the EU was more important than the nations. And there you have it. And no-one working on it thought to object. Changing the colour of the passport was a key symbolic gesture that irritated many people, and reinforced an utterly false narrative, to no end or benefit to anyone. There is simply no need for European Union passports to be uniformly coloured. It merely satisfies the bureaucrats' desire for order. And it is my belief that it is this symbolic bureaucratic exercise in territory marking by the EU that revealed, and still reveals, a fundamental disconnect between the Brussels Panjandrums, the people of the EU and the British in particular. The Eurocrats want a Federal Europe with the EU as a Government. The Nations, broadly supported by their governments don't, and have resisted any attempt.

The EU hasn't made Britain less "sovereign". All EU law, necessary to trade with as little friction as possible, is of the type that by whom it is written doesn't matter. With trading standards does it really matter WHAT they are, just that they're as universal and consistently applied? I don't need to tell you that it was never illegal to display prices of potatoes in Lbs and Oz, just that you HAD to display the price in KG and g too, in case any Frenchmen walking through the market didn't know how many Lbs are in a KG. I don't care who writes the regulations for the import of Duck eggs, just that it's done.

But there it is. The Brexiters shooting with the accuracy of a semi-trained recruit who's just dropped LSD at every figment of their fevered imagination, egged on by equally deluded fantasists who still think they're creating a Federal United States of Europe. These two groups of lunatics needed each other. And so, the passport, with 'European Union' at the top was barely noticed on the continent, but seemed to some Brits as evidence the EU was after their democracy, their identity and their Freedom. However stupid this belief is, a Blue passport could've been delivered cheaply as a quick Tabloid-Friendly win for Cameron and such was the narrow margin, it would have probably been enough.

Rosencrantz and Guildenstern are Dead.

By Jackart (noreply@blogger.com)


Wednesday saw my 40th Birthday, and to celebrate I went to see Tom Stoppard's brilliant Rosencrantz and Guildenstern are Dead at the Old Vic with a Chum. While Daniel Radcliffe & Joshua Maguire lead, the show is stolen by a magisterial performance by David Haig as The Player, a sort of luvvie-pimp-cum-impresario who holds the whole play, in its absurdity, together.


The play is Hamlet, seen from the point of view of two minor characters, Rosencrantz and Guildenstern, old friends of Hamlet's. The hapless pair spend the play wondering what they're doing and why, having been recalled to Elsinore by Claudius to find out why Hamlet's being such a dick, moping about and talking gibberish to himself ("to be, or not to be..." etc). They are eventually betrayed by their friend, who suspects them of working for his uncle which they are, sort of.

The play is therefore a meditation on the futility of existence, and the limitations of people's personal agency. Most people get on with their lives, as bit parts in a greater drama, not really sure as to the direction of events, or even of the past. After all, what have Rosencrantz and Guildenstern got to go on, but what can be gleaned from a few words of Shakespeare's, as metaphor for everyone's flawed and self-serving memory. Any interrogator or detective will tell you about the reliability of eye-witnesses and the difficulty of establishing the truth.

From everyone's point of view then, even when we're at the centre of events, most of the action is happening offstage. There will have been some point at which you could have said "no", but you missed it. Then you die.

If you can get tickets, do so.

Minimum Wages, Immigration, Culture and Education.

By Jackart (noreply@blogger.com)

Net migration to the UK has run at hundreds of thousands a year for decades, of which about a quarter since 2004 has been "A8 countries", Poland, the Baltic states, Czech Republic, Slovakia, Slovenia and Hungary, another quarter from 'Core EU' and the rest from non-EU countries, mainly India, Pakistan and West Africa. 13% of the population of the UK was born overseas, of whom over 2/3rds are non-EU migrants. This is an unprecedented migration to the rich world from the poor, and It's not clear from this EU migration is the underlying problem. The Poles will integrate fast, and leave imprints on the culture like a higher incidence of catholicism, bigos (a stew of meat and Sauerkraut) and some hard-to-spell surnames. They're often better educated than the natives, and work harder.

In general the view I've taken over the years is that minimum wages are a bad thing, arguing that they are mainly paid for by the people who otherwise wouldn't get a job at all. Only a job can lead to a better job, and if people are unemployed for a long time, they often become unemployable. So by this logic, keeping unemployment down should in the long-run be better for the poorest.

But, there is a trade off. When I grew up, late '80s and '90s, I cannot recall seeing cars washed by hand. When my father wasn't exploiting child labour by getting me and my brother to do a rubbish, half-arsed job for which we expected to be paid handsomely, we went to see the "blue Dougals" at the petrol station. The UK as a wealthy country, had substituted Capital for Labour, and cars were washed by big machines at every petrol station. But a team of a dozen hard-working and cheerful eastern Europeans can set up a car-wash, do inside and out for very little capital outlay - a jet washer, and some sponges, so when the EU accession countries citizens moved to seek work, this is what many did. The car wash machines were gradually removed and replaced by people. This is the opposite of progress.

Let's take a step back and look at the big picture.

Europe's wealth, it's vitality, its progress didn't spring from European individual or cultural superiority. It started when half the population was wiped out by Yersinia pestis in the 14th Century. There was a certain amount of luck - the same event increased the power of the landowner in Rice states and in pre-feudal societies farther East, but in Northwestern Europe, this created a shortage of Labour, and the peasants rose up a generation afterwards to demand higher wages from their lords. When this happened in Italy, the energy was put into sculpture of the nude male form, and was called "the Renaissance". When wages rise, it makes sense to build machines rather than employ labour, which has a virtuous feedback loop: skilled people running the machines drive up production, and become richer, which creates an incentive for further innovation. More widespread desire for, and access to education is grease in the wheels of this, the motor of progress that led to the industrial revolution.

The opening up of America, a nation with a perpetual and long-lasting shortage of labour not only added another motor to that European culture of innovation which grew up after the Black Death, but also absorbed the excess labour of Europe. While there is a labour shortage, immigration can be managed, though immigrants in large numbers have nowhere, ever been welcomed by the people they move to. Even when the people are kith and kin, the 'Scots Irish' (in reality, families originally from Northern England and the Scottish Borders) were moved on by the Germans and English who'd already settled the East coast. They ended up in Appalachia.

It's clear, then in the short run and in aggregate, wages aren't "driven down" by migration in a market economy. Part of that, in modern times may be due to the minimum wage, which protects some of the people most vulnerable to substitution, but also the 'lump of Labour fallacy'. Immigrants, especially young workers with families bring demand as well as supply and these things more-or-less balance. They aren't "taking our jobs" but they are changing the nature of jobs available. And the vast supply of excess labour from the subcontinent, africa and the poorer bits of Europe is not exactly an incentive to invest in productivity-enhancing machines, as the car-wash example shows. The mass immigration from the poor world has the potential to stall the western motor of innovation and may contribute to wages not rising as far as they might, especially for the lowest skilled workers.

The UK has a problem with productivity. UK employers have got good at employing the excess Labour of a serious chunk of the world, UK wages have been flat for a decade, and these things are linked. So the Chancellor is hiking the minimum wage in the hope of good headlines, and to incentivise investment to drive productivity. So. What effect will this have on immigration. Will it draw more migrants to the UK hoping for higher wages, like European immigration to the USA, or will it price low-skilled immigration out of the Labour market and allow the motor of progress to continue?

Splits that used to be geographic - some countries were rich, and others poor and the movement between the two was rare, is moving to one where there are still two countries, it's just the divide is social, educational, and cultural. You have a global, liberal, free market culture, which values education and novelty. And you have national, 'c' conservatives who just want their own culture, don't care about education all that much, won't move to find a job, and expect to be looked after who stay put and resent incomers. And the latter are disproportionately annoyed about foreigners moving into "Their" neighbourhoods while it's the former who have more to fear in the short term from highly skilled competition, minimum wages see to that. And if minimum wages rise far enough, low skilled workers will not be able to get jobs and they will stop coming to the UK. The problem is, the lowest skilled people are often native. The cost of a raised minimum wage will be borne by those least able to cope.

If we are to avoid society fracturing permanently into Morlocks and Eloi we do need to manage migration, to keep that motor humming. We cannot let the world come at will. But there was no need to pull up the drawbridge against EU migrants who always looked like collateral damage to me.

It's not all about economic self-interest, nor is it wholly naked in-group preference (what educated, open minded people call "bigotry"). It is the interplay between the two. Ultimately the stagnation of UK wages over the last 10 years isn't due to migration, but the recovery from a balance-sheet recession of 2007-9. It's the feeling of ennui caused by a decade of stagnation which has caused the anti-immigration nonsense, the rather blameless Poles have just become a Piñata and for a population that was persuaded to lash out at the EU when they really wanted to lash out at "the Muslims". The tragedy is all this happened just as we were getting back to normal.

Nicola and Theresa. Phwooar.

By Jackart (noreply@blogger.com)

The Daily Mail's headline "Legs-it" about Scottish First Minister and British Prime Minister Theresa May's shapely legs was pathetic. But remember, the Mail is written by women, for women, and women judge each other, all the time, harshly and vindictively. Judged especially harshly are women more accomplished or better looking than the average Daily Mail reader.


To call this "sexism" is to miss the point. This isn't about women being held down by sexist male tittle tattle. Clearly, two of the most powerful people in the country haven't been held down in any meaningful way. Any executive head of Government is fair game for any and all criticism. What these women have done is rise above the level at which society normally seeks to protect women from abuse.

Male politicians are made fun of for their appearance and clothing all the time. It's the sea men swim in. Whether it's Donald Trump's expensive, but ill-fitting suits and too-long ties like he's stepped out of a 1980s pop video caricature of a businessman, or Cameron's forehead, or the fact that middle-aged men are always assumed to be repulsive, this abuse is normal.  The ridicule a male politician faces when he's seen in public wearing anything other than a blue suit is extraordinary. From Tony Blair wearing a clean barbour, to William Hague's baseball cap or Cameron's beachwear, there's a reason male politicians dress identically. When women's clothing (far more interesting by the way, than the sober suits of most male politicians) is commented on, it enables a personal brand to be created that much easier. Theresa May's shoes are like Margaret Thatcher's handbag. True, women do have to think harder about their clothing - too much leg, cleavage etc... and you immediately invite scorn (of other women, mainly), but the fact the female wardrobe stands out against the endless blue/grey suits and red or blue ties of the male is as much an opportunity as it is a minefield.

Any comment about May's shoes, for example is part of her deliberately curated brand, and shoe-designers are falling over themselves to get their products onto her feet. This isn't sexist. Women like shoes, and there's no reason why Theresa May shouldn't have fun with them.

Lower down the pecking order there's a taboo against men commenting negatively on a woman's appearance, lest you hurt the poor dear's feelings. Yes male 'locker room' banter will discuss who's attractive, but it's rude to do so in front of women and by and large, gentlemen don't. Women don't typically have these conversations about men in earshot of men either, but describing men as "revolting" or "creepy" is so normal as to be unworthy of comment, and completely unnoticed. May and Sturgeon have risen above this social protection, and are subject to the same rules of engagement as men are. i.e that if we have feelings, tough.

These women are grown-ups doing important jobs. If you think the Mail's light-hearted front page is an insult to them, you're an idiot. Of course Sarah Vine who wrote the thing, knows exactly the response it would get, howls of idiot outrage from the usual suspects on Twitter, and from Sturgeon herself. This allows the paper to swat the complaints aside with contempt. This signals to their readership that the Mail is on their side against the bien-pensant left with their idiotic & totalitarian outrage about human trivialities. May by rising above it, does the same. The Mail is one of the Best-selling papers in the UK, and one of the world's most visited "news" (ish) websites. Who won that exchange?

The po-mo left, obsessed with identity politics, used to being able to bully dissenting opinion down STILL hasn't got the new rules of the game. Someone's pointed out the Emperor's naked, but he's still acting like he's in charge and hasn't noticed the mood's changed. Yet.



Completely unrelated, but thank you to the Anonymous commenter who wrote this. It cheered me up.

On Class, Culture and the New Politics

By Jackart (noreply@blogger.com)

The two tribes of politics, broadly the Tory and Labour parties divided over the 20th Century principally on the matter of economics. Simplifying: Tories preferred market solutions to state planning, and preferred lower taxes and less generous state spending.
The Labour party, which when it abandoned clause IV, surrendered on the economic question, not coincidentally a few years after the Berlin wall came down.
As a result, the great battles since then have been essentially cultural. Gay rights, racial integration etc. The confusion stems from there being no consensus within the Tory or Labour tribes on these issues. Plenty of Tories are happily socially liberal, many of the Labour tribe are socially conservative, especially when you look at voters rather than representatives.
Which brings us to the tribal division of Britain: class. The middle class: liberal, internationalist, universalists; vs a working class: authoritarian, insular and particular world view. The former is comfortable with diversity and immigration. The latter isn't. The former's kids live a long way from home, and move for work, the latters kids live in the same town and expect the work to come to them. The former don't speak to their neighbours, the latter care what their neighbours do and think. These labels are correlated roughly with, but independent of, economic status. It's possible to be middle class, in a local-authority home living on benefits, and working class, earning seven figures and living in a manor house. (Though it's likely these people's kids will change tribes)
There are elements of these cultures in all major parties in the UK, but the rest of us rarely communicate with people from the other tribe. The people you have round for dinner will most probably be from your tribe. Half the country holds its knife like a pen, yet none have sat round my table. When the two tribes meet, it's awkward. Those difficult bottom-sniffing conversations seeking common ground are easy to conclude when two members of the same tribe meet, and difficult when you meet the other half.
There have always been working class Tories, because much of the working class is as comfortable with the certainties of heirarchy as a shire Tory, and doesn't much care for this freedom and opportunity nonsense, preferring a better boss instead. And it's interesting to watch the Tories dangle the protectionism and insularity the working class has long demanded. Middle class labour fabians and the working class methodists have always sat uncomfortably together. Brexit has shattered that coalition, the labour party has been handed to the idiot socialists and will die, unless somehow moderates can oust corbyn before 2020.
Which brings us to the Tory coalition. The high-Tory have promised the old certainties back to the white working class. Meanwhile, middle-class liberals who make up most of the parliamentary party are distinctly uncomfortable with much of what is being done in Brexit's name, but will stick with the Tories, because they offer the promise of power, and however dreadful Brexit is, Jeremy Corbyn is worse. A new coalition is being forged between the Tory squirearchy, and the Working class based on nationalism, social conservatism and heirarchy, directly taking Labour's core vote. This is why UKIP, a working class movement that thinks it *is* the conservative party, apes the style of a country gent. The working class have always got on well with the Gentry, sharing sociailly conservative values. Both despise the middle class.
Brexit split the country down a line more on class values, split the country and handed it to the socially authoritarian party. Whether this is the new politics, with the Tories moving from being the middle-class party to the working class party, as the Republicans did after the war in the USA, or whether the middle-class will wrest back control over both parties in time waits to be seen.
I suspect unless May softens her tone, and thows some bones to the liberals, her coalition will only survive until there's a credible opposition. A more appropriate division of politics would be a ConservaKIP'ish alliance of WWC and high-tory squires, vs LibLabCon middle-class liberals. Therea May seems to be actively seeking it.
Over the Channel, Emmanuel Macron and Marine Le Pen exemplify this split. The candidates of the parties of left, Socialists; and right, RPR are likely to be eliminated in the first round. Macron is likely to win comfortably. His movement 'En Marche!' was only formed a year ago. There's a lesson for British liberals there.

Tories have profoundly damaged the UK. You Should Join the Tories.

By Jackart (noreply@blogger.com)

2016 happened because decent people don't join political parties, leaving the business of Government to socially inadequate, physically repellent gits with an axe to grind*. In normal circumstances, this makes politics easier for genuinely impressive people to progress through the flotsam of monomaniacs. To be a Grown-up in the Tory Party 1997-2010 was to be able to consider an issue beyond the EU. For Labour it's all about not dreaming of Strike Action by "the workers". Thus the Liberal Centre consolidated a hold on the country, but became complacent to the poison seeping into parties even as the Smug centrist consensus made everyone fat and rich.
There has been a steady, and persistent hollowing out of the political parties. Labour used to be allied to a Trades Union movement that delivered services - health insurance, education and so forth to its members. The Trades Unions of Pre-War Britain where an overwhelming force for good. Atlee's welfare state nationalised all the good the Trades Unions used to do, and so corrupted both the principle of welfare (now far, far from Beveridge's original vision of low, universal payments like Child benefit, topped up with contributory elements) and the Trades unions which became a mere tub-thumper for more state spending. This left the Labour party with the sole purpose of defending a welfare settlement that is not under threat, and a Trades Union movement whose purpose had been nationalised so simply became resistant to all and any reform which might make the system as is function better; unions a mere vested interest of public-sector workers. This isn't a place where people capable of holding more than one idea at a time feel comfortable, and so the Labour party was colonised by people who think not shaving is a political act.
This malodorous and poorly groomed cancer has destroyed the Labour party. It's over, there's no point being in Labour unless you're a Identity politics obsessed Corbynite who laments the end of the Soviet Union. 
Labour, 2010-Present
The Tories at least had the sense to try to vomit the most toxic of their nutters into a bucket marked UKIP, a bucket the dog is unfortunately returning to. The Conservative party my Grandfather joined (from CPGB, as it happens, Labour even back then were cliquey dick-heads) used to be a forum for the upper middle class (and anyone who aspired to join them) to meet, mate and do business. But the horrible young Tories of the '80s, and the Euro-nutters of the '90s meant that by 1997, the Tories were only really suitable for people who were prepared to discuss "Europe" endlessly in ever-more foaming tones, persuading themselves that the EU is a historic enemy like Napoleon, the Kaisar, Hitler or the USSR. To their credit, the Tory Leadership has long known what to do. All David Cameron ever asked of his party was to "stop banging on about Europe". They couldn't stop picking at the scab, and the result is a catastrophe that has already crashed the Pound, weakened the UK (perhaps fatally) and may yet cause a political crisis in Europe and embolden Putin to start rebuilding the USSR.
Tories, 1997-2010
The more say over policy and leadership given to the membership, the more the membership has dwindled (unless, like Labour, the membership criteria are designed to invite entryism for the purposes of choosing a leader - by people who've been quietly loyal to the Bennite project for decades). Giving members a say in who leads the party is absurd. Who the prime minister is, should be a matter for MPs, and MPs alone. It is they who must give the Prime Minister a majority and internal party democracy risks, well, exactly what has happened to Labour. 
However, that Rubicon has been crossed. Party members now expect a vote on the Leader. The question is what to do about this, and the answer is to choose to be a member of a party at all times, hold your nose if necessary. Do NOT identify with the party, but consider which is best placed to advance your objectives. At the moment, the foul bigots, monomaniacs and morons of UKIP are being re-absorbed from a position where they can do little harm beyond foaming at the mouth and masturbating to Daily Express editorials, to one where they can choose the next prime minister, and Mrs May isn't a healthy specimen. The ex-'KIPpers chance may come to choose their PM sooner than expected.
I'm often asked "How come you're still a Tory?"  
Were the Liberal Democrats stronger, I'd be considering them, but I don't trust them on electoral reform (about which they're as silly as Tories are about Europe). But as the Lib-Dems are so far from power, I don't see the tactical benefit of leaving the Tories in a huff, and I broadly agree with the Tories on everything except Brexit. What I'm worried about is the 'KIPpers who're returning to the fold. Unless you want a foul, divisive and ignorant Brexit headbanger to replace May in 2023 or so (Gove for example), Join the Tories, because thanks to Labour's meltdown, Tories and Tories alone will choose the next PM. All not joining a party does is strengthen those (*we) weirdos who still do. Labour moderates, disgusted by Corbyn should cross the floor to the Tories or Liberal democrats, instead of flouncing off to the V&A and opening the way for UKIPish Brexit-o-twats to fight and win a by-elections under Tory colours. Were Tristram hunt now a Tory, not only we could soften this brexit idiocy but also signal just how broad a church the Tories are. 40% of Tory members voted Remain. The tribe that needs to understand the value of a bit of entryism is the liberal centre, who need to abandon any loyalty to their Parties and go to where the power is. The Liberal Centre is complacent because they have for so long occupied the ground sought by all parties, they've not really had to compromise. 
At the moment the business of Government is, and will be for the foreseeable future, a Tory-only affair. That need not look like Nigel Farage, but it will, if Remainers abandon the Tories entirely.

The End of A 'Belle Époque'. 1991-2016.

By Jackart (noreply@blogger.com)

The interlocking webs of policy which 'politics' seeks to knit are complicated. Whole books can be written on how two individual policies interact. PhDs in Economics are awarded for small snapshots of the whole cloth. Most people don't have the time to keep abreast of developments or read sufficient history to understand why some policies are bad. Thus, people use heuristics - rules of thumb - to make decisions  about that which they aren't expert. "Is this person trustworthy" is a key issue, and we tend to overweight the opinion of those near us. "He is my brother, and I say he's ok" says a friend, you are more likely to believe a mutual friend, than the opinion of a stranger on the same issue.

In the evolutionary past, such a question was a matter of life and death. People only really had to trust those with whom they shared a close genetic relationship. Since the development of agriculture, we've been steadily widening that circle of trust. The wider you spread that circle of trust, the richer your society will be. Even before it had a name, Free market economics allowed people to become blacksmiths, knowing others have water, food, shelter and so forth covered in return. More specialisation, greater productivity, means greater wealth.

Eventually, this requires trust in people we've not met. Towns' food supplies require that farmers unknown and distant supply the basics of existence. Nowadays, It's unlikely the west could quickly supply all available plenty currently manufactured in China. Nor could China supply quickly the complex components and tools shipped from Japan, Europe and USA. Both China, and "the west" are richer from the exchange. And yet, we still don't trust "globalisation".

Most persistent fallacies in political economics are the result of simple policies that appeal to some base heuristics, but which when applied to the larger and wider society, fail catastrophically. Thus egalitarianism in one form or another pops up every 3 generations or so and succeeds in making everyone equal, but some more equal than others, and even more, dead. Then nationalism comes along, and says it's all [another, arbitrarily defined group of humans with slightly different modes of speech] fault, leading to more waste and piles of corpses. And even when the results aren't catastrophic, we seek out the views of those who agree with us on say, Nationalism to inform our opinion on, say, whether or not people are responsible for climate change.

Which political tribes stumble into being right or wrong on any given issue appears arbitrary, because no-one's asking for the evidence before they decide on the policy. Instead of asking "what's right", we're asking what's popular (amongst the coalition of tribes that voted for me) right now. That an opponent comes out with an identical policy, for different reasons is reason enough to oppose something, forgetting completely prior support for it. After all, whatever [another political tribe] thinks must be wrong, right.

Thus
The Labour party opposes ID cards. The Labour party has always opposed ID cards. The Tory party is for the Free market and was never in favour of the Corn Laws. We have always been at war with Eastasia. Perhaps if we could think for ourselves rather than just accepting tribal dogma, we'd get better governance. But none of us have the time. So "Democracy" is merely a means to give temporary permission to one coalition of tribes to push through dogmas over many issues, until either the population notices, or the coalition of tribes breaks up, and the electorate takes a punt on the other tribe's prejudices for a bit, and then gets on with whatever they were doing before.

Society ultimately advances by eliminating prejudices it's acceptable to hold thus widening the circle of trust, and increasing riches. By falling back on ancient heuristics to answer the wrong question ("who's fault?" is the wrong question) 2016 democracy has delivered the worst political outcomes on a broad front, as a result of which, we are poorer, and more likely to start fighting as a result of the collapse in political trust we have seen over this year. The post Cold-War 'Belle Époque', which saw half of humanity, 3 billion people, lifted out of poverty, is over.

Idiots cheer.

Boston Dynamics and The Late Sir Terry Pratchett

By Jackart (noreply@blogger.com)

Everyone knows how driverless cars will work: they will be like ordinary cars, except you read a book rather than acting as pilot. And so, people's understanding of what a technology can do is clouded by what the old technology it replaces does. Which means people without imagination, Head of IBM Thomas Watson, for example, say things like
"There may be a world market for maybe five computers"
and get it wrong. In 1943, computers were used for cryptography, and that's it. (At least he knew what a "computer" was, which few did back then). Predictions are hard, especially about the future. But it's probably worth noting here that the famous World Wide What? front page of The Sun, was in fact rather a good a spoof, by The Sun.


Boston Dynamics makes robots.


via GIPHY

Who needs Robots? Well, like computers or the internet or driverless cars, the technology is coming. And it will change people's behaviour in many, unpredictable ways. For example, mobile phones were conceived as portable analogues for the phone on your desk or in your hall. SMS text messaging was added as an afterthought, but became THE dominant means of communication. Calling someone is now rude, often you text first to see if a call would be convenient. Who (apart from mums) leaves voicemail messages any more? Few predicted that change in our behaviour. The smartphone is now ubiquitous, and is more about accessing the internet than calling friends, but wasn't imagined before the internet, Except by Douglas Adams (and John Brunner of whom I'd not heard until I discussed the issue on Twitter). Driverless cars will be as close to the car, as the car is to a buggy and four. And robots, when they become ubiquitous, will be unlike anything we've considered.

I look at Boston Dynamics Robots, the big dog is conceived as a load carrying mule for soldiers on rough terrain, and I think of The Luggage, Rincewind's inscrutable companion on the discworld. I suspect everyone will one day have a robot the size of a dog to carry daily necessaries, following them round. You could send your luggage to someone else, by smartphone app to pick something up. Your luggage could take your shopping home and collect it from the store for you. Large luggages could be sent on ahead with bags. Small luggages could replace handbags and briefcases. The labour and time saving would be vast, spawning whole new areas of employment, servicing and modifying your faithful electronic companion and providing for the opportunities they create to effectively be in two places at once. Freed from the ownership of motor vehicles by the fact we'll be taking taxis everywhere, our Robot luggage will perhaps become the next status symbol around which society is built, replacing the car.

Like cars, I suspect the battery technology will be the limiting step, and like cars, I suspect the fuel cell will be the answer. Small fuel cells will one day power your smart phone too.

But think about the opportunities for people from smart phone. There are tens of thousands of app designers round the world now, a job that had barely been considered as recently as 2007, when the first iPhone was released, and that is similar to how the jobs which will be taken by the robots, will be replaced. That is why people who fear of a "post-jobs" future were wrong in 1816 and are still wrong 200 years later. The world's only limitless resource is human ingenuity.

Anway. I for one welcome our new robot overlords, and this guy should totally be locked up.


via GIPHY

Fidel Castro is Dead. (Some of) his Legacy will Live on

By Jackart (noreply@blogger.com)

Let's be clear, Castro was a murderous bastard who impoverished his country, and whose views on homosexuality and on the importance of brevity in speeches were nothing short of horrifying. It's true, Cubans do have access to better healthcare than many countries of equivalent GDP per capita, and if I had to choose a Communist hell-hole to live in, it'd probably be Castro's Cuba. But the Cuban healthcare system is not the fantasy of western dewey-eyed left-wingers, and Cubans often are excluded from what excellence there is, as it's one of the few means the country has of generating hard currency earnings. Rich foreigners get the best doctors, and more are exported to other successful "progressive" regimes like Venezuela.

"But he was an anti-imperialist". So why were cuban troops in Africa in support of the USSR, which was by any measure or definition an Empire? Anti-Imperialsim is just the justification leftists give for knee-jerk anti-Americanism. And the flood of people risking death to reach the USA should tell you all you need to know about the relative merits of America's and Cuba's system.

Contrasting the attitudes of the USA to Castro, to their attitude to equally murderous bastards like Pinochet misses the point. The US embargo on Cuba is one of the legacies of the Cold war, kept bubbling by the politics of Florida, home to so many Cuban-Americans. There is no Doubt that the US blocade has impoverished Cubans, and that with the fall in the Berlin wall and the collapse of the USSR, such an embargo was no longer justified. However politics are what they are. Fidel Castro's death provides an opportunity for further thawing in relations.

The USA supported "our son of a bitch" all over the world, turning a blind-eye to horrific human rights abuses, though often (albeit less often than we should) working behind the scenes to try and mitigate the worst behaviour. Thatcher is rarely credited with preventing the execution of Nelson Mandela, but she consistently urged Mandela's release, even as she argued against sanctions and branding the ANC "Terrorists". This is one reason why the cold-war piles of dead of Nasty fascist bastards are usually lower than those of nasty communist bastards. I also think the point made by CS Lewis holds. Right wing dictators rarely pretend to be GOOD, making their appeal more on effectiveness.
"The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience."
And one by one, following the collapse of Communism, the support from the USA and its allies for these disgusting regimes was withdrawn. Apartheid South Africa, much of South and Central America saw right authoritarian regimes fall. Genuine democracies were often created in the rubble. The USA didn't support dictators because the USA is an imperialist power, but because it IS a power, and with that comes responsibility. They judged at the time the alternative, Communism, was worse, and represented a genuine existential threat to the USA and its core allies.

This is why for example the USA and its allies mostly support the Regime in the Kingdom of Saudi Arabia. The Saudi regime is repellent, but given the probable alternatives wouldn't be nice, liberal, democratic-minded progressives, they'd be salafist nutcases who'd have access to billions of dollars of oil revenues and the legitimacy of being the Guardians of the Two Mosques. The House of Saud is all that stands between the West and a plausible salafist caliphate with sufficient legitimacy and money to one day threaten the west. We'd rather do business with nice, stable democracies under the rule of law. But seeing as we cannot do to every country on earth what we did to Germany in the late 40s and 50s, we make the best of the options given.

Castro appeared to be a true beleiver in Socialism, so he refused to recognise his philosophy had failed, and his island limped on, a socialist throwback in the age of globalisation. The current poverty of Cuba is partly America's doing, but mostly due to decisions made by Castro himself, policies which set him and the Cuban people at odds with the regional hegemon, in persuit of an evil idealogy. Fidel Castro was on the wrong side of history, and his people suffered because of his stubborness. Now he's dead, it's Cubans turn to make the most of the positive legacy - Cubans are the best-educated poor people on earth, and the mighty economy of the USA is right on their doorstep. There is going to be a lot of money to be made there, and this time, for the first time, Cubans will share in it.

Hail, Trump! God-Emperor of the Alt.Right

By Jackart (noreply@blogger.com)

And Let's be honest, he's ghastly and despite brown-nosing by Nigel Farage, he's no friend of the UK's, because he doesn't value anything the UK brings to the table. Rumour has it, he asked Farage to intervene in an offshore windfarm decision affecting his Scottish interests, which suggests he doesn't understand the concept of 'conflicts of interests' when in elected office.

This further suggests Trump will attempt to use the office of President to enrich himself, rather than doing so after leaving office, as is accepted. All this is rather feudal; the office holder as gold-giver, distributing patronage and receiving tribute. He's an entertainer and showman, which hails to an even older tradition of politics: that of Imperial Rome, where emperors used state coffers to enrich themselves and their clients,while keeping the mob quiet with bread and circuses.

Donald J. Trump is psychologically unsuited to office in a mature democracy. He is thin-skinned, autocratic, insecure, ignorant, and completely without any understanding of the levers of power he now wields. Much like (later caricatures of?) Nero, Commodus or Caligula.


Despite (or perhaps because of) this, the adolescent losers of Alt.Right see Trump as a God-Emperor (no, really they do. Video surfaced today of people making Roman Salutes, saying "Hail Trump", and distribute Memes based on Games Workshop's futuristic figure-based tabletop wargame, Warhammer 40,000 where humanity is defended from Chaos by a psychic God Emperor). If Trump is Imperator, then the Secret Service is a Praetorian Guard. And how did the Praetorian serve Commodus, to pick one example?

Trump might, were he capable of reading a book, muse on the fact he's surrounded by armed men sworn to uphold the Constitution of the United States of America, and defend it from Enemies DOMESTIC and foreign. Thankfully, the USA is a mature democracy. Where once armed men acted as kingmaker, courts now do. For the simple reason Ignorance is no defence, and the fact that Trump's loathing of 'Washington' is fully reciprocated, I find it unlikely that Trump will survive his term alive and unimpeached. Unfortunately I cannot find odds on a Trump impeachment before 2020. Perhaps it's a racing certainty.

Sexism and the Loss Aversion Heuristic

By Jackart (noreply@blogger.com)

Men are physically stronger than women, respond quicker to physical training, and suffer less injury under physical stress. Men are more robust, suffer less morbidity than women in almost all phases of life. Obviously these things exist in a normal distribution, but men's distributions are typically platykurtic - there are more men in the tails of the distribution than women. Thus, even where the means are near identical, such as intelligence, you'd expect to see more male geniuses, and imbeciles among men than women, who're more concentrated around the mean. Feel like taking issue with any of these statements? Then you might as well be a creationist.

Men are more accepting of risk, and will prioritise pay over flexibility. So you'd expect men to make up the majority of soldiers and miners and race car drivers. It also means you'd expect to see more men make up corporate boards, everything being equal. More men are more drawn to the cut and thrust of business, and are more likely to prioritise work over other commitments. Women value stability and flexibility more highly than men. This means women, on average don't choose to make the effort necessary to climb the greasy pole. Women (sensibly, in my view as I have done the same) are more likely to think other things more important.

Thus, the brute fanny-counting of media analysis of sexism and the "gender pay gap" ignores female choices and attributes, thus denigrating both women and men for the choices they make. Women for their part see their contribution to society in caring professions such as medicine (more doctors are now women, as well as nurses) and teaching denigrated because these women aren't seeking to be at the top of BAE systems, or whatever. Likewise men, when they see women are going to hired so they form 50% of the workforce of a mining company feel devalued for their skills and attributes because the only way BHP Billiton could make 50% of its employees women is by discriminating against the larger number of men who will apply to drive a bloody great truck miles from nowhere in a bloody great hole in the ground in the middle of a bloody great desert surrounded by nothingness, and live in towns whose bars serve tinnies through wire grilles, and where kicking each others' heads in represents the primary saturday night entertainment.

But worse, by forcing women into traditionally working class men's jobs, you further alienate and disorientate a bit of society which already feels put upon, neglected, belittled and scorned. This is why they voted for Brexit in the UK, and in the USA, will vote for Trump. Working class men are lashing out, because their raison d'etre, to provide for their offspring, has been nationalised, and no other opportunity for them has been provided and they as individuals have too often been thrown on the scrap heap, derided as workshy deadbeats. The working class used to have pride in providing for their family and often doing dangerous, dirty jobs to do so. Opportunity isn't "equal access to university", for which working class men is a middle-class rite of passage, but decent jobs that will allow them to support their family, but which is blocked by the petty credentialism that values paper qualifications over experience and dumb diligence over inspiration.

That loss of pride is agonising. And people mourn loss far more than they celebrate gain. The aim of this post-modern obsession with equality of outcome therefore might as well be to make men despise themselves and women feel inadequate for the inclinations their biology and society has fitted them. Men become 2nd rate women, and women become 2nd rate men. By all means allow everyone to seek their own path, but to imagine men and women will sort 50/50 everywhere is totalitarian in its foolishness and cruelty.

The EU Deserves what's coming.

By Jackart (noreply@blogger.com)

One of the main reasons to oppose brexit is that the UK doesn't benefit from being "out" should the EU collapse. A disorderly break-up of the EU would damage the UK, independently of our status in or out. (any comment saying "it's better to bail early" will be deleted as a failure of comprehension read the post, please, it's that argument I'm dealing with). Indeed preventing a disorderly collapse should be the UK's priority. And when we were in, a disorderly collapse was unlikely. The UK kept the lid on Brussels insanity. Not only has Brexit given free rein to some of the very worst people in the UK, it also removes a brake on the insane Federasts  of Brussels.

Far from Remainers "talking the UK down", Brexiters have been doing so for decades - talking down the UK's influence in the EU to the extent we're actually thinking of walking out of the UK's proudest creation: the single market. It is now a shibboleth that the UK has "no influence in the EU", whereas the UK drove the single market, kept half the continent out of the poisonous grip of the Euro and pioneered enlargement to the east following the end of the cold war. The UK drove Russian sanctions to this day. The UK was one of the Big three and on many issues, more influential than France. The UK largely writes EU financial regulation for example (as is meet and proper).

But the EU over-reached. Voters, especially in the UK resented the EU's usurpation of the trappings of National sovereignty far more than the reality of "the laws made in Brussels" which was really just code for an underlying vision they (and I) don't like. And what is true of the UK is true of France and the Netherlands and everywhere else. Remainers like to mock the Be.Leaver's joy over the anticipated return of the blue passport. I however have long resented the words "European Union" above (ABOVE!) the crown on the front. It's like the bureaucrats are trying to rub the British People's nose in it. It's a symbol of something burning in the EU's core, which the average voter neither desires, nor trusts.

The ridiculous and unnecessary potemkin parliament with its farcical shuttle from Brussels to Strasbourg focusses the voters minds on the EU, without giving them any outlet to do anything about it. The EU looms much larger than it ought as a result of the charade of Euro elections. Democracy without a demos is pointless - what commonality do Socialist members from spain and the UK have?:

The EU was flawed, Thanks to the UK some of its worst excesses - the Euro for example were limited to countries that really wanted it. And now without a powerful country holding the reins and steering away from "ever closer union" the Brake that was put on at Maastrict and beyond will be removed. The EU will integrate itself to death, there will be chaos when the voters of Europe can take the tin-eared arrogance of Brussels no more. There was no need for all those millions of lives to be attenuated during that process. While leave voters will say "I told you so", a better analogy would be jumping out of a moving car suffering broken bones and extensive skin abrasions, but saying "it would have been worse" because the lunatic who grabbed the wheel when you bailed steered it directly into a tree.

Spending 1% of GDP to write trade and some business law could much more easily be done intragovernmentally, with a humble and small central bureaucracy. There is no need for "Presidents" and parliaments which lead to grandiose visions; visions which slam painfully, like the Euro, into the unyielding wall of reality. Unobtrusively aligning business regulation and deepening economic integration is necessary. A parliament, a flag, an anthem and a head of "state" are not. The EU has paid the price for this arrogant and pompous grandiosity.

Both the EU and UK are and will be significantly worse off as a result of Brexit. And now, just as Brexit is a bad idea that will be tested, so too will European integration. Both Brussels panjandrums and the brexiters fed off each others' fantasies. Both needed to believe integration was happening, even if it wasn't. Ultimately, the costs will become apparent to the UK pretty rapidly. The EU will suffer much more slowly. It's almost like co-operation is a non-zero-sum game, or something.

On Populism: What do we do? vs Who do we blame?

By Jackart (noreply@blogger.com)

If you ask the wrong question, the answers will not work.

"Populism" is, like pornography, hard to define, but you know it when you see it. Wikipedia defines it thus
"a political ideology that holds that virtuous citizens are mistreated by a small circle of elites, who can be overthrown if the people recognize the danger and work together. Populism depicts elites as trampling on the rights, values, and voice of the legitimate people"
It's clear Farage's lauding of a victory for "mediocre ordinary, decent scum people" he was speaking in this vein. But I don't think this captures the essence of populism. Mainstream politicians "Managerialists" in the Populists' vernacular ask "how do we solve this problem". You can be a capitalist, or a socialist, believing in different answers, but at least you agree on the question. Populists aren't asking this question, but instead "who do we blame?". The answer given by Momentum and UKIP may differ: Bosses vs Immigrants, Capitalists vs the EU but the question is the same.

There's also the populists view that MPs rather than being representatives paid to exercise judgement, are delegates paid to vote on someone else's behalf. In this, Paul Mason and Douglas Carswell are in agreement. But this is simply mob rule and behind it is a fear that legislators may Go Native, if they're allowed thanks to the corrosive influence of "[insert boogeyman]" in their long-running campaign to keep the "real" people down. But perhaps legislators know best; they have exposure and access to what passes for facts in this field, and are paid to study it, maybe there's something in the idea of representative democracy after all.

It's always easier to imagine you're the victim of an elite conspiracy, subject to "discrimination" on the grounds of class or race, or at risk from being "flooded" by immigrants, than it is to answer the question "what to I do?". Whether you're running your own life, or that of a nation, what to do is hard, and one of the stresses of modern life is the extent to which people are free, which means they have to make choices. No longer can you just follow dad into the Factory. Because many suffer from crippling loss aversion, these choices are scary, which is why stupid people yearn to be led. They look for leaders who offer answers which fit their prior prejudices and make sense of a complicated world. Corbyn and Farage have made careers finding and stroking a tribe's prejudices, soothing their people's indignation against a world they feel is against them.

The reason populism is so toxic to political discourse is that in apportioning blame, they create a slipway for the launching of vastly damaging ideas. "It's all the EU's fault" leads to Brexit*. "It's all the Fat Cats' fault" and you have a country that looks like Venezuela. If you start blaming immigrants or minorities, well we saw where that went in the last century. It's also why the Brexiteers ran from office at the moment of victory. Delivery isn't in the populists' skillset. The permanent masturbatory pleasures of opposition are what they crave, always losing so they can keep telling their people the game's rigged against them. If they win, then all those inadequate people will have to start making choices and they feel completely lost again. Much easier to simmer in resentment against an immovable object which allows you to blame it, rather than yourself for your failings.

*This isn't a place for a debate on the merits or otherwise of Brexit. Any comments on that subject will be deleted.

One of the reasons for the Populist's success (please note the "one of" at the start of this sentence) is Russia on the internet. The internet allows people to form much denser ideological defences against reality. And into the internet, there is a wounded superpower, pouring poison, poison which people use as ammunition in the defence of their ideological redoubt. Putin's toxic little propaganda swamps like RT and Sputnik are manufacturing and promoting stories which appeal to the populist mindset. Notice how Racists will share RT stories about Immigrants raping white women while members of the Green party will share horror stories about fracking from the same source. Some of these stories will be true. But many are manufactured, exaggerated and twisted specifically to support any party or idea that causes problems to the democratic governments of the west. This is not a random process. It is directed and controlled by the intelligence agency which has captured Russia. Maskirovka raised to a governing principle.

One of the reasons for the UK's relative success as a nation is that up until now, we have been mostly immune from the allure of the populist demagogue. We simply don't have it in us to put too much belief in one man, whether as protagonist or antagonist. Let's hope Brexit is a flash in the pan, and not part of a widespread descent of mature democracies into populist demagoguery. We'll know in 12 months whether democracy can survive or whether, thanks to Trump, Farage and Le Pen, we're going back to pogroms and a summer "campaigning season".

Please let's stop listening to Putin' useful idiots pedalling fallacious simplicity, and start listening to fallible and all-too-human experts again. At least the experts are asking the right question.

Flu Shot

"Wait, how often are you getting bitten by snakes? And why are you boiling water?" "Dunno, the CDC people keep showing up with complicated questions about the 'history of the property' and 'possible curses' but I kinda tune them out. At least one of them offered me the flu shot."

AI Hiring Algorithm

So glad Kate over in R&D pushed for using the AlgoMaxAnalyzer to look into this. Hiring her was a great decisio- waaaait.

Is it Christmas?

We've tested it on 30 different days and it hasn't gotten one wrong yet.

Keycloak 8.0.1 released

To download the release go to Keycloak downloads.

All resolved issues

The full list of resolved issues are available in JIRA

Upgrading

Before you upgrade remember to backup your database and check the upgrade guide for anything that may have changed.

More fun with GPS...

By noreply@blogger.com (RevK)

The NMEA sentences from a GPS work on degrees, minutes and fractions. As a format they are horrid - the sign is a separate character N/S, E/W, and the value is fixed length so DDMM.mmmm and DDDMM.mmmm format.

The 4 places for minutes is not that precise, but more than enough for absolute GPS positioning.

But there is an alternative, and it is like the IPv6 of GPS. More precision that you could ever want - ECEF format is X/Y/Z in metres from centre of the Earth to the micrometer!!!

$ECEFPOSVEL,124015.000,3985352.629186,-54462.755461,4962832.255023,-0.008022,0.003776,0.022063

Amazing. So whilst not adding absolute accuracy, it does add some impressive relative precision.


Walking in the road to spell my name, LOL.

Hayes AT

By noreply@blogger.com (RevK)

A long long time ago, in galaxy far far away (no, scratch that bit), a standard evolved for talking to modems, well, specifically Hayes modems, but it was widely adopted.

It works over a serial link by sending commands starting AT for (attention). Now, I am pretty sure they were reasonably consistent, back in the day, but like many things this standard has evolved and been bastardised.

We now have things like this SIM800 module, which is basically a mobile phone in a can, with no display or keypad. It talks on a serial bus and talks Hayes AT commands, but as modified by various GSM bodies.

Now, some AT commands are consistent. You send AT and some options, and get some data followed by OK. Good.

There is a reasonably consistent format for commands and settings and queries, mostly. There are then AT+ functions which are used for a lot of the mobile stuff.

But it is far from consistent. The AT+CIPSEND, after the > sends "SEND OK" not "SENT OK" or "OK", and AT+CIPSHUT sends "SHUT OK". Why are these not just "OK" like the rest? I am also pretty sure I had one case of a response with no OK (but cannot find it now).

Then we have the asynchronous messages from the device. Most start with a +, like "+CREG: 0,1", so can be recognised if you are expecting another response. But some are just text like "SMS Ready". These smack of debug messages left in the code.

Half the interface was clearly designed for a person sat at a terminal typing stuff, and the other half was designed for talking to a machine!

But when you get to some of the higher level functions, like establishing a UDP "connection", it gets even more special. For a start, you look for "CONNECT OK" not simply "OK", and mostly at that point it seems to avoid sending asynchronous messages (though I cannot be sure). Some still can be sent, like "+PDP: DEACT" when it loses the connection. What you do get though, with no prompt, header, or indication of length is the raw binary content of any UDP packets you receive on the connection. So send a packet containing "+PDP: DEACT" and I'll think the connection has dropped!

What would have been sensible would have been a line something like "+DATA=N" and then N bytes of data, or some such, but no, it is simply the raw UDP packet.

Then we have some interesting little bugs and discrepancies. One that took a while to work but was that the length of UDP data I can send is limited. Not to 65535 which is what IP limits to, but to 1472 which is the available UDP in a 1500 byte IPv4 packet. Except it is not. It is actually 1460, any more says "ERROR". Oh, and it won't allow me to send a zero length UDP packet - no idea why, and as NAT is often used this is something I may want to send as a keep alive. I ended up having to send one byte.

But there is an even more special feature. When you receive a UDP packet, as I say, it arrives as the raw binary data via serial. I am relieved to see no mangling of any bytes, which is good. Except it truncates the last byte of the message. Yep, I have to send an extra dummy byte on my packets to the device. WTF?

I could save myself from a lot of this by coding PPP, pretending to dial on an old style modem and talking PPP which the device fakes and converts to packet data. This is a serious consideration, to be honest.

However, after a lot of messing about, my GPS trackers are sending (and receiving, when needed) tracking data over UDP over mobile - yay!

P.S. I may have maligned the SIM800 as incoming UDP looks like it may be my bug losing a byte, and there is an option for a header with length for incoming UDP. But the Quectel M95 does look a bit nicer.

That packet looks familiar, and that one, and that one...

Sometimes I tell stories about service outages from the point of view of it having already happened, and provide this kind of "omniscient narrator" perspective as it goes along. I can tell you to look out for things, since they'll come in handy later, and so on.

Other times, I try to present them in much the same way that they happened, complete with the whole "fog of war" thing that goes with being in the moment. This is where you can't see obvious things that are right in front of you because they seem too ridiculous to contemplate.

This is one of the latter, based on multiple events I've lived through. Yes, multiple: it's happened at several distinct companies that I've either worked for, contracted for, heard in stories from friends, or read somewhere over the years.

So, let's dive in.

You're at work at the office. It's a normal business day. Lots of other people are also at the office, doing their regular jobs. Much of this work involves poking at vaguely Unix-flavored boxes which are not physically on the premises. They're somewhere relatively far away, and so you have to telnet, ssh, or whatever to get to them. The point is, you're having to cross at least one "WAN" link to reach these things.

Things are going okay. But then it seems like everything is getting slower and slower. Interactive connections are lagging: keystrokes don't echo back quickly any more. Web browsing and other "batch" network stuff is dragging: the "throbber" in the browser is busy for far too long for ordinary activities. The chat system gets slower and slower.

You manage to hear from other people via the chat system ever so briefly. Some of them are seeing it as well. Others aren't sure, but at least you know it's not just you.

Over the next minute or so, the whole thing grinds to a halt. EVERYTHING is now dead. Nothing's working.

Depending on what decade it is, you whip out a modem and dial back in to the corporate network remote access pool, jump on the guest wireless network, or start tethering through your cell phone and get back in touch with whoever managed to stay online. You find others who have done the same thing, and it's clear there's a very big problem, but only where you are. The rest of the organization's offices are fine, but everything where you are is toast.

Someone immediately thinks it's the link to the Internet and picks up the phone to the ISP to yell at them, because it has to be them. It's the network. It's always the network. Particularly when there's someone else to yell at, right? That happens in the background.

Maybe an hour into this, someone unrelated to the problem but who knows their way around a network makes a curious observation: they have a hard-wired connection, and they are seeing a LOT of crazy traffic going past their host. It's way more multicast and broadcast traffic than they are used to. (Apparently, they have done this before, and have their own internal metric for what "normal" is, which is particularly valuable now. Otherwise, how would they know?) Hooray for tcpdump!

Some of the broadcast traffic identifies its origin. More than just a MAC address or a source IP address, some of it (depending on the decade again) is NetBIOS over TCP/IP broadcasts, or MDNS broadcasts, or whatever, and they helpfully have the names of the hosts embedded. These are workstations, and they include some aspect of the human user's unixname, so you can go "oh, that's so and so".

Did that person just unleash a hell torrent on the network? Better go check. They're right around the corner on the same floor, so why not? You get there, and they're very friendly, and no, they aren't doing anything funny. Their host DOES have the IP address and MAC address corresponding to some of the packets flying past the other box over and over, but they swear up and down they didn't do anything.

You ask nicely if they'll disable their network interface(s) temporarily just for a while to attempt isolating things, and they agree. They even do you one better and power the whole machine off and will leave it off until you give the all-clear. They even unplug the network cable(s) to thwart any Wake-on-LAN magic. There's no way it's going to cause you any trouble now.

You walk back around the corner to the first person's machine where they spotted all of of the bcast/mcast traffic, expecting it to have subsided. It hasn't. Indeed, it's still running full tilt, and then you notice in the spew that the host you just disabled is still in the packet traces. This box has its NICs disabled, is unplugged from the hard-wired network AND is powered off... and yet there are Ethernet frames going by which claim to be from it.

At this point, people who have lived through this before probably know what happened. They might not know how, or why it was able to happen, but somewhere deep down inside, they have this sinking feeling that someone did something terrible to the network and this is how it's manifesting.

The story diverges from here based on which version of it I've either lived through or heard about, but it goes along similar lines in any case.

Someone introduced a loop to the network. The most likely case is that a person showed up in the morning and went crawling under their desk to plug something in, like a power cord. Maybe it "came unplugged" overnight or over the weekend, and they're fixing it.

Then, while they're down there, they see this random Ethernet patch cable just kind of hanging out on the floor. It looks like it should be plugged in to one of the ports down there, but it's not. They plug it in to "do that person a solid" so they don't have to waste time also dealing with unplugged stuff.

Of course, as it turned out, the loose cable end was one that was ON TOP of a desk, and had fallen down. It was the end that went into a computer, and did not need to be plugged "back in"... for the other end of that same cable was already plugged into a port!

Taking that cable and jamming it into another port introduced a loop.

Again, here, a bunch of people are jumping up and down, looking for the comment form, or popping back to the Hacker News tab to say something. Patience. We'll get there.

Radia Perlman figured this out a long time ago. It's called spanning-tree protocol, and it's how your network can detect and defend against such clownery as someone looping it back onto itself.

But, again, the story branches. One time, it didn't exist yet in the networking equipment at the company. Another time, it did, but the people running the network had no idea what it was, or why they'd use it. A third time, they knew about it and decided to "not use it" since "nobody here is that stupid". Yet another time, they had just turned it off days earlier because "it was breaking stuff, and things were fine without it".

For the benefit of those who haven't lived through it yet, here's what happens. Let's take a simple loop where you've managed to plug two ports on the same switch into each other. We'll say that ports 15 and 16 are looped back.

Someone, somewhere, sends out a broadcast, multicast, or possibly even a unicast packet (for a destination not yet mapped to a port). The switch (or hub, if you're "back in the day"...) takes that packet and floods it out to all ports except the one where it came from. Maybe it came from its uplink (port 24), and so it sends the packet down ports 1-23, skipping 24.

The packet goes out port 15, makes that neat little hairpin turn, and arrives back at the switch on port 16 at nearly the same time. Assuming it's a broadcast, multicast, or a still-unknown unicast address, we're right back where we started: the switch has to take THIS packet, then, and flood it out to all of its ports less the source port. It gets sent to 1-15 and 17-24, skipping 16. Hold that thought right there for now.

Back up in time a few nanoseconds. That packet ALSO went out port 16, made the hairpin turn, passing itself going the other direction (possibly on the same wires, if in in full-duplex mode), and showed up back at the switch on port 15. At that point, the switch proceeded to flood it out to 1-14 and 16-24.

So you see, every time the packet leaves 15, it arrives at 16, and every time it leaves 16, it arrives at 15. If this was a one-way situation, you'd "merely" have a train of packets flying around forever, never dying, because there's no such thing as a TTL at this layer of your network. But, since it's happening in both directions, you actually get double the fun.

None of these packets stop being forwarded, and new ones show up, so before long, your switch is doing nothing but spraying those frames everywhere just as fast as it can. Eventually you starve out all other traffic, and everything probably grinds to a halt.

Eventually, the organization learns about things like switches, STP, 802.1x, and not having massive broadcast domains, and that crazy chapter of their history ends. However, just down the street, yet another company is waiting to add it to their own history.

So, here's a challenge for anyone who's managed to get this far: pick a nice quiet time outside of business hours, declare a maintenance window, then go deliberately loop back some Ethernet ports. Look in offices, conference rooms, and on the backs of things like VOIP phones. Be creative. See if your network actually survives it.

Or, you know, don't. If you don't test it, someone else will do it for you... eventually. They won't wait for a quiet time and won't declare a maintenance window, and they sure won't know to unplug it when things go sideways, but that's life, right?

Oh, finally, don't forget to go back to that nice person and tell them that they are okay to plug their machine back in, power it up, and re-enable the network. Then also tell them they didn't cause the whole business to tank for multiple hours, because they might be sweating bullets that somehow the whole thing will land on their head and poison their next performance review.

People deserve to know when they didn't cause a problem, particularly when they think they did! Don't forget that.

Resolving a minor turkey day networking mystery

One of the things about this time of year is that I get to play sysadmin for far-flung parts of the extended family "network" which I normally can't lay hands on directly. This is when everyone gets their updates, security patches, and random wacky devices installed and set up.

This means there are some long-standing mysteries which I only really get a chance to poke at once a year, and frequently only from a smart phone. You can't do the full suite of poking and prodding, so it can be less than satisfying.

This year, however, I managed to figure one out.

There's been this one location that has never been able to load up the normal DOCSIS status page which is usually served over http from 192.168.100.1, port 80. Port 443 also hangs. However, port 8080, which seems to be something else entirely (the spectrum analyzer)? That works great.

This has been the situation for a while... but this year, I have my actual laptop along, and my brain is back online, having not been recently drained of energy by a regular 9-5 situation. It was time to get into their gateway device and do some tcpdumping.

I've replaced actual values with some placeholders below.

First, let's look at what works: port 8080.

20:52:35.490717 GW_MAC > ISP_MAC, ethertype IPv4 (0x0800), length 74: GW_PUBIP.60683 > 192.168.100.1.8080: Flags [S], seq 1736857109, win 14600, options [mss 1460,sackOK,TS val 608614883 ecr 0,nop,wscale 6], length 0

Here's our SYN going out, and it's going to the ISP's gateway based on the hardware address, unsurprisingly, since 192.168.100.1 is not a local address. However, the cable modem grabs onto it and responds from its own hardware address:

20:52:35.492376 CM_MAC > GW_MAC, ethertype IPv4 (0x0800), length 74: 192.168.100.1.8080 > GW_PUBIP.60683: Flags [S.], seq 2438961944, ack 1736857110, win 17376, options [mss 1464,nop,wscale 0,nop,nop,TS val 3992162 ecr 608614883], length 0

The rest goes as you would expect and works fine. Where it gets wacky and weird is when you try to hit ports 80 or 443, and it ends up doing something like this. First, the outgoing SYN, which is otherwise just like the one to port 8080:

20:54:36.593655 GW_MAC > ISP_MAC, ethertype IPv4 (0x0800), length 74: GW_PUBIP.40216 > 192.168.100.1.80: Flags [S], seq 1269309359, win 14600, options [mss 1460,sackOK,TS val 608626994 ecr 0,nop,wscale 6], length 0

Do we get a SYN/ACK back? No. We get ... this:

20:54:36.595446 WTF_MAC > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has GW_PUBIP tell 192.168.0.1, length 46

Some completely new Ethernet address we've not seen before this point (with a unique OUI) comes up on the network and starts demanding that someone tell it the hardware address of the host which just poked it with a SYN. Even though, you know, that host can poke port 8080 on there and NOT generate the same response.

Also, what's this 192.168.0.1 business? That's no IP address that anyone at this site has deliberately put on that cable modem. I sure didn't put it there. It just decided to make that ARP request and use that as the "who's calling" source address.

Unfortunately, Linux cares about these things. It looks at that, and thinks, hey, 192.168.0.1, I have a route for you... down the INTERNAL interface (you know, the one facing the family's devices in NAT-land). But you just spoke to me on the external interface. You're a fake! Or a spy! Or something else evil. I hate you and I'm going to ignore you.

And so it does, but not before logging something about it:

(date) (time) (hostname) kernel: IPv4: martian source GW_PUBIP from 192.168.0.1, on dev eth0
(date) (time) (hostname) kernel: ll header: 00000000: ff ff ff ff ff ff (WTF_MAC) 08 06 ..............

The cable modem never gets an answer, so it can't direct the SYN/ACK anywhere, and so the TCP connection never comes up.

After spotting this, I tried a variant on my stupid network tricks from years ago and pointed 192.168.0.1 (which is unused on the "real" internal network) back OUT to the cable modem. I did all of the fakery from that old post, and ... it worked!

In preparing this post tonight, I came back to it and realized that's even easier than that. You just need to make sure you have a route for that host pointing back out that way, and so you can get away with just this:

route add -host 192.168.0.1 gw default.ip.from.isp

Basically, look up whatever IPv4 address your ISP handed you as a default gateway (presumably via DHCP), and jam in a /32 route for it out that pipe. Linux will match that against the incoming ARP request, figure it's not a martian, and will answer. Everything else just works.

I should point out that this effectively kills any opportunity to use that IP address on your internal network. If you actually had your router/gateway itself there, this won't work. Fortunately, the site in question had a bit of sanity and had it somewhere else already.

It's interesting to note that this only bites you if a whole bunch of things are true.

First, you need the specific cable modem model which has this weird behavior, in this case an Arris SB8200. (It may also involve specific firmwares, but that's not exactly a trivial thing to test considering how DOCSIS devices work.)

Second, you need a box that does "martian filtering", i.e., reverse-path checks to see if a source is down the same interface it'd use to deliver a packet.

Next, you have to have some networking scheme whereby 192.168.0.1/32 is seen as being on the inside of your network, thus triggering the "martian" scenario.

Change models, gateways, or IPv4 networking schemes, and odds are you'll never see this.

It remains to be seen why the cable modem behaves this way. My guess, knowing very little about the innards of such beasts, is that maybe they have two different IP stacks in there, and they don't play by the same rules. It's a pretty stark difference in behavior when you go from ports 80/443 to 8080. It's almost like there are multiple tiny embedded devices all trying to serve bits of the TCP port space on 192.168.100.1 and they all have their own ideas about how to handle layer 2 stuff.

Can we get going with IPv6 for this kind of stuff already? This death-to-my-sanity-by-RFC-1918-conflicts business in IPv4 isn't fun any more.

I really hope you didn't need to know any of this.

...

Epilogue: in the time since I did my first investigation and now, something about "WTF_MAC" changed. Comparing my notes, I can see that both are the OUI of 02:10:18, but the last 24 bits changed. So, not only is it some wacky address that doesn't match the "normal" part of the cable modem, it doesn't even stay the same!

Is this some Broadcom placeholder thing? Information about that OUI online is sketchy at best. There aren't that many starting with "02", either. I hadn't seen that since my ancient 3com ISA cards back when I first got into this stuff.

So weird.

Group Chat Rules

There's no group chat member more enigmatic than the cool person who you all assume has the chat on mute, but who then instantly chimes in with no delay the moment something relevant to them is mentioned.

Shit week (or two)

By noreply@blogger.com (RevK)

Some times things go wrong, it happens, but if I did not know better I would say some deity was out to get me the last couple of weeks!


  • So, re-installed original stepper motor, tightened, applied thread lock, and bingo, working perfectly again, phew. Back in business.
  • I then tried to mill a new board on my milling machine. I have done a lot of these, and I am using a simple bed-levelling program I wrote to measure the corners and adjust the cut over the board to allow fine tracks to be reliably printed. But this one new board (which was bigger than previous ones, and has some long straight cuts) was not cutting correctly - turns out I had a bug in my code that got the Z level wrong (was for previous point not current point) causing some tracks to cut at wrong depth. Just enough to not show up most of the time, but create duff boards several times.
  • So I was improving the code, and making it probe several times for each point and more slowly, and so on....
  • And just then the main milling motor on my milling machine stopped working. It would spin up slowly and then grind to a halt.
  • I tried on a bench supply and it is fine, so clearly the power supply not the motor.
  • I ordered a 48V 3A power supply, and that was not beefy enough to run it.
  • I ordered a nice 10A bench supply, which I am waiting for now - will it work - I'll add a PS to this post later today :-)
  • In the mean time, with 3D printing working, I had managed to make one working PCB, so made a case for the latest widget and it did not fit. For some reason I had made a silly mistake in measuring the parts, which took my about 6 attempts (at over an hour per print) to get right - this is crazy.
  • However, the slightly wrong print (due to design, not printer, this time) meant the button cell battery holder moved, pulling tracks off the board, shorting it, and that took me ages to debug as the GPS refused to work with no backup battery.
  • I set up the Quectel M95 mobile module, and surprise surprise all of the commands I need are different to the SIM800 module I was using. Arrrg! Thankfully they have a good manual. So new GPS tracker working well.
  • Then something else stopped working on the board - I don't know what yet - I need to use my magnifying glasses to work on this board, and guess what, they just snapped.
  • And in the middle of this, other issues, involving someone having unexpected hospital visits (which have turned out fine, phew).
It has been a shitty couple of weeks really.
It can only get better :-)

P.S. And not the plastic cover thing on end of my glasses has come off so I stabbed myself putting them on.

P.P.S. the cheapest, unbranded, iffy looking DC power supply I have seen, from Amazon... Actually has a CE mark, but not sure I'll ever leave it unattended. But it works - yay!

In Rust We Trust: Stob gets behind the latest language craze

By Verity Stob

My, what beautiful curly braces you have

Stob  Oh hey, Verity, how's it going? What have you been up to? Busy, busy, busy?

buses and bicycles (in Oxford)

By danny

Bicycles have featured a lot in my blog, but the other key to sustainable transport, in Oxford as in much of the world, is the bus. Buses are central to Oxford's existing transport and will need to play an even bigger role in any sustainable future. More generally, such a future requires the world to […]

Troubleshooting shell scripts that grew slower with age

Sometimes, I start writing posts, and then shelve them because they aren't quite ready, or I don't feel like putting them out, or maybe it's just not clicking into place, or who knows what. Some of them hang out "on ice" for quite some time and may never surface.

Here's one which was written about four years ago and yet was never released. I did mention it in passing early in 2017, but that post deliberately skipped the full technical dive into what was happening.

So now here we are, over four years later, and bash has been fixed, so this probably doesn't affect you any more. But, for a while, it did, whether you realized it or not. I've decided to post it anyway since it still seems like an amusing troubleshooting romp. I've punched it up to fix some of the parts which fell flat originally (which is probably why I never posted it before).

To be clear: this problem was reported and fixed a long time ago. So, unless you have a truly ancient bash on your machine, this does not affect you.

Enjoy.


Lots of systems run processes which are started and are supposed to never end. A bunch of them are just overgrown shell scripts. Maybe you've seen one, or even written one. It might look something like this:

#!/bin/bash
 
while true
do
  thing1 | thing2
 
  other_thing
 
  sleep 1
done

This shouldn't come as any surprise, but people write this kind of glue constantly. What might be eye-opening is that it can make you waste a stupid amount of CPU time in bash itself. This isn't some kind of "interpreted things like shell scripts are slow" rant. No, this is more like "bash is using a linked list to try to store all of the child pids which it has created, and linked lists get slow after a while".

Our dumb little example script up there runs "thing1", "thing2" and "other_thing" once a second. That means that we get at least three new PIDs which now have to be tracked. Have you ever noticed that you can refer to processes as "%1" and it'll know which one that was? Or, have you ever done a "wait" in a script and noticed it knew which children to wait for? It has to track those someplace, and it happens in a C struct called "bgpids":

struct bgpids {
  struct pidstat *list;
  struct pidstat *end;
  int npid;
};

Okay, technically, that's not the list, but it points to it. It points to the head and the end, specifically. It also has a counter. The actual list happens in the "pidstat" struct:

struct pidstat {
 struct pidstat *next;
 pid_t pid;
 int status;
};

What would you think if I told you the linked list just gets bigger and bigger as your script runs longer and longer? It has a maximum size, sure, but that limit is probably high enough to consume a whole bunch of CPU time once your script has been running for long enough. Why does this happen? Let's dig into the details.

The ultimate size of that list is influenced by the number of unique PIDs that bash can create while running your script. This might sound a little weird, so let's take a step back for a moment to explain.

Let's say you have a fairly boring Linux box. Your kernel.pid_max (look in sysctl or /proc) is probably set to 32768. That means your system starts with 1 (init, probably from systemd these days) and goes up from there. Once it uses 32768, it wraps back around. Naturally, it skips those in use, so 1 won't get used again, but eventually, you will get "low" pid numbers showing up for brand new processes again.

Assuming the script runs long enough, the child processes will probably end up with every single process ID in that range, minus the ones where things started and never stopped -- init, sure, and things like crond, syslogd, sshd, or whatever. It'll probably be MOST of the space between 1 and pid_max!

Back to our list. Every time bash spawns something, it goes and looks in that list to see if perhaps it's already seen that one before. It not, it adds another entry. The list gets longer and longer, and the number of checks keeps going up. Comp Sci people should start thinking "big O notation" if you haven't already.

I'd guess that you'll probably end up with 32000 entries in that linked list eventually. What this means in practice is that bash will flip through all of them over and over again. It's using CPU time to do all of this pointless housekeeping, and it's taking longer and longer to come back around and start your stuff up.

Here's a chunk of code from inside bash's bgp_delete():

  for (prev = p = bgpids.list; p; prev = p, p = p->next)
    if (p->pid == pid)
      {
        prev->next = p->next;   /* remove from list */
        break;
      }

This happens any time something new starts. I grabbed the source and jammed a little bit of instrumentation in it (in my usual "debug via printf" fashion) to show exactly what's going on. It just increments that number inside the loop and prints it once we drop out.

  fprintf(stderr, "XXX ran %d comparisons\n", comparisons);

With that in place, running the new binary shows how quickly it adds up:

$ ./bash
$ touch /tmp/foo
XXX ran 0 comparisons
$ touch /tmp/foo
XXX ran 1 comparisons
$ touch /tmp/foo
XXX ran 2 comparisons
$ touch /tmp/foo
XXX ran 3 comparisons
$ touch /tmp/foo
XXX ran 4 comparisons
$ touch /tmp/foo
XXX ran 5 comparisons
$ touch /tmp/foo
XXX ran 6 comparisons
$ touch /tmp/foo
XXX ran 7 comparisons

Surprise! It happens for interactive shells, too. I originally started writing this to talk about what happens when you have a long-lived script, but it turned out to also happen in long-lived interactive sessions too. The more you start, the more gunk there is hanging out in there.

Now that we know what we're dealing with, let's torture the shell a bit and see just how bad it is. I'll start from a clean bash-4.3.30 from upstream and run the following stupid script which will expose the problem without making me wait forever.

#!/bin/bash
while true
do
  find /dev/null > /dev/null
done

Note: this is a deliberate pathological example since it doesn't sleep. That just means it shows the problem sooner rather than later. Given enough time, even a script with a sleep in it will slow down to this degree.

I called it "loopy" since that's what it does, and what it is. Now it's time to run my demented little script. Here, I demonstrate that the kernel's limits are boringly low (32K) and my session has a fairly low limit on how many processes it can create, and then start it up in the just-built bash:

$ /sbin/sysctl kernel.pid_max
kernel.pid_max = 32768
$ ulimit -u
15940
$ ./bash ../loopy

The loop prints nothing, so that's all that shows up on the terminal. After waiting a bit to let it get nice and saturated, I can clearly see the CPU time suck from another terminal: to look at the CPU time spent by this thing.

$ ps -o pid,pcpu,etime,time $(pgrep -f loopy)
  PID %CPU     ELAPSED     TIME
27778 26.6       02:12 00:00:35

There we go! After two minutes, we've been on the CPU about 35 seconds, or about 26% of the time. That's the time spent by the shell itself, NOT the work done by "find"! That's a ton of overhead just to run the same stupid do-nothing process forever, and that's how I caught this originally.

Okay, so just how bad does that structure look now? Let's just peer inside with the help of the debugger:

$ gdb -batch -ex "print bgpids.npid" /proc/27778/exe 27778 
 
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
0x00007f592dd193ec in waitpid () from /lib64/libc.so.6
$1 = 15940

So, notice, 15940 is the very same number as the max limit on user processes for that session. I can raise it, but unfortunately the shell picks that up when it starts when it does "sysconf(_SC_CHILD_MAX)", and that affects how big the list can get. That means it's too late to change it for that shell (pid 27778), and that one has to go.

Recall that I'm running this fresh build of bash-4.3.30 from inside my usual shell, so I can just identify that, raise the limit on it, and THEN start up a new child bash-4.3.30 running "loopy", and it'll pick up the new "maxchild" value, which will let the list grow larger.

So let's just abort that thing... and find out what this parent shell is:

$ ./bash ../loopy
^C
$ echo $$
7030

Right, so the parent is 7030. From a root shell elsewhere, I'll just bonk it with prlimit to give it some magic powers:

# prlimit --pid 7030 --nproc=65000:65000

Back in 7030, it's now visible, so let's get going again.

$ ulimit -u
65000
$ ./bash ../loopy

After letting that soak for a while, here's what it looks like in ps:

$ ps -o pid,pcpu,etime,time $(pgrep -f loopy)
  PID %CPU     ELAPSED     TIME
 1542 21.2       02:14 00:00:28

Strangely, it actually goes down. It turns out to be really expensive at first, but eventually it almost sorts itself out. The pids eventually wrap raound, and it manages to pull off a cleaning run.

After jamming way more debug printf cruft into the shell, it now looks like this:

$ ./bash ../loopy
XXX bgp_delete: pid=7261 (npid=0)
XXX ran 0 comparisons
XXX bgp_delete: pid=7262 (npid=1)
XXX ran 1 comparisons

... skipping thousands of lines ...

XXX bgp_delete: pid=7258 (npid=32429) XXX ran 32429 comparisons XXX bgp_delete: pid=7259 (npid=32430) XXX ran 32430 comparisons XXX bgp_delete: pid=7261 (npid=32431) XXX *** removed *** XXX ran 1 comparisons XXX bgp_delete: pid=7262 (npid=32431) XXX *** removed *** XXX ran 1 comparisons XXX bgp_delete: pid=7263 (npid=32431) XXX *** removed *** XXX ran 1 comparisons

There it is: eventually it wraps all the way around, scores a hit, and deletes it. npid is still massive, but now it's starting to find the IDs at the beginning of the list, so it can jump out of the list quickly instead of hanging around to burn CPU time.

Of course, there's a catch. This was happening on a really quiet box with almost nothing else going on. I only powered it up to do the digging for this post and very little else runs on it. Let me show you what happens when there are other processes coming and going, so the PID numbers aren't all in a neat line, and it won't manage to get lucky by finding them "up front" after it wraps.

This is what happens with a bit of process churn happening:

XXX bgp_delete: pid=18736 (npid=23404)
XXX *** removed ***
XXX ran 7632 comparisons
XXX bgp_delete: pid=18738 (npid=23404)
XXX *** removed ***
XXX ran 7632 comparisons
XXX bgp_delete: pid=18740 (npid=23404)
XXX ran 23404 comparisons
XXX bgp_delete: pid=18743 (npid=23405)
XXX *** removed ***
XXX ran 855 comparisons
XXX bgp_delete: pid=18745 (npid=23405)
XXX *** removed ***
XXX ran 855 comparisons
XXX bgp_delete: pid=18746 (npid=23405)
XXX *** removed ***
XXX ran 7633 comparisons
XXX bgp_delete: pid=18748 (npid=23405)
XXX *** removed ***
XXX ran 7633 comparisons
XXX bgp_delete: pid=18751 (npid=23405)
XXX *** removed ***
XXX ran 857 comparisons
XXX bgp_delete: pid=18752 (npid=23405)
XXX *** removed ***

Uh oh. Now things are all over the place and we're running bunches of comparisons before finding our match. Since other things are running on the box, we usually don't end up with n, then n+1, then n+2 because something else gets those instead. The number we get isn't at the beginning of the list, so the shell has to dive deep to find it.

Also, notably, this happens well before mixing out "npid":

XXX bgp_delete: pid=30379 (npid=21034)
XXX ran 21034 comparisons
XXX bgp_delete: pid=30381 (npid=21035)
XXX ran 21035 comparisons

In other words, on a normal machine, you will hit this much sooner, and it probably won't drop off to relatively little CPU utilzation. Also, if you run multiple long-lived looping scripts like this, they will all interact with each other, and will collectively make themselves much slower!

Now let's throw a nice wrench into the mix here and make things even more interesting. For a lot of places running large machines with many hundreds of processes, each with hundreds of threads, you quickly realize that even 32768 pids just isn't enough.

Incidentally, when you DO run out of pids, fork fails. Check those return codes, folks.

It's time to raise the limit from 15 bits (32768) to 22 bits (4194304) and run it again with the box in a relatively quiet state:

$ /sbin/sysctl kernel.pid_max
kernel.pid_max = 4194304
$ ulimit -u
65000
$ ./bash ../loopy
... wait a bit ...
XXX bgp_delete: pid=77782 (npid=65000)
XXX ran 65000 comparisons

Check that out. Now we won't even have a chance of finding a match (and thus ending the loop sooner) until that PID number wraps. Given that it's now something like 4.2 million before that happens, that'll be a while. Even then, we're still subject to luck and how busy the machine may be. At this point, the odds are good that we'll be doing 65000 comparisons every single time.


As mentioned before, this was fixed a long time ago.

I wonder just how much global CPU time was clawed back by that one fix.

Extending proprietary PC embedded controller firmware

I'm still playing with my X210, a device that just keeps coming up with new ways to teach me things. I'm now running Coreboot full time, so the majority of the runtime platform firmware is free software. Unfortunately, the firmware that's running on the embedded controller (a separate chip that's awake even when the rest of the system is asleep and which handles stuff like fan control, battery charging, transitioning into different power states and so on) is proprietary and the manufacturer of the chip won't release data sheets for it. This was disappointing, because the stock EC firmware is kind of annoying (there's no hysteresis on the fan control, so it hits a threshold, speeds up, drops below the threshold, turns off, and repeats every few seconds - also, a bunch of the Thinkpad hotkeys don't do anything) and it would be nice to be able to improve it.

A few months ago someone posted a bunch of fixes, a Ghidra project and a kernel patch that lets you overwrite the EC's code at runtime for purposes of experimentation. This seemed promising. Some amount of playing later and I'd produced a patch that generated keyboard scancodes for all the missing hotkeys, and I could then use udev to map those scancodes to the keycodes that the thinkpad_acpi driver would generate. I finally had a hotkey to tell me how much battery I had left.

But something else included in that post was a list of the GPIO mappings on the EC. A whole bunch of hardware on the board is connected to the EC in ways that allow it to control them, including things like disabling the backlight or switching the wifi card to airplane mode. Unfortunately the ACPI spec doesn't cover how to control GPIO lines attached to the embedded controller - the only real way we have to communicate is via a set of registers that the EC firmware interprets and does stuff with.

One of those registers in the vendor firmware for the X210 looked promising, with individual bits that looked like radio control. Unfortunately writing to them does nothing - the EC firmware simply stashes that write in an address and returns it on read without parsing the bits in any way. Doing anything more with them was going to involve modifying the embedded controller code.

Thankfully the EC has 64K of firmware and is only using about 40K of that, so there's plenty of room to add new code. The problem was generating the code in the first place and then getting it called. The EC is based on the CR16C architecture, which binutils supported until 10 days ago. To be fair it didn't appear to actually work, and binutils still has support for the more generic version of the CR16 family, so I built a cross assembler, wrote some assembly and came up with something that Ghidra was willing to parse except for one thing.

As mentioned previously, the existing firmware code responded to writes to this register by saving it to its RAM. My plan was to stick my new code in unused space at the end of the firmware, including code that duplicated the firmware's existing functionality. I could then replace the existing code that stored the register value with code that branched to my code, did whatever I wanted and then branched back to the original code. I hacked together some assembly that did the right thing in the most brute force way possible, but while Ghidra was happy with most of the code it wasn't happy with the instruction that branched from the original code to the new code, or the instruction at the end that returned to the original code. The branch instruction differs from a jump instruction in that it gives a relative offset rather than an absolute address, which means that branching to nearby code can be encoded in fewer bytes than going further. I was specifying the longest jump encoding possible in my assembly (that's what the :l means), but the linker was rewriting that to a shorter one. Ghidra was interpreting the shorter branch as a negative offset, and it wasn't clear to me whether this was a binutils bug or a Ghidra bug. I ended up just hacking that code out of binutils so it generated code that Ghidra was happy with and got on with life.

Writing values directly to that EC register showed that it worked, which meant I could add an ACPI device that exposed the functionality to the OS. My goal here is to produce a standard Coreboot radio control device that other Coreboot platforms can implement, and then just write a single driver that exposes it. I wrote one for Linux that seems to work.

In summary: closed-source code is more annoying to improve, but that doesn't mean it's impossible. Also, strange Russians on forums make everything easier.

comment count unavailable comments

Utter frustration with the eval board's tech support

While most of the machines I work on are x86 or x86_64 based, it's not always the case. There are a significant number of alternatives that you start encountering when you get off the beaten path and start talking about weird embedded projects and other things that stop resembling general-purpose computing.

In this world, you get into situations where you have something like an "evaluation board" from a vendor, and you use it as your test bed. You take whatever it is you were working on, and figure out how to get it to run on that platform. If it works out, then maybe you design your actual product around the chipset, or CPU, or ASIC, or FPGA, or whatever it is, and that's what gets shipped.

Still, long before it's a product, it's a weird little board on your desk with a bunch of cables connecting it back to your "real" computer. That's where I was at one point when this story happened: trying to get a system up and running on a weird environment I had never used before.

Obviously, something went wrong, or I wouldn't be here writing about it today. Oh yeah, something definitely happened, and I gained a new entry on my list of companies I never want to deal with ever again because they clearly don't give a damn about their customer experiences.

There I was, with an eval board fresh out of the box. I took all of the usual anti-static precautions, and gingerly set it up on my desk. I connected it to power and did the hardware test it tells you to do: flip these DIP switches this way, then push these buttons, verify these LEDs do this thing, then flip them another way, and so on.

It passed this just fine, so I flipped the DIP switches from test mode to "boot from SD card" mode, and fed it a card with an image full of their official dev environment cruft. Then I plugged it into the network, sshed in from my workstation, and started doing stuff on it. My main objective was to make sure all of the stuff I had created actually compiled and ran on this architecture and could be packaged up. After all, it would eventually run on a much bigger version of this thing, and if it didn't work here, it sure wouldn't work there.

I started up a big compile job for a massive framework and turned my attention to other things. Time passed, and I noticed at some point that my ssh had stopped scrolling. I waited for a while, figuring it was just chewing on a really big compiler or linker command, but it didn't help. ^C did nothing. Nothing happened on the serial console, either. I couldn't open a new connection to it. Nothing worked.

I powered it off, waited a few seconds, and powered it back on. Nothing happened. Normally the board would spit some random junk down the serial lines as soon as it got power, but this time it said nothing. I tried this a few times, and got the same results: nothing, nothing, and more nothing.

Thinking back to the self-test, I figured it was time to see if that would still pass. I grabbed my pen and went to flip the DIP switches back to test mode, and the head of one of them snapped off. The actual switch body was still intact, and it could be moved around with a pin or something else small enough to get in there, but the usual white pointy bit was long gone. Still, I'm sure it got flipped into test mode, and yet, that didn't work either. The board was just plain dead.

I reported this to the people who were in charge of the hardware at this particular concern and somehow, it wound up turning into "make your software engineer re-check the board". No amount of "trust me, it died without me even touching it, during a hands-off compile" would make them take me seriously, so I decided to humor them. I took the board back to my desk and tried it again, and again, and again. I tried different SD cards. I tried all kinds of DIP switch settings. I poked at the different serial ports on the thing, thinking maybe one of them would work -- they're not all consoles, after all. Some of them let you talk directly to the subsystems on the board, believe it or not. Those were also dead ends.

All the while, the customer support person from this vendor was going back and forth with us in e-mails, asking for us to do this, and that, and none of it was changing anything. This is when I made a request that I wish I had followed through on. I basically offered to buy the board from the company I was working with, just so I could take it out to the parking lot and destroy it with a hammer. Then we could get back to actually working, you know?

They didn't go for that (obviously) and the next step was to set up a call with this support tech. Of course, instead of it just being a phone call with actual phones involved, he wanted it to be a "webex". A what? I had to look this up, since all I knew of that name was "things that happen at soul sucking companies that I'm glad I've never worked for", and "a name on an office building by Great America".

I thought it meant "video conference call", and I've done plenty of those, so I was okay with that. Then I found out it involved far more. The link they sent actually tried to install all kinds of other stuff. So, no, what webex really means is "video conference call" with a healthy side of random shit we install on your machine TO TAKE IT OVER REMOTELY.

I went to join the call and my browser lit up like a Christmas tree. Install this extension? Run this application? NO! NO NO NO NO.

Yes, they expected me to load all kinds of sketchy crap on my machine so this guy could jump through it and poke at their board remotely. I swear I am not making this up. Obviously, I've done the sysadmin thing for long enough to know that you do not ever let random crap like this onto your machines or network, and reported it to the security/IT folks there. They agreed: absolutely no remote access for these weenies.

I said as much to the support person, and so, finally, it turned into a regular phone call... a call in which I managed to copy maybe 30% of what this person was saying due to the terrible audio quality, lag, and who knows what else.

I have chat logs of talking with friends during this. They all boil down to "KILL ME NOW". I spent over an hour putting up with this crap, trying to convince him that no, the broken DIP switch does not excuse you from dealing with this problem. The machine was broken BEFORE the little plastic bit split off. It has nothing to do with that. Take the damn thing back and RMA it, already! He wasn't having it.

This dragged on for days, with them wasting my time and refusing to do anything useful about the situation. Someone commented they "could feel my frustration from all the way across the office".

It's like a curse. If something's the least bit broken, I am probably going to trip over it. If there's a "stupid tree", I'm going to fall out of it and hit every single branch on the way down.

I used to read jwz's writings back in the day and initially thought "he must be really bad at this stuff since it keeps breaking". How foolish I was. I've since realized that all of these things happen to me, too.

As for the problem with my board? It got solved in a most unusual fashion. I randomly happened to meet some people from another hardware company. This one was more of an integrator place, such that they actually build the products that you'd end up buying with the chip from the place with the idiot customer support. That is, if the chip came from company X, it'd be rolled into a board and a box by company W, and that is what would get shipped as the final product.

I guess they saw the board on my desk, since we got to talking about it and the bigger situation, and they thought it was batshit insanity. They actually called up the support guy at company X and chewed him out. The gist of their conversation was "what, why would you tell the customer to do that?". Finally, someone listened.

The vendor finally took the board back, and that was the last I heard of it. As for my project? I got the lab people to check out a second board to me, and I carried on using it. The second one didn't suffer from infant mortality, and so I was able to get my actual work done. What a concept, right?

The Long Drive

Over a thousand miles over some of the loneliest areas of the USA. What's not to love?

Travel Equipment: 2019 Edition

Taking a look at some of the key travelling equipment I've grown to like in the last year of travel.

ASGI 3.0

Upgrading the ASGI spec to simplify it, while keeping backwards compatibility.

A Django Async Roadmap

Taking a look at what it would take to make Django async-native, what it enables, and if we should even do it at all.

Python & Async Simplified

Event loops, coroutines and awaits, oh my!

Channels 2.0

Finally, the promised land is here and Channels 2.0 is released. But how much has changed? And why?

The Sheets Of San Francisco

Finally, my 3D city maps return, and this time they're mapping the streets and hills of San Francisco.

Channels 2: October

What's happening in the world of Channels? How many times have I rewritten the API now?

New Zealand

Adventures in the land of volcanoes, unique birds, fjords, and long white clouds.

Towards Channels 2.0

Outlining my plan for iterating on Channels' design, and what the future might hold for both Django and Python in general.