General Notice

We at Wurldtech Security Technologies continue to discuss ways to provide and collate the most useful, relevant, and industry driven perspectives on the issues of industrial cyber security. As such, we have implemented some changes to our blog. We certainly appreciate each of our readers continued support and insight! Considering the comments, links, and re-posts that we’ve received, we know that others find this material useful in their own practices.

A couple of noteworthy changes:

• Blog Content: I (Bryan Singer) will continue to write on industry perspectives, current issues, think pieces and positional arguments on challenges facing this industry, and key events that are all relevant to issues faced today. Dr. Nate Kube will write on the value, benefit, features, ROI, and other topics related to device testing. Dr. Kube is one of the most well recognized premier talents in formal test methodologies and testing strategies for industrial components. His insight and experience continues to offer key value and strategies for uncovering issues and optimizing time to market for vendors, asset owners, and any others seeking to improve the reliability and resiliency of their industrial environment. Frank Markus will write on penetration testing, vulnerability discovery, security assessment, and other key technical issues on industrial security. Steve Kim will also drive key industry news, announcements, and other useful topics. Other authors will occasionally drop by to add their thoughts as well. With this team, we are confident that this blog will continue to drive very useful content to market!
• Comments Allowed: Part of the value in forums such as this comes from interaction with readers. Originally, this blog was more of an “announcements” page, but that has grown over time. In the next phase of this significant effort, we openly invite and ask that any and all readers submit their views, questions, and other comments. We continue to maintain some of the highest standards in the industry for responsible disclosure, and as such we will require free registration and moderated comments to ensure that no sensitive information is accidentally disclosed. Register, read, and add in your perspectives and questions today!

Thanks to all the supporters and people in the industry that read and comment. Healthy discourse and discussion only serves to assist industry in the continued challenges and opportunities in industrial security!

There has been a lot of talk out there recently around FUD and public disclosure, with the usual pundits weighing in on all sides. With videos such as the Aurora demonstration, Ira Winkler’s RSA presentation, and a whole host of other inflammatory articles, people are wondering whether such articles are doing any good or not.

I have heard some great analogies lately as well. Notably my favorite came from Jake Brodsky during a podcast interview he and I did together over at Digital Bond’s website with Dale Peterson. He likened such articles to an “Energy Drink.” Great analogy: You get a big buzz and lots of energy, and then an even greater crash. That seems to be what happens every time an article like this comes out.

I have my own analogy, and I must pay tribute to an old confidant since my early years, Col R. B. Thieme. Regrettably one of my longest known advisers is succumbing to Alzheimer’s disease at nearly 90 years old, so it seems fitting that I should borrow from such a person. The Charge of the Mosquitoes: while one is not enough to get you, over time they eventually overwhelm you. This is what I fear such articles are doing to the state of industrial security. While there may be some short lived awareness benefit, the greater fear to me is that we are desensitizing ourselves to the issue. In absence of PUBLIC events, every time something like this comes out, and we still have not seen an incident, people believe less and less, to the jeopardy of the whole issue. It’s what I and many others call the “September 10th” syndrome… the fact that few would have imagined the possibility of 9/11 until it happened.

This is a product of human psychology, known as “Cognitive Dissonance Bias” in which our brains tend to automatically reject possible outcomes based upon biases formed by our own experiential base. It is especially powerful with NEGATIVE analysis. For example: Go to a gem field…Turn over five rocks and find five gems, you will likely think you are very “lucky” but not tend to believe that it will happen under any of the next five rocks. On the other hand, turn over five rocks and find no gems, you will likely conclude that you will not find any gems at all. The problem is that this has absolutely nothing to do whether or not there are any gems under any other rocks… Absence of previous data is no indicator of future absence as well, as in security. Just because there is no evidence of past events doesn’t mean it won’t ever happen. Quite likely the opposite is true, but own cognitive dissonance bias drives us to believe that nothing will happen. Yes I know this is a simplification, but the intent of this article isn’t to go through the various ways that people’s minds can be persuaded to perceive success versus failure.

Hearing things like these inflammatory articles only helps to encourage the process. It makes people reach conclusions of “this can’t happen to us” even more quickly by steeling their minds over time, to the point that eventually they will discount it completely. I’m reminded of a story of an old mountain man near Mt St Helens before it blew. Geologists and experts warned everyone to leave, but he was actually interviewed as saying that because nothing had ever happened in all his years, nothing was going to happen. Apparently he never learned that geology doesn’t change on the scale of human lives. Talk about a misstep, he is among those never accounted for. While awareness is good, FUD and inflammatory articles are not, it is making the challenge of implementing security even more difficult. In Part 2 of this article, I intend to address some ways I think we need to change the message.

We are proud to showcase some coverage on Dr. Nate Kube’s two presentations at the Yokogawa Technology Fair & User Conference 2008 in Houston, TX.

Courtesy of Control Global, here are the links to recaps of his talks:

Talk 1: Best Practices in Securing Process Automation Networks (April 9, 2008): http://www.controlglobal.com/articles/2008/129.html

Talk 2: To Fix System Bugs, Wurldtech Gets Fuzzy Wid It (April 10, 2008): http://www.controlglobal.com/articles/2008/132.html

Enjoy.

Yesterday (April 10, 2008), the Sound OFF! blog mentioned an article, “Industrial Control Systems Killed Once and Will Again, Experts Warn”, written by Ryan Singel from the WIRED Blog Network (April 9, 2008).  Unfortunately, a past cyber incident has now been linked to a fatality.  The incident in question is the tragic rupture of a pipline which spilled 237,000 gallons of gasoline into two creeks near Bellingham, Washington.  The gas ignited and killed three, injuring eight others. 

A recent re-examination by security experts into this tragedy has revealed this incident was due to a control system computer issue.  The finding brings new impetus for the need for cyber-security standards and security policy for governments, businesses and critical infrastructure organizations…

To read the article in its entirety, please visit: http://blog.wired.com/27bstroke6/2008/04/industrial-cont.html

To see this post on Sound OFF!, please visit: http://www.controlglobal.com/soundoff/?p=2571

As a technology company, this is sometimes kind of hard to swallow. Technology is not the limiting factor in security. We have the technology to lock down something so well that it can’t even be used (which I would say is a failure of security, but that is not the point). But, while we continually look for technology or that latest and greatest component to “solve” our problems, we have to remember that the key is not what technology you acquire, but rather the application of that technology. Any doubters would do well to read the article over at http://www.itpro.co.uk/internet/news/182871/staff-forced-to-bypass-security-controls.html to see the results of their study that suggest 68% of respondents admit to bypassing security controls in order to do their job. Its not a challenge easily solved, but it helps highlight the need that we can’t fire and forget our security controls. They must be thoughtfully implemented, then monitored and audited effectively.

A quick perusal of any daily news site turns out a healthy dose of articles such as “Cell-phones more dangerous than cigarettes,” “Man-Made Global Warming will Cause all the Oceans to Swallow all Land by 2020,” “Credit Crisis will Cripple All Global Economies Until we Move Back Into Caves.” OK, so the last one is an exaggeration, but it illustrates a point. The news media understands that few things on Earth motivate humans stronger than fear. Unfortunately, security professionals work in a risk-based environment where the fear mongers seem to thrive as if it were their natural habitat.

If one were to believe the news sites and pundits, the next great worm will happen tomorrow and it will cripple the entire Internet, forcing civilization back to the stone age. Fear indeed does sell, but it’s not a message one can use for long before “Chicken Little” syndrome sets in. There is a dual frustration among security professionals as well. How does one walk the fine balance between getting people’s attention and not being lumped into the category of fear mongering? Some amount of fear is needed to capture attention, I have previously called it “Enlightened FUD,” but it is easy to go too far. We are, after all, dealing with an extremely difficult logical challenge in that we are trying to prove a negative when we tell people “when” and not “if” another event occurs.

I’m reminded of a time when I went to a “fire safety” presentation, where they fed us dinner, then showed us all these horrible pictures of burn victims and made us believe that our house would burn down that night. Their salesmen kept saying things like, “I just couldn’t sleep tonight knowing you aren’t protected!” They were selling the most expensive fire detection equipment I have ever seen for home use… and people pulled out their wallets and hemorrhaged cash their way. Effective, yes, but I left, and so did anyone else that could apply even the simplest of logic. I recognize that fire is a real risk, and take measures to protect against it. Its not a guarantee, its not perfect, but we also don’t live in constant fear.

So how does one cut through the FUD? Here are a few hints. If you are reading an article and it says things like:
• This bug is sure to be a doozy!
• NO ONE reported this, It’s an Internet First!
• I just found 10 0-Day’s!
• There are 543 vulnerabilities in that product in one scan!
• Widespread outages possible!
• Things are different and we can’t resolve the issues!
• No-one is paying attention!

I could go on all day as I read them every day…. In fact, if the author uses more than a handful of exclamation points, or ever uses the “?!” as part of communicating a message, or types in all caps frequently when not part of an acronym, there’s a good chance it’s a FUD message. Sure we all use these techniques from time to time to make specific points, but it shouldn’t be the basis of the article. Just because you are loud does not make your argument more valid. If you have ever taken a college level course in logic or philosophy, you can almost immediately identify the logical fallacies of appeal to emotion and generalization, they are clear warning signs that should be listened to. These are the kinds of people and organizations that want to create a problem just so they have something to fix.

On the other hand, if you are reading an article from a respected security professional and it says something like, “interesting behavior on port XXXX” or they actually copy and past code snippets, packet captures, or relevant technical details (such as quoting professional articles), or they write about actual effective techniques and procedures, one might want to take notice. It is these folks I tend to listen to, as they usually back up their answer with something more significant than broad assumptions and loosely formed theory. Identify a problem statement, form an argument, support it with reason. The technique is not difficult. These are the tools required on any college paper, legal document, or any other professional thesis, so why are they so often ignored here?

So today the challenge goes out (including to myself). If we want to be taken seriously, write seriously, talk seriously, and get serious. I don’t come in with multi million dollar “answers” and “fixes for all security problems,” and neither will anyone on any team or company I work for. What we do offer are strategies, plans, roadmaps, and key technical solutions that are benefit driven and focus on the problems that need to be solved, and ignore those that don’t. There is no question that there ARE CHALLENGES (sorry, I couldn’t resist a little humor), but the message is that there are solutions, and there are things that can be done to not only improve security, but improve business as well.

For anyone concerned with cyber security and the nuclear industry, this is an extremely informative post (March 24) from Joe Weiss on his Unfettered blog at ControlGlobal.com. 

Reposted with permission from Joe Weiss. 

——–

Nuclear plant cyber security has a ways to go

As a nuclear engineer who has worked inside and outside of the nuclear industry, I have my thoughts on why nuclear plants are so far behind non-nuclear facilities in securing control systems. I spent 5 years managing the EPRI Nuclear Plant Instrumentation and Diagnostics Program. Even though EPRI’s purview is R&D, I did not do “bleeding edge” research on new instrumentation and controls technologies because it would not be useable in nuclear plants until demonstrated elsewhere. I then spent 5 years managing the EPRI Fossil Plant Instrumentation & Controls Program. Here, I was able to do “bleeding edge” research in instrumentation, controls, and communications (I received 2 patents on instrumentation and controls technologies). What became obvious to me was non-nuclear facilities would implement new technologies such as Internet access and modern telecommunications if they thought it would be financially prudent while nuclear plants could not implement new technologies until it was well-proven elsewhere. This means the non-nuclear community has vastly more experience and expertise than the nuclear community in cyber security. Yet, the nuclear community refuses to take advantage of these resources. Why???

The prevailing wisdom is that nuclear plants are isolated and not connected or interconnected. At least for some nuclear plants, that is simply not true! I personally know of many nuclear plants with remote connectivity to and from their nuclear plant networks. One interesting case was mentioned at the Applied Control Solutions Conference in Knoxville last August by a representative from a nuclear utility. He mentioned they installed firewalls between their nuclear plant networks and Corporate network because their nuclear plant networks were infecting the Corporate network with malware, not the other way around.

Commercial nuclear plants have several interesting aspects:
1) Nuclear plants have been viewed as being isolated and immune to cyber events. However, there have been several documented cases where nuclear plants have experienced cyber events. Several other cyber events have occurred that have not resulted in reactor scrams or other “unusual events” and so are not documented.
2) Because all “unusual events” result in some form of NRC notification, it is possible to glean information from nuclear plant events that would not be available from non-nuclear plants.
3) In most cases, nuclear plant personnel have not participated in non-nuclear control system cyber activities such as ISA S99. As mentioned above, this has kept the nuclear industry from obtaining the relevant valuable expertise and experience from others.
4) The nuclear industry guidance for cyber security (NEI-0404) was developed primarily from an IT perspective and is also primarily a programmatic document that does not address the unique aspects of control systems. Similar to the NERC CIPs, NEI-0404 would not have prevented many of the cyber events that have occurred. Moreover, some of the guidance in NEI-0404 potentially could have either caused or exacerbated some of the cyber events that have already occurred.

Other specific details about nuclear plants and cyber security include:

- In the November 2007 issue of Power, there are two articles on nuclear plant networks- “Plantwide Data Networks Leverage Digital Technology to the Max” and “Upgrade your BWR Recirc Pumps with Adjustable Speed Drives”. Both tout the value of advanced communication networks and neither addresses the cyber security vulnerabilities they open. In the first, it is suggested that the plantwide data network (PDN) include process control (DCS, PLCs, etc) and plant communications (public address, radios, cell phones, pagers, etc). It is also suggested that process monitoring, operator support, plant security (physical), and supplemental monitoring/testing be included. These are all good ideas (ironically, 10-15 years ago before cyber security was an issue, I was writing papers and sponsoring research at EPRI encouraging this approach), but they need to include cyber security considerations in which the article is essentially silent. The second article on BWR recirculation pumps going to variable speed drives seems to ignore the Browns Ferry 3 broadcast storm experience. Variable speed drives are definitely provide a productivity improvement and networking the drives are a good idea, but ….you still need to address the cyber component you just opened.

- November 2007, EPRI issued Technical Report 1015087, “Instrumentation and Control Strategies for Plant-Wide and Fleet-Wide Cost Reduction”. The report states: “Coordinated improvements to shared communications and computing infrastructure, plant processes, and organization…”. This statement almost cries out that cyber security will be an issue. The report simply says to consider cyber, not what to do.

- The December 2007 issue of Nuclear News references an IAEA nuclear security technical guidance document. Section 1.3 of the document, “Computer Security at Nuclear Facilities” states: “The protection of the computer systems at nuclear facilities can, in principle, be achieved using the same methods and tools that have been developed within the computer community…”.  This statement is at best misleading. Control systems are composed of an HMI that may be Windows-based and field devices that are not. Traditional business IT security can be applied to the Windows-based HMI. However, for field devices, business IT security (policies, procedures, technologies, and testing) often is NOT appropriate. Several recent nuclear plant cyber events would not have been prevented by traditional IT security. Moreover, they could have been CAUSED by applying inappropriate IT security techniques.

The recent nuclear plant cyber incident resulted in an automatic scram from settings that closed valves. The cause is not one that has been considered by many and could also explain previously unexplained trips in fossil plants, chemical plants, and other process facilities. To prevent events like this from happening, it will require developing appropriate design criteria, appropriate policies and procedures, and most of all the need to have control system domain expertise as part of the cyber team.  What is also interesting about this event is that none of the existing cyber monitoring would have detected the event. Additionally, certain IT practices such as automatic patch management could CAUSE an event like this given the “right” conditions and plant design. As a result of the wide-ranging (non-nuclear) implications of this recent event, we will dedicate a session at the August Control System Cyber Security Conference in Chicago to this event.

One other interesting aspect of nuclear plant cyber security is the gap in regulations for grid reliability and continuity of nuclear power. NRC is responsible for nuclear plant safety, not continuity of nuclear power. Since nuclear power makes up about 20% of US electric power generation and each nuclear plant represents a large portion of local generation, loss of nuclear power generation can, and has, had a significant impact on grid reliability (see Northeast Outage and recent Florida outage). NRC was involved in the Browns Ferry event not because of the broadcast storm, but because the operator chose to shut the plant down. If the operator would have chosen not to shut the plant down, NRC would not have been notified, yet the grid would still have experienced the loss of more than 700 megawatts. This could easily affect grid stability and reliability. Consequently, there is a need to either develop new standards or include nuclear plant continuity of power in existing cyber security standards for grid reliability.

- Joe Weiss

——

To keep up-to-date on his latest posts, please visit the Unfettered blog at www.controlglobal.com or visit his website at www.realtimeacs.com.�

Mikko Varpiola from Codenomicon and our very own Dr. Nate Kube are taking the stage at CanSecWest 2008.   Their presentation, Fuzzing WTF? What Fuzzing Was, Is, and Never Will Be, is sure to spur additional debate on this topic!  For those of you in attendance, I hope you get the chance to catch their presentation.

As one of the foremost security and hacker conferences in Canada, it’s great to see all the major players congregate in our fine city.  One side note: it also helps us to find and interact with the best talent in the security/hacker industry, so if you meet one of our representatives, feel free to introduce yourself.

In an effort to keep everyone informed on our latest speaking engagements, here is a list of upcoming conferences and topics.  I encourage you to check out the conference links to learn more about the agendas and highlights, as there are a number of well recognized industry thought-leaders scheduled to appear at each event.

Event:  CanSecWest 2008 (http://www.cansecwest.com)
Date:  March 26 to March 28, 2008
Location: Marriott Renaissance Harbourside, Vancouver, BC

Topic:  Fuzzing WTF: What Fuzzing Was, Is and Never Will Be
Presenter:  Dr. Nate Kube, CTO, Wurldtech Security Technologies, Inc.
Presenter:  Mikko Varpiola, Founder and Security & Robustness Solutions Architect, Codenomicon Ltd.
Time:   March 28, 2008; 11:30 a.m.

Event:  Yokogawa Technology Fair 2008 (http://www.yokogawa.com/us/is/usergroup/us-ykgw-conference.htm)
Date:  April 7 to April 10, 2008
Location: Hilton Americas, Houston, TX

Topic 1: Answering the Silent Threats to Automation and Process Control
Presenter:  Dr. Nate Kube, CTO, Wurldtech Security Technologies, Inc.
Time:   April 9, 2008; 1:00 p.m.

Topic 2: Defending the Industrial Network: Meeting the Challenge of Industrial Devices and Protocols
Presenter:  Dr. Nate Kube, CTO, Wurldtech Security Technologies, Inc.
Time:   April 10, 2008; 10:45 a.m. 

Event:  ISA Edmonton 2008 Exhibit & Conference (http://www.isaedmontonshow.ca)
Date:  April 9 to April 10, 2008
Location: Northlands Agricom, Edmonton, AB

Topic 1: Security Assessment: Do you know how secure you are?
Presenter:  Bryan Singer, CISM, CISSP, VP of Security Services, Wurldtech Security Technologies, Inc.
Time:   April 9, 2008; 1:00 p.m.

Topic 2: Leading the Charge: The ISA 99 Industrial Automation and Control Systems Security Standard
Presenter:  Bryan Singer, CISM, CISSP, VP of Security Services, Wurldtech Security Technologies, Inc.
Time:   April 9, 2008; 2:30 p.m.

Wurldtech confirms new speaking engagements and events on a regular basis. For an updated list of industry events where you can find the Wurldtech Team, please visit http://www.wurldtech.com/news/index.php.

I was asked by a client what the difference was between a penetration test and a security assessment.  I quickly explained that both activities have the intention of quantifying the security of your network, but do so using different approaches that yield different data and views of the overall security posture.  When asked to elaborate, I wrote up something similar for the client before deciding that it could be useful blog post.  It begins with the obligatory thought experiment.

Imagine I’m building a house for my family.  Being a security conscious guy who hates dealing with insurance companies, I’d like to know that my family and my assets are as safe as convenience and budget allows. I don’t know much about breaking into home, but thieves do, so I hire an “ethical” thief (penetration tester) and tell him to do whatever he can to break in, every way he can. Except, don’t break windows, those are expensive and we’re on a budget… yes, I know real thieves don’t care, but seriously, don’t damage the windows.  He breaks into my house in about 5 minutes and I’m suitably impressed; looks to me that this guy is the real deal when it comes to home invasions.  I ask him how he did it, and he shows me. The back window was unlocked.  A small change later (I lock the window) and, viola, that exploit doesn’t work anymore and I’m secure, time to relax.

However, my thief says he thinks he noticed something else interesting, so I tell him to go back at it and he breaks in again, in a slightly different way.  Then again.  And again.  Shocked and dismayed that locking the window didn’t make me secure, my thief details precisely how he did it… the cat door was left open so he could reach in, he dressed up like the gas man and got my trusted neighbour to let him in, etc.  Well, that’s frustrating, but all those are easy enough to fix. I implement the changes, and I keep iterating over this process until my thief can’t break in again, and while I’m exhausted and poor, at last I’m secure.

A month later someone climbs through an open fresh air vent in my roof that my thief didn’t notice and cleans me out.  I have a rather bad taste in my mouth now because I paid a lot of money to that thief and I’m still arguing with my insurance company.  Now what?

Well, instead of looking at the house from the outside in, let’s look inside-out.  I study the assets I want to protect and the routes necessary to provide access.  I hire a locksmith to tell me all the pros and cons of household perimeter security technology, throw in an alarm system for some defense-in-depth and research the strengths and weaknesses of everything.  During analysis of the data, I start to see patterns in (like all my locks are manual) and identify risks (someone may forget to engage a lock prior to leaving the house).  I can implement solutions that mitigate entire classes of vulnerabilities (like defining a policy that every door and window must be re-locked immediately after access is complete, or install automatic timed locks) rather than trying to micro-manage individual vectors. If the lock on my door just doesn’t work the way it should (for example, if you freeze it to -10 degrees and hit it with a hammer the metal shatters, but its specified for -20 degree operation) I can go to my lock vendor and complain that I’m using his product as a critical part of my house but it is insecure and doesn’t meet specification, without “crying wolf” and asking my vendors to chase every little flaw in their product regardless of its effect on me.  Finally, I can choose the amount of risk I’m willing to bear as I’m able to understand the system-wide implications of any particular security vulnerability or set of vulnerabilities in the context of how they impact an asset of interest.

So, when performing a security assessment of a plant and my client wonders why I didn’t spend the two days I had to do my testing trying to pull off an exploit as opposed to merely collecting enough data to show that, with high confidence, a vulnerability exists, I try to paint them the picture above and show that while they have to trust my skills to a certain extent, I can generate a far richer data set than if I simply attempted a penetration test. Later, if the client deems some assets are so critical to the security, safety and/or continuity of operations that they must know the exploits that I personally can pull off against them then at least my efforts are optimized and I won’t miss the forest while concentrating on a couple of trees.

All this being said, penetration tests are not worthless - quite the opposite.  Pen-tests and security assessments are complimentary and related, but different, activities.  A security assessment is necessary for a holistic understanding of system security through the composition of a metrics-based security model.  A penetration test is an excellent tool to validate the security model generated during the assessment. Should a pen-tester find a vector not predicted by the model, you are able to analyze why the model failed to predict it.  Conversely, if your pen tester fails to find a vector the model predicts as trivial you can ask why that vulnerability wasn’t exercised.  Since such a model is based on the collection of facts about individual components, you can update the data on a single node, and then re-run the analysis to see how the change effects overall system security.  However, without having first completed a security assessment, the value of a penetration test is much less than it could be.

By all means, if you want a pen test against your plant without an assessment first, give us a call and I’ll gladly oblige, but please don’t ask me afterwards if you’re secure.  All I can tell you is that I was able to break in, or not.  I’ll leave it as an exercise of the reader to ponder the ROI of that contract.