When I See a Can in the Road All I Want to do is Smash It

Right now, the "can" I really want to smash, is the notion that SSAE16 provides assurance about Security, Availability, Processing Integrity, Confidentiality, and Privacy. The AICPA has been clear that SSAE 16 is not designed to provide assurance about these things from the beginning, but somehow the message was not understood by the market last year. For anyone that missed the video by Barry Melancon, President of the AICPA, please take a couple minutes to watch.

I have my Tweet Deck configured to alert me when anyone tweets about SSAE16. Despite the AICPA's best efforts to educate the masses about the changes in the standards, here are few of the ways SSAE 16 has been portrayed over the past few weeks:

  • We proved our data is secure by getting SSAE 16 certification
    • Achieving this certification validates our operation at all levels
    • SSAE 16 confirms that our clients are receiving the most reliable solutions
    • The SSAE 16 report validates our success in following industry standards
    • SSAE 16 compliance provides an extra measure of confidence that our managed services provider meets and exceeds the highest industry and regulatory standards
    • Successfully completing the SSAE 16 audit symbolizes maturity, safety, and security
    • Provides tangible proof of our commitment to delivering unsurpassed value, performance and security

      All of these quotes can be found in the press releases on the internet. There's just one problem, none of the statements above are true. As Barry Melancon explains in the video I linked to above, there is no such thing as a SSAE16 certification, it is not an industry standard, and SSAE16 is not designed provide assurance about any of the things listed above.

      The news is not all bad though. There are some service organizations that understood the changes in the standards, and got it right the first time. Here are a few of their quotes:
      • By meeting the criteria set forth in the Trust Services Availability Principle, the SOC2, Type II Report confirms that Internap’s data center security and operational procedures have been reviewed and tested by an independent certified auditor and validates that controls and processes are suitably designed and operate effectively to protect and safeguard customers’ equipment and data.
      • The SOC 2 examination evaluated the design and operating effectiveness of the Cbeyond Louisville data center controls for compliance with the security and availability criteria set forth in the American Institute of Certified Public Accountants Trust Services Principles, Criteria, and Illustrations for Security, Availability, Processing Integrity, Confidentiality, and Privacy. Cbeyond Cloud Services processes, procedures and controls have been formally evaluated and tested by an independent auditing firm. This examination demonstrates that Cbeyond Cloud Services is compliant with the relevant security and availability criteria and that its customers are served and hosted in a secure, controlled facility.
      We need to collectively smash the "can" of junk assurance, that is using SSAE16 for making assertions about security, availability, processing integrity, confidentiality, and privacy by understanding the following:
      1. SOC1 (aka SSAE16) does not include any industry standards. Each organization gets to decide which controls they want to be tested on, and the service auditor performs tests to validate that only those controls are effective. SSAE16 itself is a standard, but only one that governs service auditors.
      2. SOC2 is based on the Trust Services Principles and Criteria (TSPC) which address five domains; Security, Availability, Processing Integrity, Confidentiality, and Privacy. The TSPC provide an objective common denominator of comparing service organizations. Once these standards have been met by the service organization, they can honestly say they comply with them.

      I can't wait for that satisfying feeling I will get when I will hear the CRUNCH sound of service organizations asking their service auditors for SOC2 reports rather than SOC1 (aka SSAE16) reports when they need to provide their customers assurance about their Security, Availability, Processing Integrity, Confidentiality, and Privacy. Please also see my post on combining other criteria such as the Cloud Security Alliance's Control Matrix, the Payment Card Industry's Data Security Standards, and ISO 27001 with a SOC2 engagement to reduce compliance costs.

      Cheap Risk Assurance is like My Son’s Cheap Toy Basketball Goal

      They both break easily with use.

      My 9 year son started playing basketball in a YMCA league, and instantly fell in love with the sport.  All he wants to do is shoot hoops, and last week, he decided to make a basketball goal he could hang on his wall so he could practice inside.  I found out about the homemade basketball goal when he showed me what he had made so far.  It was a wire hanger shaped roughly in the shape of a circle and tied together with a string.  He was trying to figure out how to attach that to the wall when I suggested that he buy one that he can hang from his door.  He immediately went online and started looking for one he could buy, and found one that he could purchase and pick up a local store.  It was only $15.99.

      Well you probably can guess what happened to it from my title and punchline.  After two or three days of use, the backboard cracked, and the rim came off.

      We, as professionals, understand instinctively that you get what you pay for, and observe this in every area of our lives where quality matters.  That being said, please review the wording in this request for quotation (RFQ) that I came across recently.

      Lest there be any doubt about whether other criteria might be considered, please see the following comment that was included in the Q&A attachment:

      This is what third party assurance about a service organization's internal controls has become thanks to the misuse of SAS70 reports.

      Some may say, "What does quality have to do with a report from an auditor anyway?  Who cares?  All of these reports are the same!  I just need something to give my customers that says SSAE16 on it, and they will be satisfied."

      The organization that sent out this RFQ clearly does not understand the importance or value of a quality third party risk assurance, and like my son's basketball goal, the assurance they purchased will breakPlease let me explain

      The details of an award to a different RFQ, that was not quite as blatant, were made public.  This provided me details of the outcome of this kind of mentality to analyze.

      1. The details in both RFQs made it clear that the organization was seeking assurance about security, availability, processing integrity, confidentiality, and/or privacy.  The attestation standard is clear: SSAE 16 is not designed to provide assurance regarding security, availability, processing integrity, confidentiality, or privacy, and yet the winning bid was awarded a five (5!) fiscal year contract to provide SSAE 16 reports to the organization.  I hope there is a clause in the contract that lets them get out if the report is rejected by customers because it is not applicable to what they want to know about.  Broken.  At least the store my son got his basketball goal from has a money back guarantee.  Does the CPA firm that was awarded the contract?

      2. The RFQ contained an "Unqualified Peer Review" pre-requisite (shown below), but if the CPA firm that won the five year contract undergoes a peer review in year two of this contract, they could end up getting a qualified opinion based on the SSAE 16 they provided this very organization.  Please see my post titled "Who will be held responsible...?" for more details on peer review.  Will they still be as concerned about the qualified peer review when they are the ones that caused it?  Will they even check in year two?  Broken. 
      3. The contract was awarded to a firm that no one has ever heard of, and it is easy to verify that they are extremely small with the click of a mouse.  What good is assurance from someone unless I know that people trust what they have to say about me?  Facebook is releasing their IPO next week. Did they shop around for the lowest cost CPA firm to provide them an audit of their financial statements?  No, they naturally chose a firm that they thought would inspire the most confidence with their potential investors.  Sure, the organization above got a REALLY good "Cost" which was what they were looking for, but as we learn in life, usually it's either you pay now, or you pay later.  My guess is that this organization is going to be entertaining auditors exercising their "right to audit" for about 5 years.  Broken.

        I think companies like Cbeyond have the right idea.  Please see this press release from Cbeyond. (@Cbeyondinc) I have set up an interview with Fred White, Product Management - Cloud Services at Cbeyond (@FredintheClouds) to find out more about their decision making process, and will post the video once it is available.

        I know what everyone is thinking...we are not that big, and cannot afford to have a Big 4 firm come in and do this for us.  All I have to say in response is that there is a wide gap between the Big 4 and the company that won the above mentioned award  Let's all select both the type of risk assurance we need, and the service auditor we use wisely because we know that quality matters in the long run.

        The Risk Assurance Revolution has Begun

        Yesterday, some of the most influential members in the cloud computing, information security, and risk assurance industries convened in Atlanta to discuss what can be done to take some confusion out of the market around risk assurance.  COX Communication's Phil Agcaoili (@HackSec) hosted and moderated the discussion.  I was humbled and honored to be invited to give the presentation below as a basis of kick starting the discussion.

        Please take a look at the following presentation, and you can download it hereThe Cloud Security Alliance is making this idea public, and is soliciting your feedback.  Please tweet your comments using the hash #csamtg, and your comments will appear both on this page and in the presentation wherever it goes worldwide as long as Office 2010 is installed.

        I apologize for being rude, but I would ask that everyone please be respectful in their comments, and refrain from using avatars.  If it becomes a problem, I will have to start screening the comments, and I would prefer not to do that.  Thank you in advance for your cooperation.

        "Risk" is probably another term that we should define and clarify. I stumbled across this video today that will make you pretty paranoid.

        Stuxnet: Anatomy of a Computer Virus from Patrick Clair on Vimeo.

        Here's how simple it is to create a virus and launch a DDoS attack.

        Here's Paul C. Dwyer (@PaulCDwyer) explaining mitigation strategies for a DDoS attack.

        Here's an informative interview with the founder of Kaspersky about the dangers of "Hacktivists."

        Please see this article on the fact that Anonymous deleted CBS.com. This demonstrates the power of "Hacktivists", and the need to unify against them.

        Proof that ISO 27001 is a "Point-In-Time" Assurance

        This week I cross posted "So Many Security Standards, Audits, and Certifications. Which One is Right?!" to Infosec Island , and was confronted in the comments section by someone claiming that ISO 27001 is not a Point-In-Time assurance, but is in fact, an ongoing, and even real-time assurance.  I knew this was not right because of my experience with ISO as a lead auditor for a large Japanese company, so I made my case, and the commenter backed down.

        To remove any doubt, I have included in this post the language that you will find in every ISO 27001 Certificate, and excerpts from a corresponding Certification Agreement.  I have also included the language from the auditor's guidance (ISO 27006) that refers to the type of testing that is conducted during ISO assessments which amounts to a verification of "design effectiveness" rather than "operational effectiveness" which is the standard for SOC2 Type 2 engagements.




        ISO 27006 ANNEX D - Guidance for review of implemented ISO/IEC 27001:2005 controls

        SOC 2, Type 2 reports give assurance about operational effectiveness which requires a much higher level of testing.  In the examples given above, a SOC auditor would be required to confirm, through representative sampling, that:
        1. the door had remained locked over the period of time selected by observing historical access logs.
        2. people signed confidentiality agreements consistently over the period of time selected
        3. independent re-performance of how the asset register was created, and verification that all assets were captured in the register
        4. system settings are adequate and have remained in place over the period of time selected
        I am more sure than ever that combining multiple compliance requirements under a SOC 2 engagement is the right approach for creating audit efficiencies and providing higher levels of assurance to stakeholders.

        So Many Security Standards, Audits, and Certifications. Which One is Right?!

        Last week I had a great conversation with a prominent member of the information security community (@HackSec), and he raised the following concern:  Many are confused about when to use ISO 27001 certificationPCIcertification, SOC 1 (aka SSAE16), SOC 2 & 3, NIST, and CSA STAR to give customers comfort about their security.  I explained my position that I think organizations should use SOC2 as an umbrella for all of them especially because many service organizations are required by their customers to comply with multiple standards and produce multiple reports. 

        To this, he expressed his frustration that if the information security community cannot decide which one to standardize on, how can customers be expected to know what to do? He said: "We'll stay in this nexus of confusion where every attestation is required, and yet customers still do not have confidence in our security and demand the right to audit." He went on to explain that the Trust Services Principles and Criteria (TSPC), that a SOC 2 is based on, is not detailed enough, and that CSA STAR and NIST are much more thorough standards for ensuring security.

        In our continuing dialogue, I explained the idea that they can all get along under the SOC2 umbrella.  Here's why: Accountants understand auditing, security professionals know security, and the international standards organization is just that...international.  Each has something that the other does not, and if you bring it all together, you have one heck of a team.  Let me explain a little more about each:
        Accountants understand auditing:  Accountants can trace the history of auditing back to ancient Egypt, but in its more modern form of independent CPA accounting back to 1593.  They were auditing and being held responsible for their opinions (through lawsuits) long before information security was invented.  By this I am trying to say that there is an audit process that is missing from the security assessment space.  PCI and ISO 27001 assessments provide a point in time assurance which is no assurance at all, and CSA STAR is a self-assessment at this point.  While I agree that the TSPC is a weak standard, at least with SOC2 you get a period of time assurance by taking advantage of the audit process that the attestation standard requires.  Then all you have to do is add the other standards into the covered areas and enhance the audit procedures to ensure that the controls are not only "in place", but that they "have been in place" for the period of time that is covered by the report.

        Security professionals know security:  CSA STAR, NIST, and ISO27001 are great standards, and security professionals are the only ones who can test them.  Accountants know that they cannot test security which is probably why the TSPC are so vague.  Security professionals have the right security standards, but they do not understand what assurance is, or how it is achieved.  The very fact that entire industries throw around words like "certification" and "compliant" demonstrate this.  Accountants understand that when you use words like this, the entity providing the attestation is opened up to huge liability.  Accountants are very careful to design and perform their tests to mitigate this risk, and use terms like "reasonable assurance", "in all material respects", and "in our opinion" to ensure that organizations that rely on their opinion know exactly what they can rely on.
        ISO is international:  Professionals in other countries resist "American" standards because it makes them feel less sovereign.  They hate the arrogance of it.  Right now there are no International Standards on Attestation Engagements (ISAEs) for security because ISO 27001 dominates in that space, and international accountants are not so brazen that they think they can get into the security space like the AICPA is.  So for now SOC2 is all there is for period of time attestations, and it is embraced by Canada because they invented TSPC.  That is a start, and if we add ISO 27001 under a SOC2 umbrella, we're golden with the international community.  They would get period of time coverage, and strong security controls.

        The SOC2 attestation standard is flexible enough to incorporate "additional subject matter", and all of the previously mentioned standards can be covered in the auditor's opinion as long as accountants use competent "technical specialists" to test the controls.  This has led some to argue that SOC2 is the "Silver Bullet" that satisfies all compliance and reporting requirements.  However, even if accountants use competent security professionals, there is still a problem; they cannot issue the reports that customers want such as ISO 27001 certifications, PCI Reports on Compliance (ROCs), or CSA STAR attestations because they are controlled by governing bodies that CPA firms are not registered with.  Most of the large CPA firms will never associate with those organizations because they lack influence in them, and consequently cannot control the risk it exposes them to.

        There is a way that service organizations can avoid the dilemma of having to undergo multiple audits to satisfy their customer's demands for multiple reports.  The way it works is that a CPA firm partners with an ISO certifying organization, security firm proficient in CSA STAR, or QSA (in the case of a PCI report) to jointly conduct the testing.  Because there is significant overlap in the standards, service organizations can take advantage of the testing efficiencies that result. At the completion of the engagement, the organization will receive multiple reports from a single attestation engagement.  This approach takes advantage of the best of all worlds: great audit process, the best security standards, and risk assurance for their client that is meaningful.

        "...and we secure our datacenter doors with gum."

        No service organization would actually include a control like this in their SOC1 (aka SSAE16) description of controls, but it illustrates the inherent problem with letting organizations define their own controls rather than adhere to an industry standard set of controls.  It highlights the reason why a SOC1 report is not designed to give users assurance about Security, Availability, Processing Integrity, Confidentiality, or Privacy.  SOC1 reports allow service organizations to define the controls that are tested by the service auditor.  

        This was the problem with SAS70 reports:  SAS70 reports lacked credibility because the controls contained in the report did not pass the "so what?" test.

        Discerning internal auditors understand that selecting a few controls about security or processing integrity for service auditors to test does not mean an environment is secure and will promote processing integrity.  Some have compared it to writing your own exam, and then expecting a pat on the back for passing.

        Controls that are inadequate to mitigate a particular risk (like using gum to secure a datacenter door is not adequate to prevent unauthorized physical access) usually appear in control activities that correspond to control objectives defined by management.  Today I decided, however, to map a SOC1 report to the Trust Services Principles and Criteria (TSPC) to see what controls were missing at even the control objectives level.

        I understand that SOC1s are not based on the TSPC, but what use is it to include IT related controls that, "..support the complete, accurate, and timely processing and reporting of transactions and balances." if the user only receives assurance about a limited set of control objectives?  If service organizations choose to include these controls because they support the accuracy of their customer's financial statements, they should adhere to an industry standard set of control objectives rather than selectively defining their own even in a SOC1.  This problem is compounded by the fact that service organizations hold the SOC1 report out as a standard that they comply with.  See the examples in my post on "The Failed Launch of Service Organization Control Reports."

        The first column in the matrix below shows the only two Change Management and Logical Security control objectives in the actual SOC1 I reviewed.  The second column shows the corresponding TSPC control.  The third column shows the control objectives that should be included for the report to be reliable as having adequate controls around Change Management and Logical Security per the TSPC standard.

        As you can see, many control objectives key to managing proper Logical Security and Change Management are missing from management's control descriptions, and consequently were not tested by the service auditor.  As a user of this report, I am left wondering if perhaps a SOC2 report would have better suited my needs for understanding my service organization's controls related to Security and Processing Integrity.