Proof that ISO 27001 is a "Point-In-Time" Assurance

This week I cross posted "So Many Security Standards, Audits, and Certifications. Which One is Right?!" to Infosec Island , and was confronted in the comments section by someone claiming that ISO 27001 is not a Point-In-Time assurance, but is in fact, an ongoing, and even real-time assurance.  I knew this was not right because of my experience with ISO as a lead auditor for a large Japanese company, so I made my case, and the commenter backed down.


To remove any doubt, I have included in this post the language that you will find in every ISO 27001 Certificate, and excerpts from a corresponding Certification Agreement.  I have also included the language from the auditor's guidance (ISO 27006) that refers to the type of testing that is conducted during ISO assessments which amounts to a verification of "design effectiveness" rather than "operational effectiveness" which is the standard for SOC2 Type 2 engagements.


                         

ISO CERTIFICATE:



CERTIFICATION AGREEMENT:






































ISO 27006 ANNEX D - Guidance for review of implemented ISO/IEC 27001:2005 controls


SOC 2, Type 2 reports give assurance about operational effectiveness which requires a much higher level of testing.  In the examples given above, a SOC auditor would be required to confirm, through representative sampling, that:
  1. the door had remained locked over the period of time selected by observing historical access logs.
  2. people signed confidentiality agreements consistently over the period of time selected
  3. independent re-performance of how the asset register was created, and verification that all assets were captured in the register
  4. system settings are adequate and have remained in place over the period of time selected
I am more sure than ever that combining multiple compliance requirements under a SOC 2 engagement is the right approach for creating audit efficiencies and providing higher levels of assurance to stakeholders.

28 comments:

  1. Jon this is a very thought provoking posting,

    I would argue there is not one framework that by itself is capable of doing the job...there is no silver bullet. The actual job goes well beyond measuring the effectiveness of a control during the assertion period or checking a specific box.

    A well rounded program among other things must have a clear and concise set of: risk management and identification processes, critical assets definition, proper and effective access controls to those assets, effective monitoring processes (eyes and ears), and an excellent incident response program and communication plan. All confirmed and validated by systematic evidence. BTW, system is here is not a computer system but a system - a regularly interacting or interdependent group of items forming a unified whole; Merriam-Webster.

    As much as the audit community would like to tell the business what to do, what control should be implemented and what framework to use...effectiveness of controls...is a culture thing.

    I had the advantage of "cutting my teeth" in the pharmaceutical industry; when it comes to controls, I consider pharmaceutical at the same level of the airline, petroleum, diamond, nuclear and space industry. The assets at play...are extremely critical; and, for lack of better terms in these industries controls are part of life and the culture...second nature...in their DNA.

    For environments where controls are not second nature, you will never find a better framework or will make them "compliant" by checking a box in a framework. It will always be point in time, for it is seen as a regulatory thing that is part of the cost of doing business...not part of an integrated program to better manage critical assets.

    But back to your posting, using your examples, we don't need to concentrate on ISO27001. The statement apply to all regulatory schemes. I don't have an industry study on this but I am fairly certain you will not find an audit firm willing take the liability to attest to the effectiveness of a set control for 100% of the time period. Remember all the audit report really says is that based on the samples provided by management, the controls appeared to be operating during the time period.

    I imagine the only way an audit firm may agree to provide 100% attestation could be if they are on site for the entire period with an army, having access to implement and dictate all the program elements I mentioned above and doing 100% sampling on ALL parts related to the attestation. Forget the negative operational impact and the economics of such an endeavor...it would be plainly impossible to be 100% sure.

    As always a pleasure, very thought provoking exercise. I hope our other colleagues take the time to share their views.

    Sincerely,


    Kelvin Arcelay
    CISM, CISSP, CRISC, HISP, IRCA ISMS LI/LA, PMP, QSA, SSGB
    Author & Expert Reviewer for The Cloud Security Alliance, ISACA, Pearson and QUE Publishing

    ReplyDelete
  2. To paraphrase John DiMaria, all audits and examinations are a snap shot of a “Point in time.”

    What's interesting is that ISO, NIST (FedRAMP), and CSA are trying to provide real-time verification measurements for cloud services using SCAP and XBRL. Perhaps we can can never get away from point in time validation, but the smaller the window that we have for measurement to provide assurance with respect to SLAs and controls, the closer that we will be to continuous and real-time validation.

    I believe that your first post was closer to getting many to common ground on this topic: http://www.infosecisland.com/blogview/19296-Standards-Audits-and-Certifications-Which-One-is-Right.html#.Tw3oUS0Dg1k.linkedin

    Thanks,

    Phil Agcaoili

    Here's more on SCAP and XBRL:
    http://scap.nist.gov/
    http://www.xbrl.org/

    ReplyDelete
  3. Thank you Kelvin and Phil for taking time to comment on my post. I believe our collaboration can help revolutionize the risk assurance industry. I agree with most of the things both of you said, and have carefully, and I hope respectfully, responded below where I do not.

    1. There is not one framework that by itself is capable of doing the job. There is no silver bullet.
    --Jon This is why I recommend the pairing of the SOC2 audit methodology and reporting with other subject matter such as ISO 27001, NIST, or Cloud Computing Alliance (CSA) Control Matrix.

    2. The job goes beyond checking a specific box...effectiveness of controls...is a cultural thing.
    --Jon Agreed.

    3. For environments where controls are not second nature, you will never make them "compliant" by checking a box in a framework… it is seen as a regulatory thing, part of the cost of doing business.

    --Jon I partly agree. For the type of organization you describe, it definitely is a mindset thing at first where audits are looked at as an expensive waste of time. When you test over a period of time, however, and do it right, the culture is gradually forced to change. They realize that they cannot just sweep things under the rug before the auditors get there because it is all exposed in the audit process. I have experience with this at company after company.

    4. The statement applies to all regulatory schemes, not just ISO 27001. You will not find an audit firm willing to take the liability to attest to the effectiveness of a set control for 100% of the time period.
    --Jon I disagree. Please see the language of a SOC 2 opinion below. It is not absolute, but it is reasonable assurance covering 100% of the time period selected.

    In our opinion, in all material respects, based on the description criteria identified in XYZ Service Organization’s assertion and the applicable trust services :
    a. the description fairly presents XYZ Service Organization’s [type or name of] system and related privacy practices that were designed and implemented throughout the period [date] to [date].
    b. the controls stated in the description were suitably designed to provide reasonable assurance that the applicable trust services criteria would be met if the controls operated effectively throughout the period [date] to [date].
    c. the controls we tested, which were those necessary to provide reasonable assurance that the applicable trust services criteria were met, operated effectively throughout the period [date] to [date].
    d. XYZ Service Organization complied with the commitments in its statement of privacy practices throughout the period [date] to [date].

    5. Remember all the audit report really says is that based on the samples provided by management, the controls appeared to be operating during the time period.
    --Jon I disagree. See the report language above. Management does not provide samples in SOC 2 engagements, auditors access them based on completely open access provided to auditors as a prerequisite to accepting the engagement.

    6. The only way an audit firm may agree to provide 100% attestation is if they are on site for the entire period, having access to implement and dictate all the program elements and doing 100% sampling on ALL parts related to the attestation.
    --Jon I agree that trying to achieve the utopia of absolute assurance is impossible. SOC2 reports provide provides reasonable, but not absolute assurance over the period of time selected. That is much, much better than point in time assurance that is the best ISO 27001, PCI, and others can offer.

    ReplyDelete
  4. 7. All audits and examinations are a snap shot of a “Point in time.”
    --Jon I disagree for the reasons already discussed.

    8. ISO, NIST (FedRAMP), and CSA are trying to provide real-time verification measurements for cloud services using SCAP and XBRL... the closer we will be to continuous and real-time validation.
    --Jon I think it is great that the security world is moving in the direction of a continuous auditing model. Perhaps someday, an opinion letter will be as simple as the push of a button after all security parameters have been verified. Unfortunately, many processes today are still manual, and cannot be measured in the same way automated controls can be.

    Thank you again for your comments. We are sharpening each other much like iron sharpens iron.

    ReplyDelete
  5. After you have performed an SOC 2 examination, you'll understand the errors in your premise. Godspeed to any company that takes your advice to combine SOC 2 and ISO 27002 controls.

    ReplyDelete
  6. Thank you Hedge Hog. I appreciate your well wishes and your blessing.

    ReplyDelete
  7. Jon,

    I appreciate your well considered opinion and largely agree. Any audit is a point in time exercise - the SOC 2 time-frame is just a longer point in time. However, that is only 1/2 the equation. The "standard" you are providing assurance to is the other -- and I believe ISO-27001/2 is a true standard where SSAE is not. So while I may get less point in time assurance with 27001 the quality of the assurance is higher.

    Independent of the standard/methodology used -- anyone interested in validating the security posture of a third party should look for evidence that the ISMS and controls are fully operational beyond the third party certificate/report.

    Enjoyed the blog,

    John Verry
    www.pivotpointsecurity.com

    ReplyDelete
  8. Thanks for your comment John. I agree that the Trust Services Principles and Criteria (TSPC) standard that a SOC2 report is based on is not as good as the ISO standard. That is why I recommend pairing SOC 2 audit methodology with the ISO standard, or whichever standard the service organization prefers such as the CSA STAR standard. Please see my post titled "So Many Security Standards, Audits, and Certifications..." for a more in depth analysis of this approach.

    ReplyDelete
    Replies
    1. Jon,

      I agree that in a perfect world the comprehensive approach you outline would be ideal. Just the same way that building your ISMS around COBIT and harmonizing it with ISO27001 and ITIL would be -- but the number of folks who have the resources (and expertise) to do either is really limited. It gets further complicated by the confusion caused by the number of ambiguous/overlapping laws/guidelines that may be relevant (e.g., http://www.pivotpointsecurity.com/risky-business/electrical-utilities-information-security-blackout ).

      Not sure if I should be bummed out or grateful for job security. :>)

      Where it really gets interesting is if you only have the resources for one form of third party assurance -- which one?

      John

      Delete
  9. Six Sigma Certification is another certification that, bit by bit, rises to fame.

    This course is very important since business is what runs within an organization. Thanks a lot for sharing!

    ISO 27001

    ReplyDelete
  10. Hi,

    Just read http://riskassuranceguy.blogspot.com/2012/01/so-many-security-standards-audits-and.html and now got here.

    A couple of (probably noob ;) questions.
    As i understand from the other article a CPA and form the opinion that financially things are and have been in order.
    ISO27001 says at the point of audit (in our company every 6 months) there was a system in place managing the risks in a way designed by the customer.
    But how can a CPA assert that something was in place all that time, is that because the financial system is organised in a way that it is possible or because that have to look at all the data to get to that conclusion?

    Thanks!

    ReplyDelete
  11. Franc, Thank you very much for your comment. CPAs are branching off into risks other than financially related ones now. With the replacement of SAS70 with SOC reports, The AICPA has introduced a report called the SOC2 report that addresses risks similar to ISO 27001.

    The reason I said that SOC2 and ISO 27001 will work well together is that CPAs can apply their audit methodology that includes "operational effectiveness" or "period-of-time" testing methods to security risks, and stakeholders will receive a better assurance.

    The standard that SOC2 is based on, however, is not as good as ISO 27002, therefore, combining the two is a good approach for those wanting the best standard, as well as the best assurance.

    As to how operational effectiveness testing provides reasonable assurance that the controls were operating effectively over the period of time that the SOC2 report covers, this is accomplished through sampling as well as other methods depending on the conrol. For example, if you want to confirm that an organization followed their change management policy over the period of time selected, you would first obtain a system generated list of changes from the application or database in question. You would then select a sample of changes spanning the period of time covered by the report, and track backwards to the authorization of that change.

    Many auditors will not do a thorough job of testing this control, and will select their change from a list of Change Request Forms, and then look for an authorization. That approach is inadequate because it does not account for the changes made without completion of a Change Request Form.

    Does that help?

    ReplyDelete
  12. ISO 27001 Focuses on all business processes and business assets Focuses on reducing the risks for information that is valuable for the organization Information may or may not be related to information technology, may or may not be in a digital form.

    ISO 27001

    ReplyDelete

  13. ISO 27001:2005 standards by improvement activities specified within the management system

    ISO 27001 is relevant for any organization, irrespective of its large or small size or location in any part of the world. This standard is suitable for almost all the organizations, from government agencies to commercial enterprises and even for not for profit organizations. Information Security Management System (ISMS) is primarily suitable in the field where protection of information is vital, e.g. finance, health, public and IT sectors.

    ReplyDelete
  14. I appreciated information provided by you on Information security management system. I have found similar information and resources for ISO 27001 documentation, I fill its been wonderful way to train employees for ISMS system.

    ReplyDelete
  15. Jon - I found this whole post exceptionally useful. I am having this very specific debate (SOC 2 or an ISO cert).. I am leaning as I think you are here to having a SOC 2 conducted and working with the auditors to add controls to bring the completeness of coverage to = to ISO coverage. I have also heard AICPA is looking to do more comparative work of the two (I would love to see an official mapping of the delta's).

    We already have a SOC 1 Type II conducted as we handle some financial information for our clients. I can't see us every getting away from that and there is in my estimation an almost 30-40% overlap of SOC 1 controls with what would be SOC 2 controls.

    If anyone has ever done both a SOC 2 Type II and an ISO 27000 I would love to hear their comments on the depth of each.

    ReplyDelete
    Replies
    1. Thanks. By any chance are you with M/H? I would love to help you win that debate. Reach me at jlong@compliancepoint.cmo (mispelling intentional) sometime. Perhaps I can help you with that ISO 27K to SOC2 mapping.

      Delete
  16. Nice post, I bookmark your blog because I found very good information on your blog, Thanks for sharing more information
    Quality Services & Training Pvt.Ltd.

    ReplyDelete
  17. What do you think of the new CSA STAR certification? Does it test the "design effectiveness" of the CCM control matrix, rather than "operational effectiveness"?

    ReplyDelete

  18. Hey,Shocking; this presentation is based on a complete misunderstanding. The presenter does not talk about the ISMS, he explains only about the normative Annex A of the standard. Not one word about the requirements. Obviously the presenter does not understand the difference between ISO/IEC 27002 and ISO/IEC 27001.No word about risk management requirements, treatment options, the statement of applicability. Sorry to say, but please read the standard before making your professional sales material.Thanks to all!!!
    >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
    iso 13485 quality system

    ReplyDelete
  19. I realize I'm responding to a very old post, but I disagree with Jon's assessment. We do have both ISO27001 and SOC2 audits and I believe any experienced auditor would agree that the ability of ISO27001 to measure operational effectiveness depends on the (in)ability (or willingness) of the auditor to measure it - not on a failing of the standard itself. A properly trained ISO27001 auditor will measure all of the things that Jon mentioned. To use his example, my organization has been ISO certified for the past 5 years and every year during the annual ISO audit the auditors are provided access to ALL records, not just the the one's we choose to provide. The auditor or auditor(s) are onsite for up to 5 days and during that time they confirm (among other things):

    1. the door had remained locked over the period of time selected by observing historical access logs.
    2. people signed confidentiality agreements consistently over the period of time selected
    3. independent re-performance of how the asset register was created, and verification that all assets were captured in the register
    4. system settings are adequate and have remained in place over the period of time selected

    I'm looking at both audit reports (SOC2 and ISO27001) right now, and although the SOC2 report goes into more detail, the number and types of records examined are virtually the same. The effectiveness of any assessment or audit obviously depends on having a competent auditor. Unfortunately they are not all created equal and if your ISO27001 (or SOC2) auditor is not adequately measuring operational effectiveness then you should demand a different one.

    ReplyDelete
    Replies
    1. You should support your assertion by posting excerpts from the fine print in your certificate and certification agreement. In the end, it is not about what assessors validate, but what they are taking responsibility for. I am sure that you will find that the fine print disclaims responsibility per my analysis.

      Delete
  20. Excellent ISO certification Process and also provide ISO training ,best customer satisifaction Quality Services in singapore Qscert

    ReplyDelete
  21. This comment has been removed by the author.

    ReplyDelete
  22. Hmmm..I agree with all your thoughts! ISO certified companies like PAPTI are really of high service and product quality!

    ReplyDelete
  23. Iso certification in singapore, iso certification bodies in singapore, iso 14001:2004 certification, iso 9001:2008 certification, ohsas 18001:2007 certification providers, iso management in singapore Qscert

    ReplyDelete