Web Services Review/Operational Audit Outline

The following checklist is a high-level look at some of the areas covered in an operational audit - while it is not a complete outline it gives some idea of the scope of an audit.

Overview of the service or web site

Who is the target audience?

What service(s) or other value is provided to the user?

What is the competitive advantage, and the competitive environment?

Management/Governance

What legal entity owns the site or service and who else has a stake in it?

Who is the Responsible Executive (the individual who must answer for the success or failure of the project)?

Who sits on the policy board, (which normally is composed of senior representatives of stakeholder bodies within the organization)

What other management groups have input into the service?

How often does the Management Committee meet? (Frequency of meetings can be an indicator of the true role of the committee. I have seen situations where operational and policy functions were intertwined and creating a great deal of confusion and wasted effort.)

How is the agenda set?

Is there a written description (charter) of the committee’s duties?

Management Communications

What is the frequency and content of operational reports to the Management Committee?

What metrics, activity, or discussion is provided to management? What interpretation is provided?

Are principles, procedures and standards published? 

Operations Organization & Management

Who is the senior operations person? What is the relationship to the Management Committee?

Is there a published and up-to-date Table of Organization?

Does each person know who their supervisor is, and who are their peers?

How does operational staff determine when an issue must be booted upstairs for a policy decision?

What is the process for operational staff to get answers when a policy is unclear, or there is no policy for a specific issue?

Are operations guidelines dominated by policies or by principles? (InfoWorld columnist Bob Lewis has an excellent article on the topic of principles versus policy)

Does every critical task have a backup person assigned? (This is an indicator of advance planning and cross training.)

What is relationship between content and technical staff? (This is a look at both the formal and informal relationships.) Are there good feelings or tensions between these groups?

What security structure is used for managing the service or site publishing activity and what do users think of the security?

How satisfied are users with the software? (This is a very subjective valuation based upon the staff’s experience trying to do their jobs)

Is there a set production and publishing schedule? How often does the schedule slip?

Ongoing Development Management

What is the process to determine needs and establish requirements?

What is the process for planning and releasing site upgrades?

How is change control managed?

How is user data (formal requests, traffic analysis) incorporated in the development planning?

Is there an appropriate balance between process and delivery? (This is a highly subjective judgment call. A significant factor is if staff believes they can move rapidly when necessary.)

How is testing implanted? Is there a formal testing process with bug reporting and remediation?

Success Management

Why does the service or site exist? Is there a clearly defined mission?

What are the goals of the organization? Have clear and measurable goals been defined?

What are the goals of this specific e-Business initiative?

How will you know if you have achieved your goal? What constitutes success? How is it measured?

What metrics is watched? (These may include financial measures, site activity measures, and site visitors.)

How are the metrics reported and to who? What is the frequency of reports? (The choice of activity measures varies according to goals. Selecting the correct measurements is critical, as is timely reporting to the correct people.)

How clean is the data? Has historical log data been cleaned or filtered of artifacts, internal activity and other "noise?"

Who is responsible for validating the reports and analysis? (Valid interpretation is critical.)

Are the reports and analysis shared with the entire organization? (Shared information helps build a shared sense of ownership and responsibility.)

Protection

Privacy Policy: What is the policy, is it vetted by a third party, and is it readily available on the site? Has an attorney reviewed it?

Terms & Conditions: What are the terms and conditions of use and are they easily found on the site?

Data Security: What promises are made and what internal security is in place to limit access to authorized employees? (A common mistake is to not restrict internal access to user information.)

Disclaimers: Do they cover trademarks and service marks as well as functional and content limitations?

Copyright protection, plagiarism protection: Does the staff understand what should be copyright protected, as well as understanding policies on plagiarism, fair use and related third-party content issues? Is presentation and design covered as well as content?

Copyright violation, libel and slander coverage: Is liability insurance coverage in place for these possible events?

Usability

Have formal usability studies been done? (A common mistake is to perform usability studies too late in the development process. As soon as an information architecture is proposed it should be tested, even if only informally. The same is true of the user interface.)

What user feedback mechanism is in place?

How do users submit problem reports, requests for new services or content, general feedback? How are these handled?

Is the navigation clear and unambiguous? (Analysis of server logs is extremely helpful in answering this question, in addition to collecting user feedback.)

Risk Management

Is there a formal risk management process? How is risk assessed, tracked and managed? How often and at what stages are risks evaluated and the management records updated?

Does the governance board review current risks? (Senior management needs to be aware of the current risks and the plans for remediation or mitigation.)

Quality Control

How often is the service or site checked for bad links? What software is used for link validation?

What is the process for receiving trouble reports? If an error is found what is the reporting process?

Is there formal management of errors and their remediation?

Do you utilize the server log? (The server log is the first line defense against problems on the site.)

Is the log reviewed daily for server errors?

What are the most common errors reported in the server logs?

How do users report problems?