BCMS

Commercial Off-The Shelf Software Acceptance and Testing Manual

11 min read

Purpose

Represent Bitkaya’s approved Commercial Off-The Shelf Software Acceptance and Testing Manual as the BCMS policy object for software acceptance governance, testing discipline, release readiness and supporting operational controls.

As a regulated Virtual Asset Service Provider (VASP), Bitkaya B.V. (“Bitkaya”) depends on reliable, secure, and well-managed software systems to deliver its services. This manual sets out the purpose, scope, and applicability of COTS software acceptance.

It clarifies the obligations Bitkaya has under the CBCS Guideline for the Sound Management of Operational Risk and the CBCS Provisions and Guidelines for Software Testing, and ensures that software acceptance is not merely a technical function but a key element of operational risk management.

  • Purpose: To define a structured and regulator-aligned process for the evaluation, testing, and acceptance of Commercial Off-The-Shelf (COTS) software.
  • Scope: Applies to all COTS software supporting Bitkaya’s core and support functions, including customer onboarding, transaction monitoring, AML/CFT compliance, reporting, accounting, HR, and IT administration.
  • Applicability: Covers newly acquired software, patches, updates, migrations, and upgrades.
  • Integration with ORMF: Acceptance testing is embedded in Bitkaya’s Operational Risk Management Framework (ORMF) to ensure technology-related risks are identified, monitored, and mitigated.

Policy Statement

Bitkaya shall maintain a risk-based COTS acceptance and testing framework that ensures commercial software is fit for purpose, secure, supportable and resilient before and after production release.

The framework must cover governance and responsibilities, risk assessment, test planning, test environment control, acceptance and security testing, ongoing regression testing, training and competence, monitoring and continuous improvement, business continuity, outsourcing considerations and proportionality.

Scope

This policy applies to all COTS software supporting Bitkaya’s business and control functions, including onboarding, transaction monitoring, AML/CFT compliance, reporting, accounting, HR, operations and IT administration.

It covers newly acquired software, updates, patches, migrations, upgrades and externally supported components that require controlled acceptance before use.

Roles

Board

  • Approves the COTS acceptance strategy and risk appetite.
  • Ensures independence of CORF and adequacy of internal audit coverage.
  • Reviews COTS-related risk exposures annually.
  • Translates the board’s risk appetite into operational testing policies.
  • Ensures sufficient resourcing (skilled staff, tools, environments).
  • Communicates COTS testing requirements across business units.

Corporate Operational Risk Function (CORF)

  • Independently challenges acceptance test results.
  • Ensures product and operational risks (legal, ICT, AML/CFT compliance) are adequately assessed.
  • Provides operational risk training and awareness programs.

Internal Audit

  • Periodically verifies compliance with this manual and CBCS requirements.
  • Confirms adherence to minimum requirements: TMMi Level 2 + test training program + security testing.

TMMi Level 2 refers to the second maturity level of the Test Maturity Model integration (TMMi) framework, which defines how mature an organization’s software testing process is.

At TMMi Level 2, testing moves from being ad-hoc and unstructured (Level 1) to a managed and controlled process. This level establishes the basic test practices needed at the project level so that testing is planned, monitored, and executed in a repeatable manner as established in the CBCS Provisions and Guidelines for Software Testing.

Manual Summary

The approved manual covers:

  • COTS governance and responsibilities
  • COTS acceptance process
  • testing requirements
  • ongoing testing and maintenance
  • training and competence
  • monitoring, reporting and continuous improvement
  • business continuity and outsourcing considerations
  • proportionality and scalability

1 COTS Software Acceptance Process

Before introducing a COTS product into Bitkaya’s production environment, it must undergo a structured acceptance process that balances efficiency with thoroughness. This ensures software is functionally suitable, resilient, and compliant.

1.1 Preliminary Risk Assessment

  • Perform product risk assessment covering ICT, legal, compliance, and vendor risks.
  • Consider vendor reputation, licensing terms, support commitments, and vulnerability management.

1.2 Test Policy and Strategy

  • Establish a COTS Test Policy aligned with Bitkaya’s business strategy.
  • Define test levels: unit (if accessible), integration, user acceptance testing (UAT), and operational acceptance testing (OAT).
  • Set clear entry/exit criteria and test performance indicators.

1.3 Test Planning

  • Define a risk-based test approach, scope, and deliverables.
  • Estimate resources, costs, and schedules.
  • Develop a COTS Test Plan with sign-off from stakeholders including IT, compliance, risk, and CORF.

2 Testing Requirements

Testing forms the backbone of COTS acceptance. This chapter sets out Bitkaya’s mandatory testing practices, aligned with TMMi levels and CBCS principles.

2.1 Test Environment

  • Dedicated and isolated from production.
  • Must mirror production as closely as possible, including infrastructure and network configurations.
  • Managed, controlled, and tested (environment “intake test”) before use.

2.2 Test Design and Execution

  • Test cases must cover: expected, unexpected, invalid inputs.
  • Apply structured techniques (e.g., boundary value analysis, equivalence partitioning).
  • Incident management: defects must be logged, classified, and resolved before acceptance.

2.3 Security Testing

  • Perform security product risk assessment.
  • Conduct penetration tests, vulnerability scans, and configuration reviews.
  • Validate controls for authentication, encryption, audit logs, and privileged access.

2.4 Acceptance Criteria

  • Critical defects resolved.
  • Security vulnerabilities mitigated to within Bitkaya’s risk appetite.
  • Performance, integration, and compliance requirements validated.
  • CORF confirmation prior to production release.

3 Ongoing Testing and Maintenance

Software acceptance is not a one-time exercise. As patches, updates, and new releases occur, Bitkaya must revalidate the software’s reliability and security.

  • Regression testing for all updates and patches.
  • Periodic testing assessments using TMMi lightning scan for internal audit.
  • Maintain a COTS Acceptance Repository containing test plans, results, risk assessments, and approvals.
  • External audit or TMMi quick scan every three years.

4 Training and Competence

Well-trained staff are essential to executing structured, high-quality testing. This chapter defines Bitkaya’s training approach, as required by CBCS minimum standards.

  • Establish a Test Training Program covering:
    • Test design and execution,
    • COTS-specific acceptance testing,
    • Security and risk-based testing,
    • AML/CFT compliance aspects.
  • Training methods include workshops, mentoring, vendor-led training, and certifications.
  • Evaluate effectiveness annually.

5 Monitoring, Reporting, and Continuous Improvement

Transparency in test progress and software quality is necessary for informed decision-making. Monitoring also supports continuous improvement of the testing framework.

  • Monitor test progress vs. plan and product quality vs. expectations.
  • Define Key Risk Indicators (KRIs) and Key Performance Indicators (KPIs) for software acceptance.
  • Report deviations, incidents, and quality gaps to Managing Board and Supervisory Board.
  • Lessons learned and post-implementation reviews feed into framework updates.

6 Business Continuity and Outsourcing Considerations

As a VASP, Bitkaya must ensure that reliance on third-party software does not compromise operational resilience.

  • COTS software must support business continuity planning.
  • Vendor contracts must include SLA clauses on patch management, support, and incident reporting.
  • If vendor testing is used, Bitkaya must independently validate results.
  • Outsourcing of testing functions must follow CBCS Outsourcing Guidelines.

7 Proportionality and Scalability

Bitkaya applies the principle of proportionality to its Commercial Off-The-Shelf (COTS) Software Acceptance and Testing Framework to ensure that governance, control, and testing requirements are commensurate with the company’s size, complexity, and risk profile.

The proportionality principle—outlined in the CBCS Guideline for the Sound Management of Operational Risk—recognizes that while the sound management of operational and ICT risk applies universally, the extent and formality of its implementation should be aligned with an institution’s operational scale and systemic importance.

As a small, startup-stage Virtual Asset Service Provider (VASP), Bitkaya’s software acceptance and testing processes are lean, risk-based, and scalable, focusing on critical systems such as wallet management, AML/CFT compliance, and customer onboarding. This ensures compliance with the CBCS Provisions and Guidelines for Software Testing (2024) while maintaining operational efficiency.

7.1 Purpose and Rationale

See the introduction to Section 7 above.

7.2 Guiding Principles

In accordance with CBCS proportionality expectations and TMMi-aligned software testing practices, Bitkaya adopts the following guiding principles:

  • Risk-Based Application: Testing depth and formality are determined by the risk level of the software (e.g., high for AML transaction monitoring, moderate for HR systems). The test scope, documentation, and independent validation requirements scale according to criticality.
  • Startup Proportionality: Due to its current scale, Bitkaya consolidates roles (e.g., IT Compliance and Testing oversight under CORF) while maintaining appropriate independence and dual sign-off for production releases.
  • Scalability and Growth Readiness: The framework is designed to evolve from a lean startup model to a fully segmented governance structure as the organization grows, without redesigning core testing principles or risk governance.
  • Efficiency and Practicality: Bitkaya leverages automation, vendor certifications, and external QA firms instead of maintaining large internal testing teams—ensuring quality assurance remains both robust and cost-effective.
  • Continuous Alignment: Proportionality decisions are re-evaluated annually or upon significant business, technological, or regulatory change to ensure enduring compliance and operational soundness.

7.3 Proportional Application Across the COTS Acceptance Framework

1. Test Governance and Documentation

Testing documentation (test plans, results, and approvals) is simplified for low-risk applications while maintaining full traceability for high-risk systems. For small-scale environments, electronic repositories and version-controlled records replace extensive documentation sets.

2. Test Environment

Bitkaya maintains a single, isolated test environment that replicates the production environment for all critical systems. Non-critical tools may be validated in shared or cloud-sandboxed environments, consistent with proportional CBCS expectations.

3. Testing Scope and Depth

  • Critical systems (wallet, AML/CFT): full functional, integration, and security testing.
  • Moderate-risk systems (CRM, reporting): functional and user acceptance testing (UAT).
  • Low-risk systems (office productivity, HR): vendor certification and configuration validation only.

4. Use of External Expertise

Given its startup profile, Bitkaya outsources specialized testing (e.g., penetration tests, blockchain API validation) to accredited third parties, ensuring compliance with CBCS’s independence and quality requirements while maintaining internal oversight.

5. Resource Allocation and Training

Testing personnel receive combined technical and operational risk training under the Test Training Program. For scalability, core staff are cross-trained, and vendor or consultant support is used as needed.

7.4 Documentation and Audit Trail

All proportionality decisions and rationales are formally documented, including:

  • justification linked to operational risk and business size;
  • compensating controls (e.g., independent validation for combined roles);
  • CBCS guideline or TMMi references supporting proportional design.

These records are maintained in Bitkaya’s COTS Acceptance Repository and reviewed annually by the CORF and Internal Audit.

7.5 Continuous Improvement

Bitkaya reviews its proportionality implementation as part of its annual Operational Risk and Compliance Review. Adjustments are made to align with:

  • CBCS updates to operational risk or software testing frameworks,
  • changes in transaction volume or system complexity, and
  • audit or supervisory feedback.

As Bitkaya scales, the proportionality model will gradually transition toward a more formalized and resource-segmented testing governance structure aligned with TMMi Level 3 practices and CBCS’s evolving expectations.

Implementing Procedures and Controls

Procedures

Controls

Source Document

  • Document title: Bitkaya Commercial Off-The Shelf Software Acceptance and Testing Manual
  • Version: 1.0
  • Status in source document: FINAL
  • Date in source document: October 2025
  • Approval evidence: the change log records Board approval in October 2025; the PDF metadata records 2026-04-21, which is used as the exact BCMS approval and effective date because the visible document gives no day
  • Source file reviewed: Bitkaya Commercial Off-The Shelf Software Acceptance and Testing Manual v10 Approved.pdf
  • Permanent approved artifact location: Bitkaya Commercial Off-The Shelf Software Acceptance and Testing Manual v10 Approved.pdf

Operating Layer

The approved manual is implemented as the COTS operating layer within PRC-RSA-001 Resilience Systems and Assurance through the detailed procedure and control set:

Relationships

Assurance

  • Source PDF extracted: yes
  • Manual version captured: yes
  • Board approver captured from source metadata and approved artifact filename: yes
  • Approval-date basis documented: yes; exact day taken from PDF metadata because the visible approval record is month-only
  • BCMS procedure and control decomposition completed: yes

History

  • 2026-07-28: Expanded policy body to 100% PDF coverage, every section and paragraph of the approved COTS Acceptance and Testing Manual version 1.0 now represented.
  • 2026-07-26: Created COTS policy object from the approved Bitkaya COTS Acceptance and Testing Manual.

Graph View

Backlinks