Category Archives: Software Testing

What are your Biggest V&V Challenges?

Verification Futures has been running since 2011 in the UK and has also run in France, Germany and India.  During that time over 30 verification managers and engineers have presented their 3 top verification challenges, resulting in close to 100 different challenges being shared with the community (see table below).

Analysing that data tells us that their biggest challenge is integrating different languages, tools and strategies.  Innovation in hardware verification has brought us a slew of verification technologies to help us verify increasingly complex designs.  However, understanding how to select the right strategy combining methods, languages and tools, and then combining the results continues to be the biggest challenge we face.

As mentioned, design complexity continues to increase with a consequent increase in verification complexity.  This is cited by 8 of the speakers making it the second biggest challenge closely followed by completeness, mixed signal and debug.  Verification managers see closing verification as a major challenge leading to potentially long tails in the project execution.  Mixed signal verification challenges have been on the increase as speakers see more analog blocks appearing in designs.

Debug continues to appear very high in the list and industry research also puts debug as taking the largest percentage of verification effort on our projects. To my amazement most debug is still performed with waveforms and “printf” (although UVM allows us to write “printf” in more sophisticated ways 🙂 )  Surely debug is ripe for a major innovation to help us all save time for our scarce verification resource.

The table below gives the summary of the hardware verification challenges presented over the past 6 years.  This year the conference includes more software testing, with tracks on both safety and security.  The challenges are coming from Qualcomm, Thales and GE Aviation and as expected there are challenges around qualifying safety-related products but also familiar ones shared between both hardware and software; namely dealing with complexity and predictable execution of verification plans.

Join Us for Verification Futures 2017

Why not join us for Verification Futures this year on 6th April 2017 to hear more about the challenges and the solutions being offered. It is free to attend and there is both physical attendance in Reading, UK or virtual attendance available.

What Are Your Biggest V&V Challenges?

Help us to discover the top challenges that the community is currently facing in performing efficient hardware and software V&V by completing a short survey.  It has just two questions and shouldn’t take more than 2 minutes to complete.  Results will be presented at VF2017.  Thank You.

Table of Challenges

Challenge Description # Challenges
Integrating Languages, Tool and Techniques 10
Complexity 8
Completeness 7
Mixed Signal 7
Debug 7
Productivity 5
Requirements Driven Verif/ISO 26262 5
Reuse 5
Resources 5
FPGA Specific 5
Scalability 4
System 4
HW/SW 3
Integration Verification 3
Power Verification 3
EDA tool Integration 3
Design for Verif 2
Demonstrating Bug Absence 2
Synthesis/Timing Constraints 2
Performance 1
Change 1
Leading Edge Technology 1
Verification Data Mgt 1
Predictability 1
Measuring Test Bench Quality 1
IO Muxing at SoC Level 1
Lack of Models for Verification 1

Is Your Website or Mobile App Losing You Money?

It is essential to understand how your customers interact with your digital products since the chances of a potential customer returning to your site or application after a negative experience are zero.  Your website or mobile application must enable customers to easily find information, purchase products or perform any number of business-critical tasks.

To ensure your customer engagements are positive T&VS offer a wide range of innovate solutions that combine traditional and modern testing approaches to ensure your customers have the best possible experience when interacting with your products.

Benefits

By engaging with T&VS you will have access to:

  • A wealth of experience testing websites and mobile apps
  • Vast knowledge of traditional and innovative new testing approaches
  • Crowdsourced testing partner to test usability in the real world
  • Automation expertise
  • Experienced Performance and Load testing capabilities

Software Testing Services

A wide range of software testing services are available, including:

  • Functional
  • Automation
  • Performance
  • Load
  • Security
  • Accessibility
  • Usability

Solutions

  • Product Launch Approval – Utilise the crowd experience and diversity to protect your brand and essential customer experience. In 72 hours (90 man days of testing will be performed) providing a product launch approval report, highlighting key concerns, giving essential confidence the product is ready to launch.
  • Conversion Rate Validation – Enable the power of the crowd to perform a fast and effective review of your digital shopping experience and purchase conversion issues.

Crowdsourced Testing with T&VS

T&VS has collaborated with BugFinders to deliver access to a global crowd of 55,000 expert testers across 145 countries, with T&VS managing the crowd experience:

  • Management of the remote workforce, with summary reports on bugs found
  • Detailed defect reporting including: process description, results, frequency and screenshots
  • Combine a T&VS Software Testing team for all your onsite testing requirements with Crowd Software Testing

Additional information on T&VS Crowdsourced Testing services.

Find Out More

Contact one of our software testing consultants today to discuss your requirements. No hard sales, just pertinent questions to understand your needs and to discuss how we may be able to help.


Alternatively call one of our local sales offices.

NMI Software Validation & Verification Event – November 10, 2015

nmi-parent-logo-complete-loT&VS have supported the NMI in the development of a series of events to help companies better understand and overcome the challenges faced in the validation and verification (V&V) of complex electronic systems.

The first of these events will focus on the V&V of Software where the complexity of the environment and the potential for unexpected emergent behaviors means that traditional approaches are not sufficient.

This full day event will look in depth at how software across multiple sectors (automotive, avionics, medical, etc.) can still be made compliant to the appropriate safety standards, and secure from external threats given a systems likely connectivity.

T&VS Presentation

Mike Bartley Founder & CEO, TVSAt the event, T&VS will presenting in the “State of the Art” track on the following:

What can we Learn from Other Disciplines
Mike Bartley, CEO and Founder, T&VS

Event Summary

  • NMI Software Validation & Verification
  • November 10, 2015. — 9:00am – 5:00pm
  • UWE (University of the West of England), Bristol
  • Additional Event Information

Provisional Agenda

Please check the NMI Event Website for the very latest agenda.

  • Welcome & Introduction
    Dr. Derek Boyd, Chief Executive, NMI
  • Keynote Presentation
    Jack Cunningham, Director of Software Engineering, Thales UK
  • Software Verification and Validation Competition (see below)
    Robin Kennedy – Knowledge Transfer Manager, KTN
  • Challenges on Software V&V
    Rob Burton, Head of Software Engineering, Raytheon UK
  • Validation of Mars Missions
    Roger Ward, Technology Manager – Space Division, SCISYS UK
  • Case Studies
  • Disrupting the Smart Cities Market
    Wolfgang Bruchner, CTO, Cascoda
    Sean Redmond, Founder, Vertizan
  • Validating Requirements for the Automotive Industry using Model-Based Systems Engineering (MBSE)
    Professor Jon Holt, Director, Scarecrow Consultants
  • Networking Lunch
  • State-of-the-Art Solutions
  • What can we Learn from Other Disciplines
    Mike Bartley, Managing Director, T&VS
  • Model-based Test Generation for Coverage-Driven Verification of Robotic Code
    Kirsten Eder, Reader, University of Bristol
  • Contestor Verification Approach
    Jamie Hutcheson, Principal Consultant, Altran
  • Verification of Avionics Software to Meet DO-178B/C. What, Why and How?
    Dr. Guillem Bernat, Rapita
  • Verification Techniques for Better Code and Higher Productivity
    Mark Richardson, Field Applications Engineer, LDRA
  • Networking Break
  • Group Discussion on Main Challenges & Potential Solutions
  • Event Close

AESIN 2015 – Ensuring Automotive System Integrity through Advanced Software Verification

The rising tide of opportunities for electronics in the automotive brings with it various technical hurdles not the least of which is ensuring system integrity. Consider for example ADAS (Advanced Driver Assistance Systems) where the operating environments are so complex and so diverse that traditional approaches no longer scale.

aesin-2015b Continue reading

T&VS wins avionics verification project in China to DO-254 standard

PRESS RELEASE

Bristol, UK, 4th November 2014 – T&VS, a leader in software test and hardware verification solutions, today announced that it has won a significant verification service project in China. T&VS is working with its Chinese partner, TopBrain Design Systems, verifying a new avionics FPGA design in compliance with avionics standard DO-254.

The DO-254 standard is recognized by the Federal Avionics Administration (FAA) as a means of compliance for the design of complex electronic hardware in airborne systems. Such hardware includes FPGAs, PLDs and ASICs. The DO-254 standard is the counterpart to the well-established software standard RTCA DO-178B/EUROCAE ED-12B. With DO-254, the FAA recognises that avionics equipment contains both hardware and software, and each is critical to the safe operation of aircraft.

The verification project service by T&VS helps the leading company to verify its safety critical avionics FPGA designs. T&VS will be applying advanced verification techniques such as constrained random verification, functional verification and assertion-based verification to the project. As required by DO-254 requirements traceability will be applied and T&VS will use its unique asureSIGN technology to ensure that requirements can be traced to the verification data generated through the advanced verification techniques being applied.

Mike Bartley, CEO and founder of T&VS, commented, “This deal confirms the strength of our partnership with TopBrain. There is a strong demand for advanced verification expertise in China and with more than 130 skilled engineers globally, T&VS is well equipped to respond to those needs, and it makes perfect sense for T&VS to collaborate with TopBrain.”

Said TopBrain’s president Roger Lee, “This significant win is an important step forward for TopBrain as we look to capitalise on the strong growth that is occurring in the civil aviation industry in China. We appreciate this leading customer’s partnership and look forward to delivering an innovative, quality hardware verification solution that will enable them to get official process compliance certification for their leading project.”

Bartley added, “There are also a number of additional avionics products being designed in China that all require DO-254 compliance where T&VS has lots of expertise.”

T&VS and TopBrain are considering further DO254 projects in China and as part of its strategy in the country, T&VS plans to open an office there in 2015 to strengthen and underpin its presence in the region.

For information on TopBrain’s support and services for T&VS in China visit: www.topbrainds.com.

Further information on T&VS’ products and services is available at: www.testandverification.com.

About T&VS
T&VS (Test and Verification Solutions Ltd) provides services and products to organisations developing complex products in the microelectronics and embedded systems industries. Such organisations use T&VS to verify their hardware and software products, employ industry best practice and manage peaks in development and testing programmes. T&VS’ embedded software testing services includes onsite/offshore testing support including assistance with safety certification and security testing. T&VS hardware verification services include onsite/offshore verification support and training in advanced verification methodologies. T&VS also offers Verification IPs and its own Verification (EDA) signoff tool.

About TopBrain
TopBrain Design Systems provides advanced verification solutions to customers developing complex electronics designs. TopBrain helps its customers to get advanced verification capability in a short time. TopBrain’s products and services include advanced hardware verification methodology service, tools, VIP, training courses, onsite/offshore supports to the local customers. In 2014 TopBrain became the official solution partner of T&VS in China.

T&VS Company Contact
Dr. Mike Bartley – T&VS
+44 7796 307958
[email protected]

TopBrain Company Contact
Roger Lee – TopBrain
+86 21 6630 6399
[email protected]

Web Application Security Testing: The What, Why, Who & How? – EuroSTAR Conference (24-27 Nov. Dublin)

declan-oriordan-thumbnailAt this year’s EuroSTAR conference on software Testing (Dublin, Nov 24-27) I’ll be discussing web application security and how it is completely different to traditional mainframe testing. If you not sure what SQL Injection, Cross-Site Scripting, Cross-Site Request Forgery are, or a host of other ways of attacking your system, then I’d urge you to come along to my talk at 4:00pm on Wednesday 26th November.  In the talk I’ll cover:

  • What web application security testing is
  • Why it’s important
  • Who should be doing it
  • How it should be done

I’ll even be giving away copies of my Application Security Testing procedures.

Please  click here for additional information and to register for EuroSTAR 2014.

I look forward to meeting you at the conference.

 

Declan O’Riordan
Head of Security Testing, T&VS

 

Abstract: EuroSTAR Conference

eurostar-2014-logo
 

A penetration testing expert is better at pen-testing than me, but should I simply delegate application security to specialists and network firewalls? Actually no, I shouldn’t and neither should anyone else involved in the systems development lifecycle.

For years I treated security testing as something akin to black magic beyond my comprehension and penetration testers as technical wizards who could cast out evil hacking spells. Obviously that was daft, but it took some effort to see what was really happening behind the smoke and mirrors of application security, and to de-mystify it for my colleagues.

Follow the journey that led me to believe every well-formed tester can and must have a basic understanding of what application security is, why it is important, who should be doing it, and how.

For a more in-depth look at this topic you may find my white paper on Application Security Testing a useful read.

 Video Trailer

 

 

 

Intelligent Testing Conference (18 Nov. 2014, UK): Improving the Effectiveness of Software Testing

The third Intelligent Testing conference will take place at the UWE Conference Centre in Bristol on the afternoon of 18 November 2014, under the theme “Improving the efficiency and effectiveness of software testing”.

The conference keynote address will be provided by Paul Gerrard, a distinguished software test expert, author of several testing books, recipient of both the EuroSTAR European Testing Excellence award in 2011 and The European Software Testing Awards (TESTA) award for Lifetime Achievement in 2013, and host of the UK Test Management Forum and Chair of the Eurostar 2014 Conference in Dublin later this year. Paul is a popular speaker and is always worth listening to.

Abstract of Paul’s Keynote:

“This talk proposes a model of the thought processes that every tester uses.  In a sentence, what we do is this: “We explore sources of knowledge to build test models that inform our testing”. The model identifies two modes of thinking – exploration and testing – and we use judgement to decide when to flip from one to the other.  Exploration has four activities: enquiring, modelling, predicting and challenging. Testing has six activities: informing, applying, interpreting, logging, reporting and refining. Separating out these ten activities clarifies what testers do.  It identifies the capabilities and skills that all tester need to acquire, to practice and excel in.”

Other Intelligent Testing Presentations

Paul will be joined by a number of other industry speakers who will be presenting the latest developments in software test and verification that can improve software quality and reduce time to market, including the following:

  • Automated Software Defect Prediction Using Machine Learning
    Dr Leandro L. Minku, University of Birmingham – Research Fellow II
  • Service Virtualisation and its Benefit to Testing
    Andrew Thompson, Parasoft – UK Solution Architect
  • Application Security Testing – Where Are We Now?
    Declan O’Riordan, Test and Verification Solutions – Head of Security Testing
  • The SEI Personal Software Process: Improve Yourself!
    Dr Roderick Chapman, Protean Code Ltd – Director

How Do I Find Out More and Register?

Places will be limited and the turnout for our previous conferences has been larger than we expected so please don’t delay your registration for this FREE conference.

To find our more and to Register please Click Here.

 

DVClub – C/C++ Verification Effectiveness

The next DVClub is taking place on Monday, 28th April 2014 where Jean-Marc Forey of Synopsys will be talking about C/C++ Verification Effectiveness and how it is now pervasive in design conception, either through models at different stages of the design creation or by being embedded as firmware in the design itself.  If you have a bug escaping up to production, this can lead to high costs and delays in reaching the market, making it mandatory to assess and improve the effectiveness of the verification of the C/C++ code.  Certitude has been extended from its initial RTL focus to allow users to identify verification issues early in the design cycle.

To listen to Jean-Marc and the other interesting speakers we have lined up, register for one of the venues:  Bristol, Cambridge, Eindhoven, Grenoble and Sophia.  If you are unable to attend one of the venues, you can register via Remote Access.

Intelligent Testing Conference Slides and Recordings Available

The slides and recordings from the second “Intelligent Testing” Conference are now available on our website.  You can listen to the keynote from Rolls Royce talking about using agile in embedded software development and static analysis plus many more:

  • Intelligent Testing in Complex Environments
  • Get Yourself Covered
  • Static Analysis – What, Why, Where, When, Who and How
  • Agile and CI for Embedded Software and Tool Development
  • Static Analysis and Formal Proof for Software Systems using the MALPAS Toolset
  • Virtual Platform Software Simulation for Enhanced Multi-core Software Verification
  • Harnessing Sophisticated Assertion Checking Through Runtime Testing

The presentations were informative and worth reviewing if you couldn’t attend the conference (or hearing again).