Inspired by the results presented on the GPT creators'  regarding its approach to various exams, we decided to subject AI to another test.

 

Due to limitations in the publicly available version of ChatGPT, the experiment was unable to ask questions based on graphics shown in the test. We also assume that ChatGPT, even if it learned based on the ISTQB® syllabus, treats it as one of many sources of knowledge. Its incorrect answers are therefore incorrect only with respect to this particular source.

We assume that ChatGPT does not have access to ISTQB® answers and develops its answers "independently".

 

Questions asked

Question 1.
Which one of the following answers describes a test condition?

a) A distinguishing characteristic of a component or system
b) A testable aspect of a component or system identified as a basis for testing
c) The degree to which a software product provides functions which meet stated and implied needs when the software is used under specified conditions
d) Test cases designed to execute combinations of conditions and actions resulting from them

Select ONE option.

ChatGPT's answer:
b) A testable aspect of a component or system identified as a basis for testing

A test condition is a specific and testable aspect of a component or system that has been identified as the basis for testing. It describes a characteristic, function, behavior, or attribute that must be tested to confirm that the software meets requirements and operates as expected. Test conditions are often extracted from requirements, specifications, design documentation, or other sources of information and are used to design and execute test cases.

CORRECT!

Question 8.
Match the following test work products (1-4) with the right description (A-D).

1. Test suite
2. Test case
3. Test script
4. Test charter

A. A set of test scripts to be executed in a specific test run
B. A set of instructions for the execution of a test
C. Contains expected results
D. Documentation of test activities in session-based exploratory testing

a) 1A, 2C, 3B, 4D
b) 1D, 2B, 3A, 4C
c) 1A, 2C, 3D, 4B
d) 1D, 2C, 3B, 4A

Choose one answer.

ChatGPT's answer:
c) 1A, 2C, 3D, 4B

INCORRECT. The correct answer is a)

Question 12.
Which one of the following is the BEST definition of an incremental development model?

a) Defining requirements, designing software and testing are done in phases where in each phase a piece of the system is added
b) A phase in the development process should begin when the previous phase is complete
c) Testing is viewed as a separate phase which takes place after development has been completed
d) Testing is added to development as an increment

Select ONE option.

ChatGPT's answer:
a) Defining requirements, designing software and testing are done in phases where in each phase a piece of the system is added

CORRECT!

Question 18.
You will be invited to a review. The work product to be reviewed is a description of the in-house document creation process. The aim of the description is to present the work distribution between the different roles involved in the process in a way that can be clearly understood by everyone.

You will be invited to a checklist-based review. The checklist will also be sent to you. It includes the following points:

i. Is the person who performs the activity clearly identified for each activity?
ii. Are the entry criteria clearly defined for each activity?
iii. Are the exit criteria clearly defined for each activity?
iv. Are the supporting roles and their scope of work clearly defined for each activity?

In the following we show an excerpt of the work result to be reviewed, for which you should use the checklist above:

"After checking the customer documentation for completeness and correctness, the software architect creates the system specification. Once the software architect has completed the system specification, he invites testers and verifiers to the review. A checklist describes the scope of the review. Each invited reviewer creates review comments - if necessary - and concludes the review with an official review done-comment."

Which of the following statements about your review is correct?

a) Point ii) of the checklist has been violated because it is not clear which condition must be fulfilled to invite to the review
b) You notice that in addition to the tester and the verifier, the validator must also be invited. Since this item is not part of your checklist, you do not create a corresponding comment
c) Point iii) of the checklist has been violated as it is not clear what marks the review as completed
d) Point i) of the checklist has been violated because it is not clear who is providing the checklist for the invitation to the review

Select ONE option.

ChatGPT's answer:
a) Point ii) of the checklist has been violated because it is not clear which condition must be fulfilled to invite to the review

The text does not clearly specify what conditions must be met to invite testers and verifiers to the review. The lack of clear input criteria violates point i) of the checklist.

INCORRECT. The correct answer is d)

Question 22.
Which one of the following is the description of statement coverage?

a) It is a metric, which is the percentage of test cases that have been executed
b) It is a metric, which is the percentage of statements in the source code that have been executed
c) It is a metric, which is the number of statements in the source code that have been executed by test cases that are passed
d) It is a metric, that gives a true/false confirmation if all statements are covered or not

Select ONE option.

ChatGPT's answer:
b) It is a metric, which is the percentage of statements in the source code that have been executed

CORRECT!

Question 25.
An employee’s bonus is to be calculated. It cannot be negative, but it can be calculated down to zero. The bonus is based on the length of employment:

• Less than or equal to 2 years
• More than 2 years but less than 5 years
• 5 to 10 years inclusively
• Longer than 10 years

What is the minimum number of test cases required to cover all valid equivalence partitions for calculating the bonus?

a) 3
b) 5
c) 2
d) 4

Select ONE option.

ChatGPT's answer:
Considering the division into categories of length of employment, we have 4 equivalence classes. To cover correct cases, we need to test one case for each of these classes, giving a total of 4 test cases.

Answer: d) 4

CORRECT!

Question 29.
A video application has the following requirement: The application shall allow playing a video on the following display resolution:

1. 640x480
2. 1280x720
3. 1600x1200
4. 1920x1080

Which of the following list of test cases is a result of applying the equivalence partitioning test technique to test this requirement?

a) Verify that the application can play a video on a display of size 1920x1080 (1 test case)
b) Verify that the application can play a video on a display of size 640x480 and 1920x1080 (2 test cases)
c) Verify that the application can play a video on each of the display sizes in the requirement (4 test cases)
d) Verify that the application can play a video on any one of the display sizes in the requirement (1 test case)

Select ONE option.

ChatGPT's answer:
c) Verify that the application can play a video on each of the display sizes in the requirement (4 test cases)

CORRECT!

Question 35.
The project develops a "smart" heating thermostat. The control algorithms of the thermostat were modeled as Matlab/Simulink models and run on the internet connected server. The thermostat uses the specifications of the server to trigger the heating valves.

The test manager has defined the following test strategy/approach in the test plan:

1. The acceptance test for the whole system is executed as an experience-based test.
2. The control algorithms on the server are checked against standard of the energy saving regulation.
3. The functional test of the thermostat is performed as risk-based testing.
4. The security tests of data / communication via the internet are executed together with external security experts.

What four common types of test strategies/approaches did the test manager implement in the test plan?

a) Methodical, analytical, reactive, and regression-averse
b) Analytical, standard-compliant, consultative, and reactive
c) Model-based, methodical, analytical, and consultative
d) Regression-averse, consultative, reactive, and methodical
Select ONE option.

ChatGPT's answer:
b) Analytical, standard-compliant, consultative, and reactive

CORRECT!

Question 38.
You are testing a new version of software for a coffee machine. The machine can prepare different types of coffee based on four categories. i.e., coffee size, sugar, milk, and syrup. The criteria are as follows:

• Coffee size (small, medium, large)
• Sugar (none, 1 unit, 2 units, 3 units, 4 units)
• Milk (yes or no)
• Coffee flavor syrup (no syrup, caramel, hazelnut, vanilla)

Now you are writing a defect report with the following information:

• Title: Low coffee temperature.
• Short summary: When you select coffee with milk, the time for preparing coffee is too long and the temperature of the beverage is too low (less than 40 °C).
• Expected result: The temperature of coffee should be standard (about 75 °C).
• Degree of risk: Medium
• Priority: Normal

What valuable information was omitted in the above defect report?

a) The actual test results
b) Identification of the tested software version
c) Status of the defect
d) Ideas for improving the test case

Select ONE option.

ChatGPT's answer:
a) The actual test results

INCORRECT. The correct answer is b)

Question 39.
Which one of the following is MOST likely to be a benefit of test execution tools?

a) It is easy to create regression tests
b) It is easy to maintain version control of test assets
c) It is easy to design tests for security testing
d) It is easy to run regression tests

Select ONE option.

ChatGPT's answer:
d) It is easy to run regression tests

CORRECT!

In this simplified test, ChatGPT was able to correctly answer 7 out of 10 ISTQB questions, resulting in a score of 70%. Since the passing threshold is set at 65%, it has a good chance of passing the full ISTQB® Foundation Level exam.