StdErr output is after StdOut in step definition `the output should contain`

[archerzz]

Summary

I found that in version 1.0.0, the contents of StdOut and StdErr do not keep their original sequence in step definitions like the output should contain. More specific, StdErr is after StdOut.

Version 0.14.x is fine. So I think this should be a regression.

Expected Behavior

The output of StdErr and StdOut should keep their original sequence in step definitions like the output should contain.

Current Behavior

If the app prints some contents to StdErr first, and then to StdOut, like below:

Errors
Normal

We write the scenario according to the actual output.

However in step definitions like the output should contain, it will treat the output like:

Normal
Errors

Thus the comparison will fail.

Possible Solution

Roll back to old versions.

Steps to Reproduce (for bugs)

1. Try an CLI app which will print to StdErr before StdOut
2. Compose and run a BDD test case under version 1.0.0
3. Compose and run a BDD test case under old versions, like 0.14.4

Context & Motivation

This will break CLI apps which mix StdOut and StdErr outputs (like put some warning messages between normal output)

Your Environment

• Version used: 1.0.0
• Operating System and version: all
• Link to your project: N/A Sorry I could not provide it, since it's not an open source project.

[mvz]

Thanks @archerzz, that is indeed a regression.

[mvz]

Hi @archerzz I've just tried to reproduce this and it looks like this particular behavior is the same in 0.14.4, 0.14.14 and 1.0.0. In all versions, the combined output contains all of stdout and then all of stderr.

Therefore, I think the problem may be more subtle.

I know you can't share your project, but can you create a minimal test case that succeeds with 0.14.14 and fails with 1.0.0?

[archerzz]

Hi @mvz, sorry for the late response.

My original report is not correct. I doubled checked my daily test logs, and I think this problem should exist in 0.14.x or maybe even in earlier versions as well.

[mvz]

No problem. I've changed the title of this issue accordingly.