By a lot of work, I mean the work of hunting down the problems when they are detected by the randomized tests. Also such test sometimes/usually/mostly exercise the tests themselves and then require a lot of work to "resist" the randomized tests. So it is really not just about putting in place randomized testing with a few config lines. It involves a lot of maintenance work down the line. On 1/14/2022 10:01 AM, Ceki Gülcü wrote:

On 1/14/2022 9:00 AM, Vladimir Sitnikov wrote:

...

...

I am coming from afar and had not heard of Github Actions previously. However, I am somewhat familiar with Travis and its predecessor (name?).

GitHub Actions is similar to Travis, and the key differences are: a) Travis is dying nowadays (they have strict limits, etc) b) Actions are better integrated with GitHub UI c) Actions are more powerful

For instance, GitHub Actions allow randomizing test matrix on the fly, so you can have good coverage without hard-coding all the possibilities and without waiting for the feedback for hours.

I'm planning to switch matrix generation to https://github.com/vlsi/github-actions-random-matrix <https://github.com/vlsi/github-actions-random-matrix> a bit later. That would introduce testing with unusual locales, unusual languages, JVMs, etc.

That sounds amazing but also a lot of work. But amazing...

I am still looking at the GitHub Actions PR.

...

Vladimir

-- Ceki Gülcü Please contact support(at)qos.ch for donations, sponsorship or support contracts related to SLF4J/logback/reload4j projects.

On 1/14/2022 10:16 AM, Vladimir Sitnikov wrote:

...

It involves a lot of maintenance work down the line.

Just a question: do you think reload4j should be buildable in Turkish or Russian locale out of the box? Is it tested right now?

Yes, of course. It should build everywhere. No, not tested currently.

Do you want to catch bugs like "wrong results if Object#hashCode yields collisions"?

That is not kind of problem I am talking talking about. I am talking about a problem which occurs only on saturday April the 24th, 2021 between 23:30 and 23:45 in a country with a 30 minute time zone and DST. These are horribly difficult to reproduce and when solved affect only the test code and not the code under test.

Of course, it is more work, if you want to capture those bugs and kill them.

...

By a lot of work, I mean the work of hunting down the problems when they are detected by the randomized tests.

Al I am saying is that such infrastructure may look nice and cuddly but the step may not be so innocuous.

Not really. I introduced a randomized test matrix to the TestNG project a year ago, and it does look reasonable and easy to understand.

Here's the recent build: https://github.com/cbeust/testng/actions/runs/1696812649 <https://github.com/cbeust/testng/actions/runs/1696812649>

OK. Time will tell.

Vladimir

-- Ceki Gülcü Sponsoring SLF4J/logback/reload4j at https://github.com/sponsors/qos-ch

On 1/14/2022 1:01 PM, Vladimir Sitnikov wrote:

...

OK. Time will tell.

Exactly. CI is easy to reconfigure or rollback or whatever.

There are downsides, however, I see no reasons to even discuss them. What do we do with the downsides?

Do you have other options? Do you suggest hand-crafting the test matrix? Do you suggest we should ignore jvm, os, locale, timezone, hashcode, ... variations? Do you suggest we should always execute all the possible combinations?

Even the current "3 JVM across 3 OS" yields 9 CI jobs that take noticeable time, and multiplying that by several timezones and locales would yield 30+minute feedback cycles.

I am looking into it right now.

Vladimir

-- Ceki Gülcü Sponsoring SLF4J/logback/reload4j at https://github.com/sponsors/qos-ch

fail-fast: true, so the CI terminates on the first failure. You still need to merge the second commit from PR12, or fix MDCTestCase somehow. Vladimir