run-make tests initialized

[lordshashank]

• resolves Run other types of tests like run-make rust-lang/rustc_codegen_gcc#263
• Added run-make tests
• made failing-run-make-tests.txt for failing tests.

[lordshashank]

@antoyo added libstdc++-12-dev (if this is how you meant), also removed all failing tests so that I can get all the tests failing in ci, independent of my local system, hope thats not an issue.

[antoyo]

It's good. Thanks!

[antoyo]

Weird. We still get the error "/usr/bin/ld: cannot find -lstdc++: No such file or directory". Did you ever get that locally?

[lordshashank]

@antoyo I am thinking to deprecate version to 11, and putting this in ci to debug

    - name: Install packages
      # `llvm-14-tools` is needed to install the `FileCheck` binary which is used for asm tests.
      run: |
          sudo apt-get install ninja-build ripgrep llvm-14-tools libstdc++-11-dev
          locate libstdc++
          echo $LD_LIBRARY_PATH

should I do that (not sure we should do echo in ci runs)?
ALso do review my code, in case there's some mistake, there.

[antoyo]

You want to try version 11 to see if this will fix the tests? If so, yes, this is a good idea.

I wanted to review the PR when the CI pass.
The PR mostly looks good: the changes I would want to see are less code duplication because we'll need to do the same for other test suites in Rust like codegen-units,run-pass-valgrind, and incremental.
So, functions like test_rustc_inner_run_make and the callbacks sent to this function should get refactored to remove code duplication with test_rustc_inner_ui.

Another thing I'm still thinking about is whether we split those new tests like run-make since there seems to be much less of them than UI tests.

[lordshashank]

@antoyo I tried to reduce duplication by modularising some code (thought a lot, let me know if you have better way). Also I tried to remove the ui tests here to failing-ui-tests, but ci failed so reseted them, but CI still fails. These are running in my device successfully, not getting why are they failing here, could you please help.

[antoyo]

It looks like some test files are being removed twice, causing this error:

Command failed to run: Failed to remove `build/rust/tests/ui/impl-trait/equality-in-canonical-query.rstests/ui/asm/x86_64/issue-96797.rs`: Os { code: 2, kind: NotFound, message: "No such file or directory" }

Since we're removing files, I guess we might need to do a git checkout somewhere between the two runs to restore the files.

[lordshashank]

@antoyo I am not able to get the issue here as both the runs are failing here, moreover I am not able to replicate this, any views.

[antoyo]

Would you be OK if I push commits to this PR to attempt to fix the issue?

[lordshashank]

ya sure 😓, there have ben too many commits, should I squash them first?

[antoyo]

Yes please.

[lordshashank]

@antoyo , you can do changes, actually its showing the change in same file in failing-ui-tests.txt , but its irrelevant i guess cause it shows same line added and removed, some ci issue i guess, look forward to what solution is.

[lordshashank]

could shifting this test to test.rs for ui-tests from failing-ui-tests.txt resolve the issue?, cause its this test only which is giving issue on all failing ci runs (I rechecked them)

[lordshashank]

@antoyo, so checks are successful finally, let me know of any more improvements,
One I wanted your view on was I removed the tests splitting for run-make tests, but now they are getting ran on both runs unnecessarily, should I reinstate splitting or put a check to run run-make test only when current_parts is 0? (I think later one would be good)

[antoyo]

Very nice work!

[lordshashank]

@antoyo, so checks are successful finally, let me know of any more improvements, One I wanted your view on was I removed the tests splitting for run-make tests, but now they are getting ran on both runs unnecessarily, should I reinstate splitting or put a check to run run-make test only when current_parts is 0? (I think later one would be good)

@antoyo should I do this?

[antoyo]

@antoyo, so checks are successful finally, let me know of any more improvements, One I wanted your view on was I removed the tests splitting for run-make tests, but now they are getting ran on both runs unnecessarily, should I reinstate splitting or put a check to run run-make test only when current_parts is 0? (I think later one would be good)

@antoyo should I do this?

I'll think about this. The reason why the ui tests are splitted is to make the CI faster, but this doesn't seem necessary for the make-run tests.

You might not know, but there's a summary of failing tests here which shows the number of failing tests for the Failures CI task.
With your PR, the results shown there only show the failures for the make-run tests: perhaps this is because the ui failing tests do not seem to be ran.

I'd like to see how this will render in the summary when those ui failing tests are ran in the CI: if it looks good, perhaps we can just keep the make-run tests the way it is (ran twice).

[lordshashank]

all tests successful LFG 🚀. @antoyo let me know if some more changes are needed.

[antoyo]

I missed a few things.
After that, we can merge.
Thanks a lot!

[antoyo]

One more nitpick ;) .

[lordshashank]

Really sorry @antoyo , I don't know why that spacing at last got ommitted even though I tried to add it 😭, this should work now.

[antoyo]

Really sorry @antoyo , I don't know why that spacing at last got ommitted even though I tried to add it 😭, this should work now.

No worries.
I know that some text editor removes the last new line in files. Perhaps your text editor has a setting to configure this behavior?

[antoyo]

Sorry, I missed those two comments.
After that, we can merge for real :) .

[lordshashank]

Done and sorry, I should not have removed the comments, would remember for next time.

[lordshashank]

@antoyo , what other test suites of rust do you look forward to implement, I can try them as I guess they would be fairly easy to implement after this modularization.

[antoyo]

I would find the most important to be the standard library (std) tests.