Excluding files from indexing

[felixfbecker]

For example, node_modules

[Alanaktion]

I would particularly like the ability to exclude certain directories from the Problems list (e.g. vendor), while retaining Go to Definition functionality. I'm not sure if that would be considered a separate issue.

[felixfbecker]

#164
Warnings from vendor are already ignored.

[ryanbowden]

Really need this feature hope it comes soon. to make the searching much quicker.

[felixfbecker]

@ryanbowden feel free to do a PR

[damienflament]

The main use case is to ignore the cache files.

[damienflament]

After seeing the code, the Indexer have to get the excluded path from options and give it to the FileFinder as second parameter. The FileFinder exclude files matching the given Globs.

I think it's the simplest way to do it as trying to compile a glob pattern using the main pattern (which will be dynamically generated due to another feature request) and the excluding patterns seems complicated.

I need to make a development environment and try to submit a PR! When I got free time...

[alex-pex]

from #381:
I'd like php intellisense to read files.exclude setting + php.excludeFiles (or a similar new config entry) with this value by default

{
  "php.excludeFiles": {
    "**/node_modules": true"
  }
}

[jens1o]

As soon as the large pr has been merged, I planned to do this, but there are some things that need to be implemented first. (For example the client needs to send the settings to the server.)

[dominikzogg]

atom/ide-php#20

[AeonDigital]

For those who do not want to wait for an update and do not mind making a "in the hand" follows a simple and homely solution:

1. Open "C:\Users[user].vscode\extensions\felixfbecker.php-intellisense-1.5.1" in vscode.

2. Open "\out\extension.js" and edit:
   look for line 22 and add the new instruction below:

   22: const memoryLimit = conf.get('memoryLimit') || '-1';
   23: const intellisense = conf.get('intellisense') || {'excludePaths': ''}; // new instruction
   24: if (memoryLimit !== '-1' && !/^\d+[KMG]?$/.exec(memoryLimit)) { ...

   So, go to line 68 and add this:

   68: args.push('--memory-limit=' + memoryLimit);
   69: args.push('--exclude-paths=' + intellisense.excludePaths); // new instruction
   70: const childProcess = child_process_1.spawn(executablePath, args);

   These new instructions will start the extension passing a value that you indicate in your configuration file as the template:

   "php.intellisense": {
   "excludePaths": "path1, path2"
   }

   As this is a solution without major pretensions it is important to note that:
   a. only directories will be properly removed.
   b. you must enter the full path of the target directory starting from the root of the application that is currently open.
   c. the targets must be separated by commas.
   d. it is not possible to use wildcards

3. Open "\vendor\felixfbecker\language-server\src\Indexer.php" and edit:
   Go to line 110 and after it add the following

   $uris = yield $this->filesFinder->find($pattern);
   
   // new lines
   $options = getopt('', ['exclude-paths::']);
   $excludePaths = (($options['exclude-paths'] ?? ''));
   
   if ($excludePaths !== '') {
       $excludeDirs = array_map('trim', explode(',', $excludePaths));
   
       $ed = [];
       foreach($excludeDirs as $d) {
           $ed[] = str_replace(['\\', ' '], ['/', '%20'], 'file:///' . $this->rootPath . '/' . trim($d, '\\/')) . '/';
       }
       $excludeDirs = $ed;
   
       $nUris = [];
       foreach ($uris as $uri) {
           $include = true;
           foreach($excludeDirs as $exc) {
               if (strpos($uri, $exc) !== false) {
                   $include = false;
                   break;
               }
           }
   
           if ($include) {
               $nUris[] = $uri;
           }
       }
       $uris = $nUris;
   }
   // end of new lines
   
   $count = count($uris);
   

keep in mind: this solution I did in a few hours of this idle Saturday just because I found it fun, so I have no intention of providing a definitive or perfect solution but for the cases where I need it it worked perfectly

I hope someone can enjoy it.

[felixfbecker]

@AeonDigital I appreciate the time you spent on writing this together, but why not invest that time into a PR?

[AeonDigital]

@felixfbecker Hello. sorry but I do not think it would be good for the project to make a PR of something that was not sufficiently tested. I confess that I know that the way I have resolved the issue is not the best. I just decided to publish the way in which I resolved something that was bothering me and I considered that other people could take advantage of it at their own risk.

[ryanbowden]

@felixfbecker Want me to take this code and try and get a pull request sorted? I can have a go tonight.

Something I really need because otherwise, the CPU maxes out.

[felixfbecker]

Sure. Here's what it'd have to do:

• Introduce a Configuration class
• Implement didChangeConfiguration
• Whenever it changes, change the Configuration instance
• Pass the Configuration instance to the Indexer
• Read that setting
• Use the glob to determine files to be indexed

[Ralle]

I am editing code over a network share and it seems like this LSP is crashing because there's a protected folder it shouldn't index.

UnexpectedValueException: RecursiveDirectoryIterator::__construct(z:/Hive-XF2\docker-data\mysql\mysql,z:/Hive-XF2\docker-data\mysql\mysql): An unexpected network error occurred. (code: 59) in C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\webmozart\glob\src\Iterator\RecursiveDirectoryIterator.php:43 Stack trace: #0 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\webmozart\glob\src\Iterator\RecursiveDirectoryIterator.php(43): RecursiveDirectoryIterator->__construct('z:/Hive-XF2\\doc...', 4128) #1 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\webmozart\glob\src\Iterator\RecursiveDirectoryIterator.php(55): Webmozart\Glob\Iterator\RecursiveDirectoryIterator->__construct('z:/Hive-XF2\\doc...', 4128) #2 [internal function]: Webmozart\Glob\Iterator\RecursiveDirectoryIterator->getChildren() #3 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\webmozart\glob\src\Iterator\RegexFilterIterator.php(99): FilterIterator->rewind() #4 [internal function]: Webmozart\Glob\Iterator\RegexFilterIterator->rewind() #5 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\felixfbecker\language-server\src\FilesFinder\FileSystemFilesFinder.php(24): IteratorIterator->rewind() #6 [internal function]: LanguageServer\FilesFinder\FileSystemFilesFinder->LanguageServer\FilesFinder\{closure}() #7 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\coroutine.php(64): Generator->valid() #8 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\coroutine.php(118): Sabre\Event\{closure}() #9 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\felixfbecker\language-server\src\FilesFinder\FileSystemFilesFinder.php(33): Sabre\Event\coroutine(Object(Closure)) #10 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\felixfbecker\language-server\src\LanguageServer.php(207): LanguageServer\FilesFinder\FileSystemFilesFinder->find('z:/Hive-XF2/**/...') #11 [internal function]: LanguageServer\LanguageServer->LanguageServer\{closure}() #12 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\coroutine.php(88): Generator->send(NULL) #13 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\coroutine.php(118): Sabre\Event\{closure}() #14 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\felixfbecker\language-server\src\LanguageServer.php(292): Sabre\Event\coroutine(Object(Closure)) #15 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\felixfbecker\advanced-json-rpc\lib\Dispatcher.php(160): LanguageServer\LanguageServer->initialize(Object(LanguageServerProtocol\ClientCapabilities), 'z:\\Hive-XF2', 1956, 'file:///z:/Hive...') #16 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\felixfbecker\language-server\src\LanguageServer.php(131): AdvancedJsonRpc\Dispatcher->dispatch(Object(AdvancedJsonRpc\Request)) #17 [internal function]: LanguageServer\LanguageServer->LanguageServer\{closure}() #18 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\coroutine.php(64): Generator->valid() #19 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\coroutine.php(118): Sabre\Event\{closure}() #20 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\felixfbecker\language-server\src\LanguageServer.php(154): Sabre\Event\coroutine(Object(Closure)) #21 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\EmitterTrait.php(88): LanguageServer\LanguageServer->LanguageServer\{closure}(Object(LanguageServer\Message)) #22 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\felixfbecker\language-server\src\ProtocolStreamReader.php(56): Sabre\Event\Emitter->emit('message', Array) #23 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\Loop\Loop.php(311): LanguageServer\ProtocolStreamReader->LanguageServer\{closure}() #24 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\Loop\Loop.php(233): Sabre\Event\Loop\Loop->runStreams(NULL) #25 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\Loop\Loop.php(194): Sabre\Event\Loop\Loop->tick(true) #26 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\sabre\event\lib\Loop\functions.php(122): Sabre\Event\Loop\Loop->run() #27 C:\Users\Rasmus\.vscode\extensions\felixfbecker.php-intellisense-2.3.10\vendor\felixfbecker\language-server\bin\php-language-server.php(55): Sabre\Event\Loop\run() #28 {main}

[erralb]

Hi,
Is it now possible to ignore some files/folders like node_modules? How?

[kinoute]

I would like to know if it's possible as well.

[smilearric]

Same here!

Hello,

Thanks for the efforts to exclude files & folders from being indexed by the PHP Language Server. I'm using Intelephense so I wouldn't like overlapping features and their performances issues.

I'd like to know how to exclude some cache or log files. Having read this issue, its duplicates and scrolled the related preferences settings I still could not figure it out.

On a Symfony project, thousands of cache files are unnecessarily indexed, eventhough the "PHP › Validate: Enable" (php.validate.enable) or "PHP › Suggest: Basic" (php.suggest.basic) setting are unset.

Folders have been globally excluded this way as well but that doesn't work:

    "files.watcherExclude": {
        ".idea/**": true,
        ".vscode/**": true,
        "**/var/cache/**": true,
        "**/var/logs/**": true,
        "**/var/sessions/**": true
    },
    "search.exclude": {
        ".idea/**": true,
        ".vscode/**": true,
        "**/var/cache/**": true,
        "**/var/logs/**": true,
        "**/var/sessions/**": true
    },
    "files.exclude": {
        ".idea/**": true,
        ".vscode/**": true,
        "**/var/cache/**": true,
        "**/var/logs/**": true,
        "**/var/sessions/**": true
    },

I could not find some documentation either.

Regards

---

EDIT: After Felix's answer "No, that's why this issue is open", I would like to say good luck to the devs.

[mmcarvalho]

I can't find the proper settings to exclude folders. Is this implemented yet?

[felixfbecker]

No, that's why this issue is open.

[buttflattery]

@AeonDigital you instructions for the out/extension.js isnt valid any more.

[AnthonyBonnier]

You can still modify the Indexer.php file to change the pattern used.
The file is located here : C:\Users\[USER]\.vscode\extensions\felixfbecker.php-intellisense-[VERSION]\vendor\felixfbecker\language-server\src\Indexer.php

The function to change is named index and the pattern is built at the beginning :

public function index(): Promise
    {
        return coroutine(function () {
            // Old code using the rootPath
            //$pattern = Path::makeAbsolute('**/*.php', $this->rootPath);
            // My new pattern 
            $pattern = Path::makeAbsolute('**/*.php', 'C:/Users/[USER]/Projects/sources/app/code');

            $uris = yield $this->filesFinder->find($pattern);
            // ...

Restart VS Code after saving the changes and it will only index the needed path.

[teratux]

    $ed[] = str_replace(['\\', ' '], ['/', '%20'], 'file:///' . $this->rootPath . '/' . trim($d, '\\/')) . '/';

This works, the only fix is to change the quoted line to:

$ed[] = str_replace(['\\', ' '], ['/', '%20'], 'file://' . $this->rootPath . '/' . trim($d, '\\/')) . '/';

two "//" instead of "///", works for me on vscodium 1.36.1

[joas8211]

I've created some pull requests that might be solution for this (#749, felixfbecker/vscode-php-intellisense#437).

I'm usinig language server initialization options for passing excludes derived from VSCode's workspace settings (files.exclude). The proper way I guess would be to synchronize configurations, but this is the easiest. User currently just has to remember to reload (restart language server) when the setting (files.exclude) is changed.

[pabloGillariCes]

Hey, gals and guys.

I wanted to share a workaround I'm using that it maybe can help someone reading.

Since I'm using Docker to run all my projects, and ONLY Docker, if I call composer commands from inside the containers, applications just work no matter much the owner of those files.

If I want to update a dependency file (which should never happen) or I want to delete the vendor folder to run composer again from scratch, then I'll need to set the right permissions from outside the container, I mean, from my host machine.

So...

If you set chmod 600 (not recursive) to vendor, var/cache or whichever folder you'd like to exclude from scanning, the PHP Language Server is not able to parse files or find problems inside those folders anymore.

This is different from using "files.exclude" because you won't see them on folders navigator but also won't keep them from being scanned, which defeats the purpose of this feature request. You'll also not be able to read any file on those folders, but maybe you really don't care... or you shouldn't.

I went from 53.000+ files scanned along 8 projects with one of my CPU processors at 99% during 15/20 minutes, to 1560 files that actually belong to my applications.

[Info  - 9:59:44 AM] 1560 files total
[Info  - 9:59:44 AM] Indexing project for definitions and static references
...
...
...
[Info  - 10:01:03 AM] 0 Packages
[Info  - 10:01:03 AM] All 1560 PHP files parsed in 79 seconds. 158 MiB allocated.

Regards.

[gnoe]

The indexer is created in the LanguageServer::initialize method, but is not assigned to the language server, nor the Server\Workspace
would be a good idea to pass the indexer to the Server\Workspace?
on a Workspace::didChangeConfiguration we can re-index if the exclude patterns change

[otech-nl]

Thanks @pabloGillariCes
Works like a charm!

[JenCant]

@pabloGillariCes Thanks this worked for me - I had this error when accessing a repo stored on a mounted drive on MacOS. It was breaking on the system .Trashes folder.

[buttflattery]

@pabloGillariCes that means if someone is developing an extension using the following settings in the composer.json

 "repositories": [
        {
            "type": "path",
            "url": "../yii2-somextension"
        },
  ]

It wont scan those files, which is not someone would want as the composer extension development is more easier and time saving this way, where he/she can create a symlink to the folder where the files actually reside and the symlink is under the vendor directory

[dpolkhov]

Hello guys.
Do you have any update about this issue? For the moment I have more than 500 problems in .history folder.

Only #159 (comment) helped, but it's not a good idea for me to edit vendor files.

[sarim]

Its sad to see this is still open. I quickly hacked together a solution by editing vendor -> Indexer.php directly. I know its a bad solution.

$uris = array_values(array_filter($uris, function( $v, $k ) {
    return strpos($v, 'var/cache') === false;
}, ARRAY_FILTER_USE_BOTH));

$ignoredCount = $count - count($uris);
$count = count($uris);
$this->client->window->logMessage(MessageType::INFO, "$ignoredCount files ignored");

Before:
[Info - 5:26:45 PM] All 5709 PHP files parsed in 32 seconds. 412 MiB allocated.
After:
[Info - 5:28:51 PM] All 4858 PHP files parsed in 3 seconds. 404 MiB allocated.

It is symfony project, with very little actual code, the difference is mainly in time bcz the cache files are very big and and many of them fails due to big size, so either way they don't contribute much to memory, but initial scan takes very long time. 32 v 3 sec is a huge win.

I've read the comments above, I'm opting for a separate php language server ignore files option, as I think using .gitignore or other settings would not be the perfect use case here. I'll try to make a PR, but no promises.

[jasonwilliams]

I ended up moving to Intellephense, it indexes a lot better https://marketplace.visualstudio.com/items?itemName=bmewburn.vscode-intelephense-client

[kralos]

Based on felixfbecker.php-intellisense-2.3.14 I hacked an ignore into:

~/.vscode/extensions/felixfbecker.php-intellisense-2.3.14/vendor/felixfbecker/language-server/src/Indexer.php

diff --git a/src/Indexer.php b/src/Indexer.php
index 7ebff3f..ca30170 100644
--- a/src/Indexer.php
+++ b/src/Indexer.php
@@ -204,6 +204,15 @@ class Indexer
     {
         return coroutine(function () use ($files) {
             foreach ($files as $i => $uri) {
+                // Skip paths
+                foreach ([
+                    '/var/cache/',
+                ] as $skipPath) {
+                    if (false !== strpos($uri, $skipPath)) {
+                        continue 2;
+                    }
+                }
+
                 // Skip open documents
                 if ($this->documentLoader->isOpen($uri)) {
                     continue;

[reduardo7]

Based on felixfbecker.php-intellisense-2.3.14 I hacked an ignore into:

~/.vscode/extensions/felixfbecker.php-intellisense-2.3.14/vendor/felixfbecker/language-server/src/Indexer.php

diff --git a/src/Indexer.php b/src/Indexer.php
index 7ebff3f..ca30170 100644
--- a/src/Indexer.php
+++ b/src/Indexer.php
@@ -204,6 +204,15 @@ class Indexer
     {
         return coroutine(function () use ($files) {
             foreach ($files as $i => $uri) {
+                // Skip paths
+                foreach ([
+                    '/var/cache/',
+                ] as $skipPath) {
+                    if (false !== strpos($uri, $skipPath)) {
+                        continue 2;
+                    }
+                }
+
                 // Skip open documents
                 if ($this->documentLoader->isOpen($uri)) {
                     continue;

My solution is with REGEX:

// ...
        // Skip paths
        foreach ([
          '/\.history/',
          '/\.git/',
          '/\.vscode/',
          '/\.data/',
        ] as $skipPath) {
          if (!preg_match($skipPath, $uri)) {
            continue 2;
          }
        }
// ...

I'm working in a PR, using VSCode settings.